You need an efficient, maintainable workflow to process hundreds of tables: files land in Cloud Storage unpredictably, Dataproc transforms and writes to BigQuery, then table-specific BigQuery transformations run (may take hours). What workflow design is best?

This option provides the most efficient and maintainable solution. Using a Cloud Storage object trigger to launch a Cloud Function that triggers the DAG ensures that processing starts immediately when a file lands, addressing the "unpredictably" requirement. Creating a separate DAG per table, while initially more setup, offers better maintainability and isolation for table-specific BigQuery transformations that "may take hours." This prevents one long-running transformation from blocking or delaying other tables' processing. Building Airflow DAGs in Cloud Composer with appropriate operators is the standard for complex data workflows on GCP. The incorrect options fail in several ways: Scheduling DAGs hourly (options 1 and 2) doesn't address the unpredictable arrival of files and introduces unnecessary latency. Using a single shared DAG for all tables (options 1 and 3) creates a monolithic workflow that is harder to maintain, debug, and scale, especially when individual table transformations can take hours. This also makes it difficult to handle failures or retries for specific tables without affecting others.