You created a Dataprep recipe on a BigQuery sample and need to run it daily after a load job with variable completion time. How do you automate and orchestrate this?

The correct approach is to export the Dataprep job as a Dataflow template and integrate it into a Cloud Composer (Apache Airflow) workflow. Dataprep jobs internally use Dataflow for execution, so exporting as a Dataflow template allows for programmatic control. Cloud Composer provides robust orchestration capabilities, including dependency management (e.g., waiting for the load job to complete), scheduling, and error handling, which are essential for production data pipelines with variable completion times. Creating a cron schedule in Dataprep is too simplistic; it lacks the ability to wait for external dependencies like a load job. An App Engine cron job could trigger the Dataprep job, but it offers less sophisticated orchestration and dependency management compared to Composer. Exporting as a Dataprep template and using Cloud Scheduler is also insufficient as Cloud Scheduler is a simple cron service and cannot manage complex dependencies or workflows.