Monitor BigQuery, Dataflow, and Dataproc pipelines across multiple projects and notify the pipeline team on failures using managed GCP features. What should you do?

Exporting pipeline metrics and logs to Cloud Monitoring and creating an alerting policy is the most direct and managed GCP approach. Cloud Monitoring is designed for collecting, visualizing, and alerting on operational data from GCP services. It natively integrates with BigQuery, Dataflow, and Dataproc, allowing you to set up alerts based on predefined metrics (e.g., job failures, error rates) or custom log-based metrics. This leverages GCP's built-in observability features without requiring custom application development. Running a Compute Engine VM with Airflow is an orchestration tool, not primarily a monitoring and alerting solution for pipeline failures across multiple projects. Exporting logs to BigQuery and using App Engine to read them, or developing an App Engine app to poll logs via GCP APIs, involves custom development and management overhead that is unnecessary given Cloud Monitoring's capabilities.