You have on‑prem Spark jobs processing Parquet daily. Migrating to Google Cloud, you want managed services, minimal ETL changes, and BigQuery availability for future pipelines. What should you do?

This option leverages managed services while minimizing changes. Migrating Parquet data to Cloud Storage is straightforward. Dataproc Metastore (DPMS) provides a managed Hive Metastore, allowing existing Spark jobs to continue using their schema definitions without significant refactoring. Dataproc Serverless offers a fully managed, autoscaling Spark environment, aligning with the desire for managed services and eliminating infrastructure management. BigQuery can then easily query the data directly from Cloud Storage using external tables, fulfilling the requirement for future BigQuery availability. Registering the bucket as a Dataplex asset is useful for data governance but doesn't directly replace the need for a metastore for Spark jobs. Migrating data into BigQuery would require significant refactoring of Spark jobs, as they would need to interact with BigQuery's API instead of directly reading Parquet files, and it doesn't align with "minimal ETL changes." BigLake is a storage engine, not a replacement for a metastore, and running on Dataproc on Compute Engine is less managed than Dataproc Serverless.