You migrated an HDFS data lake to Cloud Storage. Data scientists need to run Apache Spark and SQL with column-level security, in a cost-effective solution that scales into a data mesh. What do you build?

The correct option leverages BigLake, which allows BigQuery to query data directly from Cloud Storage while enforcing granular security. BigLake tables provide a unified access layer over data lakes, supporting column-level security through Data Catalog policy tags. This approach is cost-effective because it avoids data ingestion into BigQuery for every query and scales well for a data mesh architecture by centralizing metadata and access control. The Spark-BigQuery connector allows Spark to interact with BigLake tables, and BigQuery SQL provides direct query capabilities. The incorrect options are less suitable: Deploying long-running Dataproc clusters with Apache Hive and Ranger is less cost-effective due to persistent cluster costs and requires more operational overhead for Ranger management. Loading data into native BigQuery tables is not ideal for a data lake scenario where data often resides in Cloud Storage, increasing storage costs and potentially duplicating data. Applying IAM at the Cloud Storage file level is too coarse-grained for column-level security and doesn't integrate seamlessly with SQL engines for fine-grained access control.