An aerospace system produces proprietary flight data that must be streamed into BigQuery efficiently while minimizing resource use. What ingestion approach should you use?

The correct approach is to use an Apache Beam custom connector to write a Dataflow pipeline that streams the data into BigQuery in Avro format. This method is efficient for streaming data, as Dataflow (based on Apache Beam) is designed for scalable, real-time data processing. Using a custom connector allows for handling proprietary data formats, and Avro is an excellent choice for BigQuery as it's a row-based, schema-rich format that BigQuery can ingest directly and efficiently, minimizing resource use. The other options are less suitable: A shell script triggering periodic Cloud Functions for batch ETL is not ideal for efficient streaming and introduces latency. Storing raw data and transforming later with a standard Dataflow pipeline might work, but it's less efficient than directly ingesting in an optimized format like Avro, especially for proprietary data that might need specific parsing. Using Apache Hive and Dataproc for streaming in CSV format is inefficient. CSV is not optimized for BigQuery ingestion compared to Avro, and Dataproc is generally better suited for batch processing of large datasets rather than real-time streaming.