New app produces ~150 GB/day JSON by year-end. Requirements: decouple producer/consumer, cost-efficient indefinite raw storage, near real-time SQL, and 2+ years of SQL-queryable history. Which ingestion and storage pipeline meets these needs?

The correct option leverages Pub/Sub for decoupling and real-time ingestion, Dataflow for scalable, managed ETL (converting JSON to Avro for efficiency), Cloud Storage for cost-effective indefinite raw storage, and BigQuery for near real-time SQL querying and historical analysis. This combination directly addresses all requirements. Providing an API and polling to Cloud Storage lacks real-time querying capabilities and requires custom polling logic. Writing directly to Cloud SQL is not scalable for 150 GB/day of raw JSON and periodic exports/loads introduce latency, failing the near real-time requirement. Publishing to Pub/Sub and using Spark on Dataproc with HDFS persistent disks introduces operational overhead for managing a Spark cluster and HDFS, which is less cost-efficient and managed than Dataflow and Cloud Storage/BigQuery for this use case.