You need near-real-time cleansing of ads data to identify long-tail and outlier points before feeding AI models and analytics. What should you use?

Dataflow is ideal for near-real-time, stream processing tasks like data cleansing and identifying outliers programmatically. Its autoscaling capabilities handle varying data volumes efficiently. BigQuery serves as an excellent sink for the cleansed data, providing a scalable and performant data warehouse for subsequent AI model training and analytics. Using Cloud Storage with shell scripts is not suitable for near-real-time processing. BigQuery alone is not designed for programmatic, real-time data cleansing and transformation before ingestion. Cloud Composer is an orchestration service, not a data processing engine, making it inappropriate for the core cleansing task.