Customer data is stored daily in compressed, date-partitioned files in S3. Analysts monthly download, validate, and upload results to QuickSight. The engineer must automatically check data quality before it’s sent to QuickSight with the LEAST operational overhead. Which solution fits best?

The correct answer leverages AWS Glue Data Quality (DQ) rules, which are purpose-built for this exact scenario, offering the least operational overhead. Glue DQ allows you to define rules directly on your Glue Data Catalog tables and automatically evaluate data quality, providing metrics and insights without needing to write custom code for common DQ checks. The Glue crawler ensures the Data Catalog is up-to-date with the new partitions. The option involving a Glue job with PySpark requires writing and maintaining custom code for data quality checks, increasing operational overhead compared to Glue DQ rules. Running Python scripts on Lambda for data quality is less suitable for large, partitioned datasets and would also require custom code, increasing complexity. Configuring S3 event notifications to SQS and using CloudWatch Insights for data quality is not a standard or efficient pattern for evaluating data quality within the data itself; CloudWatch Insights is for log analysis, not data content validation.