An ML engineer must merge and transform weekly data from two sources: large CSV files in S3 (each with millions of rows) and an Amazon Aurora cluster. The merged output must be written to another S3 bucket. Which option provides this with the LEAST operational overhead?

AWS Glue is a fully managed extract, transform, and load (ETL) service that natively supports Apache Spark. It automatically provisions and de-provisions resources, minimizing operational overhead. Glue Data Catalog can discover schemas from both S3 CSV files and Amazon Aurora, and DynamicFrames simplify data manipulation. This makes it ideal for scheduled, recurring data transformations from diverse sources. Provisioning a transient Amazon EMR cluster requires managing cluster spin-up and spin-down, increasing operational overhead. AWS Lambda has execution time and memory limits, making it unsuitable for large-scale Spark jobs. AWS Batch on EC2 instances requires manual management of EC2 instances and Spark environment setup, leading to higher operational overhead compared to Glue.