A data scientist must migrate an on-premises ETL process to AWS. The current process runs on a schedule and uses PySpark to merge and format several large data sources into one consolidated output for downstream jobs. Requirements for the cloud solution: combine multiple sources, reuse existing PySpark code, run on the existing schedule, and minimize the number of servers to manage. Which architecture meets these requirements?
Choose an answer
Tap an option to check your answer.
Correct answer: Store the raw data in Amazon S3. Create an AWS Glue ETL job implemented in PySpark to reuse the existing logic. Configure an AWS Glue trigger to run on the current schedule and have the job write its processed output to a processed location in Amazon S3 for downstream use..
Why this is the answer
The correct answer is AWS Glue. AWS Glue is a serverless ETL service that natively supports PySpark, allowing the data scientist to reuse existing code without re-implementation. It can be scheduled to run at specific intervals, fulfilling the scheduling requirement. Being serverless, it minimizes operational overhead and the number of servers to manage. The option with a persistent Amazon EMR cluster is incorrect because while EMR supports PySpark and scheduling, a persistent cluster incurs ongoing costs and requires more management than a serverless solution like Glue, conflicting with the "minimize servers to manage" requirement. The Lambda function option is incorrect because Lambda has execution time limits and memory constraints, making it unsuitable for large-scale ETL jobs, and it would require re-implementing PySpark logic in standard Python, violating the "reuse existing PySpark code" requirement. Kinesis Data Analytics is incorrect because it's designed for real-time stream processing using SQL, not batch ETL with PySpark, and would require a complete re-architecture and re-implementation of the existing PySpark logic.
Pass your exam — without the endless answer hunt
Get every verified question and explanation for this exam in one place, and save hours of prep. 1,000+ certifications · 20+ languages · free to start.
Pass your exam faster → No card needed