Amazon DEA-C01: Data Transformation and Processing — Study Guide

By Laurynas Brazys · 2 August 2026 · #amazon #dea-c01 #certification #exam-prep #study-guide

Part of the Amazon Data Engineer Associate DEA-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

This domain covers the AWS services and patterns used to clean, transform, and prepare data for analytics and ML at scale. It focuses on selecting the right compute and tooling (Glue, Lambda, EMR, DataBrew) based on data volume, latency requirements, and cost constraints. Understanding service limits, job configuration knobs, and how data formats and catalogs interact is critical to build reliable, incremental pipelines.

AWS Glue ETL jobs (Spark and Python shell)

Glue Spark jobs are the primary choice for large-scale, distributed ETL: they run on AWS Glue-managed Apache Spark, use GlueContext, and operate on DynamicFrame and Spark DataFrame types. Configure via the console or CLI with job type “glueetl”, workerType (G.1X, G.2X, G.4X), and NumberOfWorkers; start with CLI: aws glue start-job-run –job-name my-spark-job –arguments ‘–execution-class=STANDARD’. Use DynamicFrame APIs (create_dynamic_frame.from_options, apply_mapping) when you need schema-flexible transforms, built-in transforms (Relationalize, Unnest) and automatic handling of semi-structured data; convert to Spark DataFrame with dyf.toDF() when you need Spark SQL, higher-performance joins, or custom UDFs.

Glue Python shell jobs use job type “pythonshell” for lightweight scripting and control-plane tasks. They are single-DPU (1 DPU) with limited parallelism and are best for small file manipulations, metadata updates, or orchestration. Configure via aws glue create-job –name my-pyjob –command ‘{“Name”:“pythonshell”,“PythonVersion”:“3”}’ and start with aws glue start-job-run. Note that Python shell jobs have a 1 DPU limit — use Spark for large datasets.

Glue job bookmarks enable incremental processing by tracking previously processed S3 objects and partitions. Enable bookmarks in the job configuration or when starting runs: aws glue start-job-run –job-name my-job –job-bookmark-option job-bookmark-enable. Bookmarks work for S3-based sources using built-in connectors; JDBC sources do not support bookmarks by default and require custom watermarking or state storage.

Glue now supports ExecutionClass FLEX for cost-optimized, non-urgent jobs. Start with aws glue start-job-run –job-name my-job –execution-class FLEX to allow Glue to schedule the job at lower cost with relaxed start-time SLAs. Use FLEX for batch backfills and non-latency-sensitive workloads; use STANDARD for predictable latency.

Decision criteria — quick comparisons:

• DynamicFrame vs Spark DataFrame

  • Use DynamicFrame when ingesting semi-structured JSON/Parquet with schema drift, leveraging apply_mapping, relationalize, and glue transformations.
  • Use Spark DataFrame when you need Spark SQL performance, complex joins, window functions, and third-party Spark libraries.
  • Convert between them via DynamicFrame.fromDF(df, glueContext, “name”) and dyf.toDF().

• Glue Spark vs Python shell

  • Choose Spark for multi-node, distributed ETL on large datasets and when using Glue Catalog integration at scale.
  • Choose Python shell for small, quick tasks or orchestration steps that fit within 1 DPU.

AWS Lambda for lightweight transformations

Lambda is ideal for event-driven, low-latency, lightweight transformations directly triggered by S3, Kinesis, or EventBridge. Typical uses include file validation, metadata extraction, JSON to CSV conversion for small files, or stream processing of records. Configure memory and timeout with aws lambda update-function-configuration –function-name myFunc –memory-size 2048 –timeout 300. Provisioned concurrency can be set with aws lambda put-provisioned-concurrency-config to mitigate cold starts for latency-sensitive streaming pipelines.

Be mindful of Lambda limits for data processing: 15 minute maximum execution, 10 GB memory (10,240 MB), and only 512 MB of ephemeral /tmp storage. For larger file processing, chain processing (split files), stage to S3 and invoke a Glue or EMR job, or use multi-part processing with AWS Step Functions. Use environment variables for small config and IAM role policies that strictly grant the least privilege.

Decision criteria — when to pick Lambda:

• Use Lambda when per-invocation processing fits within the 15-minute, 10 GB memory, and 512 MB /tmp constraints and when sub-second to second latency is required.
• Avoid using Lambda for large, long-running, or heavy memory-bound transformations; instead, use Glue Spark or EMR.

Amazon EMR for large-scale processing

EMR is the go-to for customizable, large-scale big data processing (Spark, Hadoop, Presto, Flink) where cluster-level control, custom bootstrap actions, or specialized libraries are required. Create clusters via CLI with aws emr create-cluster and choose either –instance-groups or –instance-fleets. Instance fleets provide flexible instance type mixes and spot/ON demand combinations; instance groups are simpler, fixed-size groups.

EMR cluster patterns to consider:

• Transient clusters: start with –auto-terminate and submit steps so the cluster terminates when steps complete. Good for cost-control but ephemeral HDFS and local state will be lost at termination.
• Long-running clusters: do not auto-terminate; use for interactive workloads, persistent HDFS, or when many small jobs benefit from JVM warm-up. Persist critical data to S3 or EBS-backed Hadoop storage if cluster termination is expected.

Configuration examples:

• Instance group CLI: aws emr create-cluster –name Prod –release-label emr-6.6.0 –use-default-roles –instance-groups InstanceGroupType=MASTER,InstanceType=m5.xlarge,InstanceCount=1 InstanceGroupType=CORE,InstanceType=m5.xlarge,InstanceCount=4
• Instance fleet CLI uses –instance-fleets with OnDemand/Spot allocations and multiple instance types for resilience and cost optimization.

Use EMR when you need full control over Hadoop ecosystem components, custom bootstrap scripts, or persistent HDFS for intermediate data; otherwise, Glue Spark is simpler for managed Spark ETL that integrates with the Glue Data Catalog.

Glue DataBrew and visual transforms

Glue DataBrew is a visual, no-code/low-code tool for data profiling, cleansing, and transformation targeted at data analysts and engineers working interactively. Create a dataset from S3 or Glue Catalog, build a recipe of transformations in the console, preview on a sample, and run jobs to apply recipes at scale. Schedule DataBrew jobs or run them via CLI with aws databrew start-job-run –name my-databrew-job.

DataBrew is optimized for data preparation tasks like standardization, deduplication, type conversion, and column-level transforms with built-in functions. It integrates with Glue Catalog and writes outputs to S3. Choose DataBrew when business users need self-service cleaning and quick profiling; for heavy transformation logic, complex joins, or very large datasets, prefer Glue Spark or EMR.

Comparison of visual vs code-based transforms:

• Glue DataBrew
  • Pros: quick profiling, recipe-based, non-coders can build pipelines, integrated scheduling.
  • Cons: not suited for very large or highly complex distributed joins and custom libraries.
• Glue Spark / EMR
  • Pros: full programmatic control, handles massive datasets, supports third-party libraries.
  • Cons: requires developer skills and more configuration.

Common Pitfalls and Decision Criteria

• Assuming Glue job bookmarks work for JDBC sources — bookmarks only track S3 object/partition state; for JDBC incremental loads use watermark columns, change-data-capture, or store progress in DynamoDB/S3.
• Treating transient EMR clusters like stateful systems — transient clusters terminate after steps and lose HDFS; persist intermediate data to S3 or use EBS volumes for durable storage.
• Ignoring Lambda cold starts in streaming pipelines — cold starts increase latency; mitigate with provisioned concurrency for critical paths or use long-lived compute for strict low-latency requirements.
• Running large transforms in Glue Python shell — Python shell jobs are limited to 1 DPU; for large datasets use Glue Spark jobs with appropriate workerType/NumberOfWorkers.
• Misconfiguring EMR instance types: choose instance fleets for cost and flexibility with spot instances; use instance groups when you need predictable instance composition.
• Overusing Glue Flex for urgent workloads — FLEX reduces cost but can delay start times; use STANDARD for predictable start and execution times.

Practical Problem: Use-Case Scenario

AcmeRetail processes nightly clickstream Parquet files into a unified customer activity table and wants incremental processing to avoid reprocessing months of data while keeping costs low for non-urgent backfills.

1. Use an S3 partitioned layout (year=/month=/day=) and register dataset in the Glue Data Catalog.
2. Create a Glue Spark job (glueetl) that reads DynamicFrame.from_catalog for schema flexibility, apply mappings, convert to DataFrame for complex joins, and write partitioned Parquet back to S3.
3. Enable Glue job bookmarks (job-bookmark-enable) for nightly runs to process only new partitions; for JDBC enrichment sources, implement watermark columns persisted in DynamoDB to track max processed timestamp.
4. For routine nightly runs, use ExecutionClass=STANDARD; for non-urgent historical reprocessing, submit runs with –execution-class FLEX to save cost.
5. Monitor with CloudWatch metrics and set alarms for job failures; for very large scale or custom libraries, consider EMR transient clusters that write intermediate results to S3 and auto-terminate.

Rationale: This approach leverages Glue’s managed Spark for scalable transforms and DynamicFrames for semi-structured inputs, uses job bookmarks for S3 incremental processing, and uses FLEX to reduce cost for backfills—preserving cost, reliability, and operational simplicity.

← Data Cataloging and Metadata Management · All domains · Data Orchestration and Workflow Management →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Amazon →