Amazon DEA-C01: Data Orchestration and Workflow Management — Study Guide

By Laurynas Brazys · 2 August 2026 · #amazon #dea-c01 #certification #exam-prep #study-guide

Part of the Amazon Data Engineer Associate DEA-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

Orchestration and workflow management are central to building reliable, maintainable data platforms: they coordinate extract-transform-load jobs, manage dependencies, handle failures, and integrate event-driven processes. This domain covers managed AWS options for batch ETL, complex DAGs, serverless state machines, and event scheduling — each with different execution semantics, durability, and scaling tradeoffs. Understanding when to use AWS Glue Workflows, MWAA, Step Functions, or EventBridge Scheduler — and how to configure error handling and observability — is critical for predictable pipelines and operational cost control.

AWS Glue Workflows and triggers

AWS Glue Workflows group Glue jobs, crawlers, and triggers into a dependency graph and let you run coordinated ETL. Create workflows via the console or CLI (aws glue create-workflow –name MyWorkflow). Triggers attach to workflows and come in three types: scheduled, on-demand, and conditional. Example CLI create for a scheduled trigger:

• aws glue create-trigger –name hourly-trigger –workflow-name MyWorkflow –type SCHEDULED –schedule “cron(0 * * * ? *)” –actions ‘[{“JobName”:“etl-job”}]’

Conditional triggers use a Predicate that references job name and state (SUCCEEDED, FAILED). Example predicate JSON: {“Logical”:“AND”,“Conditions”:[{“JobName”:“prev-job”,“State”:“SUCCEEDED”}]}. By default Glue conditional triggers fire on success; to handle failures configure conditions with State=FAILED or create an explicit FAILED trigger to route errors to remediation jobs or SNS alerts.

Operational patterns and decision criteria:

• Use Glue Workflows when you need native Glue job/crawler orchestration and lineage; choose triggers for cron scheduling or chaining jobs on completion.
• For ad-hoc invocation use aws glue start-workflow-run –name MyWorkflow or start-trigger for on-demand triggers.
• For complex branching or non-Glue tasks, prefer Step Functions or MWAA; Glue workflows are best when the pipeline is Glue-centric.

Error handling: add FAILED triggers, emit CloudWatch metrics for job success/failure, and push failures to an SQS/SNS dead-letter queue via Lambda for automated retries and investigation.

Amazon MWAA (Managed Airflow) for complex DAGs

MWAA provides a managed Apache Airflow environment to express complex DAGs, task dependencies, sensors, and custom operators. Create environments with aws mwaa create-environment –name MyEnv –airflow-configuration-options Key=core.executor,Value=CeleryExecutor and supply a DAGs S3 path and execution role. Important sizing and networking details:

• MWAA requires a VPC with private subnets and a NAT gateway for internet access; public-subnet-only setups are not supported.
• Worker and scheduler behavior is controlled through Airflow configuration options provided at environment creation (AirflowConfigurationOptions). Tune celery.worker_concurrency, celery.worker_autoscale, and scheduler settings to match task concurrency and DAG complexity.
• Monitor CloudWatch metrics (SchedulerHeartbeat, TasksFailed, TasksRunning, QueuedTasks) and scale worker autoscales or increase max workers when seeing queue growth.

Decision criteria:

• Use MWAA when you need Airflow features: complex DAGs, rich operators, cross-DAG dependencies, SLA/missed-task sensors, and custom Python logic.
• If tasks are short-lived and extremely high-throughput, prefer serverless Step Functions Express or Glue for managed ETL operations.
• Keep heavy, long-running tasks in managed compute (Glue/EMR/EKS) and use MWAA tasks as orchestration only — avoid running massive data transformations on the MWAA workers themselves.

Error handling in Airflow: use task retries and retry_delay in DAG definitions, set on_failure_callback to notify or push to an SQS dead-letter queue, and configure task-level SLA handling to trigger remediation DAGs.

AWS Step Functions for serverless orchestration

Step Functions provide stateful orchestration with a JSON-based Amazon States Language and integrate widely with AWS services. Choose between Standard and Express workflows:

• Standard Workflows: designed for long-running, durable state machines (months to years), with exactly-once (unique) execution semantics, built-in execution history, and per-execution tracing/logging. Start with aws stepfunctions start-execution –state-machine-arn arn:… –input ‘{“key”:“value”}’.
• Express Workflows: optimized for high-throughput, low-latency, short-duration processing and are cost-effective at scale; they use at-least-once execution semantics, so tasks must be idempotent or use deduplication patterns.

Use cases and decision criteria:

• Use Standard when you need durable, auditable workflows that may run for long periods and require once-only semantics.
• Use Express for event-driven micro-orchestrations with thousands of executions per second where short duration and cost efficiency matter, and you can design idempotent tasks or dedupe downstream.

Error handling and integration patterns:

• Use Retry blocks in ASL to define retries with ErrorEquals, IntervalSeconds, BackoffRate, and MaxAttempts.
• Use Catch blocks to redirect failures to alternate branches or to a Fail/Success state and to populate ResultPath with error details for diagnostics.
• For asynchronous dead-lettering, push failed messages to SQS/SNS or design a Step Functions pattern that sends error payloads to an SQS DLQ for offline processing. Enable CloudWatch Logs and X-Ray tracing via LoggingConfiguration and TracingConfiguration for observability.

EventBridge Scheduler and event-driven pipelines

EventBridge provides rich event routing and a Scheduler feature for cron and one-off tasks. Create schedule-based rules with aws events put-rule –name dailyRule –schedule-expression “cron(0 2 * * ? *)” and attach targets via aws events put-targets. For event-driven (pattern) routing, use put-rule with –event-pattern ‘{“source”:[“aws.s3”],“detail-type”:[“Object Created”]}’ to route S3 events to Lambda, Step Functions, or SQS.

Key operational points:

• EventBridge supports schedule expressions (cron and rate). Be aware of the 5-minute minimum interval for EventBridge rules when using rate expressions; for finer granularity consider Step Functions or a polling layer.
• Use EventBridge Scheduler for one-off, ad-hoc future invocations and recurring schedules; Scheduler supports time zones and flexible retry settings per target and can configure a dead-letter SQS queue for undeliverable invocations.
• For high-reliability pipelines, attach targets like Step Functions, Lambda, or SQS and configure per-target retry policies and DLQs. For example, put-targets accepts a DeadLetterConfig with Arn of an SQS queue.

Error handling: configure target-specific retry attempts and backoff, use DLQ for failed deliveries, and combine EventBridge with Step Functions for complex error-handling and compensating transactions.

Common Pitfalls and Decision Criteria

• Mistake: Using Express Workflows for non-idempotent tasks. Correct approach: design idempotency (dedupe keys, idempotent Lambda) or use Standard workflows for exactly-once semantics.
• Mistake: Assuming Glue conditional triggers fire on failure. Correct approach: explicitly create FAILED triggers or include State=FAILED in the trigger Predicate to route errors.
• Mistake: Deploying MWAA in public subnets or without NAT. Correct approach: place MWAA in private subnets and provide a NAT gateway or VPC endpoints for required service access.
• Mistake: Expecting sub-minute EventBridge schedules. Correct approach: remember EventBridge rules have a 5-minute minimum interval; use Step Functions or Lambda timers for sub-5-minute needs.
• Mistake: No centralized retry/catch strategy across services. Correct approach: standardize retry/backoff (ASL Retry, EventBridge retry config, Airflow retries) and use DLQs to preserve failed events for manual/automated remediation.
• Mistake: Overloading MWAA workers with heavy data processing. Correct approach: orchestration only on MWAA, run heavy transforms on Glue/EMR/EKS and pass pointers (S3 paths) between tasks.

Practical Problem: Acme Retail hourly ETL with spikes

Acme Retail needs an hourly ETL that runs Glue jobs for raw ingestion, a complex enrichment DAG with Python operators, and a short-lived SKU aggregation that must respond to high-frequency inventory events. They require robust retries and failure capture.

1. Use EventBridge to trigger an hourly scheduled rule that invokes a Step Functions Standard workflow to coordinate the overall pipeline.
2. In Step Functions, orchestrate long-running Glue jobs (StartJobRun) with Retry and Catch handlers; on failure route to an SQS DLQ and a remediation Lambda via a Catch block.
3. Deploy the complex enrichment DAGs in MWAA and invoke them from Step Functions using the Airflow REST API or by placing DAG run messages on SQS; size MWAA workers via celery.worker_autoscale settings based on expected concurrency and monitor CloudWatch metrics to adjust.
4. For high-frequency inventory events, use EventBridge event-pattern rules to push to an Express Step Function or Lambda with idempotency keys and an SQS-backed DLQ to absorb bursts.
5. Implement centralized monitoring (CloudWatch Logs/Metrics, X-Ray for Step Functions), and set alerts on DLQ growth and task retry exhaustion.

Rationale: This design uses the right tool for each requirement — Step Functions for durable cross-service orchestration and error handling, MWAA for complex DAG logic, Glue for managed ETL, and EventBridge for scheduling and reactive events. It enforces idempotency and DLQs for resilient, observable pipelines aligned with AWS best practices.

← Data Transformation and Processing · All domains · Data Query and Analytics →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Amazon →