Amazon DEA-C01: Data Quality, Validation, and Observability — Study Guide

By Laurynas Brazys · 2 August 2026 · #amazon #dea-c01 #certification #exam-prep #study-guide

Part of the Amazon Data Engineer Associate DEA-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

This domain covers the end-to-end practices and AWS services used to ensure data correctness, detect anomalies, and keep pipelines reliable. Strong validation and observability reduce downstream defects, meeting SLAs and enabling safe re-processing. Data engineers must combine Glue Data Quality, validation frameworks, CloudWatch anomaly detection, DLQs, and idempotent design to build robust pipelines.

AWS Glue Data Quality rules and evaluation

Glue Data Quality uses Data Quality Definition Language (DQDL) rulesets to define assertions about datasets (row counts, null thresholds, uniqueness, custom SQL checks). Create rulesets in the console or with the CLI (example pattern: aws glue create-data-quality-ruleset –name MyRuleset –rules file://dqdl.json). Rulesets can be attached to Glue ETL jobs or run independently through StartDataQualityRuleRecommendationRun / StartDataQualityRulesetEvaluationRun APIs to evaluate datasets stored in S3, catalog tables, or Glue DynamicFrames.

Key configuration details and decision criteria:

• DQDL structure: rules include ruleName, expression (DQDL or SQL), severity, and action on failure. Explicitly set the action on failure to FAIL the job when critical checks fail; otherwise Glue may log results without failing the run.
• Evaluation context: provide table or S3 path references and sampling options (full scan vs. sample) via job parameters or evaluation run inputs to balance cost vs. coverage.
• Output: rule evaluation writes results to Glue metrics and to Amazon S3 as JSON; use those artifacts for audit and automated remediation.

When to use Glue Data Quality vs. external frameworks:

• Use Glue DQDL for standard schema, completeness, and simple uniqueness rules that integrate natively into Glue jobs and lineage.
• Use Great Expectations (see next subdomain) when you need richer expectation libraries, more expressive checks, or shared expectation suites across multiple systems.

Data validation patterns in pipelines

Validation belongs at multiple touchpoints: ingress, transformation, and sink. Common patterns:

• Lightweight pre-ingest checks in Lambda/Kinesis producers for schema and basic value ranges to reject or route bad rows before the pipeline.
• Server-side validation in Glue ETL (Spark) jobs using DQDL rulesets and explicit validations in code. In Glue Studio, add a Data Quality transform that references a ruleset; in CLI, pass –arguments ‘{"–enable-glue-datacatalog":""}’ or include job parameters to trigger evaluation runs.
• Post-transform validation with Great Expectations integrated into Glue Python Shell or Glue Spark jobs. Deploy expectations to S3 (expectations/ directory) and load DataContext in the job: DataContext(root_dir="/tmp/ge") after syncing S3 expectations to the job environment.

Comparison of validation options:

• Glue DQDL
  • Pros: native integration, low operational overhead, writes results to Glue catalog/metrics
  • Cons: less expressive for complex logical expectations
• Great Expectations
  • Pros: rich expectations, data docs, integrated checkpoints, extensible backends
  • Cons: requires packaging and managing expectations artifacts in S3 and orchestration in Glue jobs
• Manual checks in code (Spark/DataFrame)
  • Pros: full flexibility, high performance for custom logic
  • Cons: higher maintenance, no standardized reporting without extra work

For streaming, validate in-flight records and on failure push to an SQS DLQ (configure RedrivePolicy with maxReceiveCount via AWS CLI or console) rather than discarding. For batch, generate a validation report artifact and fail or quarantine outputs based on policy.

Anomaly detection and data drift monitoring

Use CloudWatch anomaly detection for operational metrics (records processed, error rate, job duration). Create a detector with the CLI: aws cloudwatch put-anomaly-detector –namespace “Glue” –metric-name “JobRunTime” –statistic “Average” –single-metric-anomaly-detector ‘{“MetricName”:“JobRunTime”,“Namespace”:“Glue”,“Stat”:“Average”,“Dimensions”:[…]}’ and then create CloudWatch alarms referencing the anomaly detection band. For data-level drift (distribution shifts, null rate changes), schedule Glue DataBrew profile jobs (aws databrew create-profile-job) to compute statistics, histograms, and quantiles; store profiles in S3 as baselines.

Decision criteria for anomaly vs threshold alarms:

• Choose CloudWatch anomaly detection when metric patterns are seasonal or variable; it learns past behavior and reduces manual threshold tuning.
• Use static threshold alarms for binary conditions (e.g., job stuck > X hours) where predictability is high.

For automated drift detection:

• Schedule regular DataBrew profile jobs (daily/weekly depending on data velocity) to capture metric baselines (null %, cardinality, percentiles).
• Compare new profile outputs to baseline profiles with either Glue rulesets (custom SQL checks) or Great Expectations custom expectations that reference baseline statistics.
• Alert using SNS / EventBridge when drift crosses policy thresholds or when anomaly detectors flag unusual metric behavior.

SLA management and pipeline reliability

SLA management ties observability to remediation and reliability engineering. Instrument every pipeline with these baseline metrics: throughput (records/sec), latency (ingest→sink), error rate, job/runtime, and downstream counts. Use CloudWatch Metrics for Glue jobs (job run metrics), Kinesis/ Kafka consumer lag, and custom application metrics via PutMetricData.

Reliability patterns and configuration details:

• Dead-letter queues (SQS DLQ): for streaming consumers (Lambda, Kinesis consumers), configure a DLQ and set RedrivePolicy(maxReceiveCount). Use DLQ retention and a separate processing job to inspect and reprocess DLQ messages.
• Idempotent design: ensure sinks support idempotent writes — examples:
  • DynamoDB: use PutItem with conditional expressions or a composite key for idempotency token.
  • S3: write with atomic rename patterns or use content-based keys (hash of record) so replays overwrite rather than duplicate.
  • Redshift/Glue ETL: use staging + MERGE by key to dedupe after reprocessing.
• Checkpointing: enable Kinesis/DynamoDB connector checkpoints and manage consumer checkpoint frequency to balance reprocessing window and duplication risk.

Decision criteria for retry vs fail-fast:

• For transient failures (downstream throttling), implement retries with exponential backoff and a DLQ only after max retries.
• For data-quality failures (schema mismatch), fail fast and write offending records to a quarantine S3 prefix with metadata for manual review.

Common Pitfalls and Decision Criteria

• Glue Data Quality rules log failures but do not fail the job by default — configure the rule action to FAIL for critical checks and attach the ruleset evaluation to the job run.
• Streaming pipelines without DLQs discard or lose failed records — always configure SQS DLQs (or persistent S3 staging) and a redrive policy to inspect and reprocess.
• Re-processing without idempotency produces duplicate records — design deterministic keys, use upsert/merge semantics at the sink, or apply content-based object keys for S3.
• Data drift detection without baselines produces noisy alerts — schedule Glue DataBrew profile jobs to create and store baseline statistics, then compare new profiles against them.
• Over-reliance on static CloudWatch thresholds causes false positives — use CloudWatch anomaly detection for seasonal/variable metrics and reserve static thresholds for non-negotiable limits.
• Setting maxReceiveCount too high defers DLQ routing and increases processing latency — choose a sensible maxReceiveCount so DLQs receive persistently failing messages in a timely manner.

Practical Problem: Use-Case Scenario

Streamline Retail faces frequent downstream reporting errors after nightly ETL: occasional schema spikes, silent rule violations, and duplicate orders when reprocessing failed runs.

1. Implement Glue Data Quality rules (DQDL) for schema, null thresholds, and uniqueness on order_id; set action on failure to FAIL the job and publish evaluation artifacts to S3.
2. Add Great Expectations in a Glue Python Shell step for complex business checks (order consistency across tables); store expectations in S3 and run checkpoints in the pipeline.
3. Schedule DataBrew profile jobs to capture daily baselines (cardinality, null rate, percentiles) and use automated comparisons to detect drift.
4. For streaming order events, configure SQS DLQ with an appropriate RedrivePolicy and create a replay job to process DLQ messages idempotently (using order_id as the dedupe key).
5. Instrument CloudWatch anomaly detectors for job runtime and error count; attach anomaly-based alarms to SNS for on-call escalation.

AWS best-practice rationale: combine native Glue quality controls for fast integration, Great Expectations for expressiveness, DataBrew for baseline statistics, and CloudWatch anomaly detectors for adaptive monitoring. DLQs and idempotent sinks close the loop for safe retries and re-processing while preserving SLAs.

← Cost Optimization for Data Workloads · All domains

Practice these questions → · Timed practice on ExamRoll.io →

Browse Amazon →