Amazon DEA-C01: Data Pipeline Monitoring and Troubleshooting — Study Guide

By Laurynas Brazys · 2 August 2026 · #amazon #dea-c01 #certification #exam-prep #study-guide

Part of the Amazon Data Engineer Associate DEA-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

Data pipeline monitoring and troubleshooting is critical to ensure timely, accurate delivery of streaming and batch data across AWS. This domain covers the telemetry, alerting, and diagnostic techniques for services like Kinesis, Firehose, Glue, DMS, Lambda, and the AWS audit trails that support incident investigation. Effective monitoring reduces Mean Time To Detect/Recover by exposing consumer lag, job resource pressure, delivery latency, and unauthorized access. The following sections give concrete signals, CLI/console patterns, and decision criteria to operate and remediate production data flows.

CloudWatch metrics and alarms for data services

CloudWatch is the primary telemetry plane: create metric filters, dashboards, and alarms for key service metrics and integrate alarms with SNS, EventBridge, or Systems Manager for automated remediation. Use aws cloudwatch put-metric-alarm to create alarms programmatically; typical flags include –metric-name, –namespace, –statistic (or –extended-stat), –threshold, –evaluation-periods, and –comparison-operator. For dashboards, push custom metrics (e.g., from Glue job metadata) using aws cloudwatch put-metric-data with a namespace like “MyCompany/DataPipeline”.

Focus on these actionable metrics and patterns:

• Glue: monitor BytesRead, BytesWritten, RecordsProcessed, and DPUHrs to detect data volume changes, skew, and cost. Alarms: sudden drop in RecordsProcessed or spikes in DPUHrs per record.
• Kinesis: monitor GetRecords.IteratorAgeMilliseconds for consumer lag and IncomingBytes/IncomingRecords for source pressure.
• Firehose: monitor DeliveryToS3.DataFreshness and DeliveryToS3.Records to spot delivery latency and data loss.
• DMS: monitor FullLoadRows, CDCLatencyMilliseconds, and AppliedChanges for replication health.

Decision criteria for alerting:

• Use composite alarms (CloudWatch composite alarms) to reduce noise: combine IteratorAgeMilliseconds > X for 3 datapoints AND consumer error rate > Y.
• For threshold selection, derive baselines from 7–14 day historical data and set dynamic thresholds using anomaly detection models (PutAnomalyDetector) when workloads are seasonal.

Glue job monitoring and error handling

Glue emits metrics to CloudWatch and writes logs to /aws-glue/jobs/output (job run logs) and /aws-glue/jobs/error (errors). Use CloudWatch Logs Insights to query job runs: run queries via console or aws logs start-query with a query string such as fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20. Track BytesRead, BytesWritten, RecordsProcessed, and DPUHrs from the Glue job run metrics—DPUHrs correlates directly to cost and job parallelism.

Common Glue failure modes and remediation:

• OutOfMemory (OOM) or Executor lost: increase worker type/DPU count, switch to G.2X worker type for heavier memory, or optimize Spark partitioning (repartition/coalesce) and use pushdown predicates to reduce input volume.
• Data skew causing stragglers: use partition keys to rebalance, increase parallelism, or use Glue DynamicFrame’s split/resolve choices where appropriate.
• Job stalls or long startup: enable job bookmarks and monitor Glue JobMetrics for “TimeWaitingForResources” to identify capacity contention.

Decision trade-offs:

• Increase DPU when CPU/memory are the bottleneck and run-time predictability matters; prefer code optimizations (partitioning, caching only when needed) if costs must be controlled.
• Use Glue streaming for near-real-time transforms; use Glue ETL batch for complex Spark transformations and larger spot-friendly workloads.

Kinesis and Firehose monitoring

Kinesis Consumer Lag: rely on GetRecords.IteratorAgeMilliseconds to detect how far consumers are behind. If IteratorAgeMilliseconds is consistently high:

• Scale by increasing shard count (reshard/scale), or
• Improve consumer performance by batching, using enhanced fan-out (for per-consumer throughput up to 2 MB/sec) or Kinesis Client Library (KCL) v2 with improved checkpointing.

Use aws kinesis describe-stream to inspect shard count and aws cloudwatch get-metric-statistics for IteratorAgeMilliseconds. When comparing remediation options, consider:

• Adding shards: increases ingest and read throughput; requires resharding and rebalancing.
• Enhanced fan-out: avoids shared read throughput but increases cost per consumer.
• Consumer optimization: reduces need for extra shards and cost but requires engineering effort.

Firehose delivery metrics: DeliveryToS3.DataFreshness quantifies delivery latency; typical buffering_delay settings are 60–900 seconds and will hold records until either bufferSize or bufferInterval is reached. If DeliveryToS3.DataFreshness is high:

• Check Firehose buffer hints (BufferIntervalInSeconds, BufferSizeInMBs) in the console or via aws firehose describe-delivery-stream.
• Inspect CloudWatch Errors (DeliveryToS3.RecordsFailed) and S3 bucket permissions (KMS errors if encrypted).

Remember Firehose buffering semantics: the service intentionally delays up to the buffer interval; reduce buffer interval to lower latency at cost of more frequent S3 writes.

CloudTrail and data access auditing

CloudTrail provides API activity and, optionally, data events for S3 and Lambda which are not enabled by default. To capture object-level S3 events, explicitly enable data events on the CloudTrail via console or aws cloudtrail create-trail –include-global-service-events and add S3 data resources. Without enabling S3 data events you will not see GetObject/PutObject in CloudTrail, which is a common gap during investigations.

Use CloudTrail logs combined with CloudWatch Logs Insights to correlate operational metrics (e.g., Glue job logs) with access events. Query patterns:

• CloudWatch Logs Insights: filter @message like /GetObject/ | stats count() by userIdentity.principalId
• Use EventBridge rules to react to specific API calls (e.g., PutBucketAcl) and forward to SNS for rapid alerts.

DMS replication tasks also publish CloudWatch metrics: monitor FullLoadRows for initial copy completeness, CDCLatencyMilliseconds to detect replication lag, and AppliedChanges to ensure transactions are being applied on the target. Alarm on CDCLatencyMilliseconds exceeding business SLAs and low AppliedChanges after an increase in full load rows.

Common Pitfalls and Decision Criteria

• High IteratorAgeMilliseconds on Kinesis mistaken for source issues — correct approach: check consumer checkpointing and processing time; scale by adding shards or use enhanced fan-out only after profiling consumer CPU/IO.
• Glue job OOM errors handled by blindly increasing DPU — correct approach: profile Spark stages, optimize partitioning and data filtering; increase DPU or worker type only if resource limits are confirmed.
• Firehose buffering delay causing perceived data loss — correct approach: verify BufferIntervalInSeconds and BufferSizeInMBs; reduce interval for low-latency needs and accept higher write rates/cost.
• Assuming CloudTrail records S3 object reads by default — correct approach: enable S3 data events in CloudTrail to capture GetObject/PutObject for forensic audits.
• Missing DMS CDC lag alarms — correct approach: create CloudWatch alarms on CDCLatencyMilliseconds and compare AppliedChanges vs FullLoadRows; investigate network or transaction backlog when lag grows.
• Over-alerting on transient spikes — correct approach: use evaluation-periods, datapoint-to-alarm, or anomaly detection to reduce noise and use composite alarms for correlated conditions.

Practical Problem: Use-Case Scenario

Acme Analytics runs real-time clickstream ingestion via Kinesis, enriches events using Glue ETL jobs, persists stale batches via Firehose to S3, and replicates legacy DBs with DMS. They observe end-to-end delays: consumer lag on Kinesis, Glue job OOMs, and Firehose showing large DeliveryToS3.DataFreshness.

1. Profile Kinesis consumers: pull GetRecords.IteratorAgeMilliseconds metric, inspect consumer logs, and run a Kinesis enhanced fan-out vs shard-scaling cost analysis.
2. Examine Glue job CloudWatch metrics and Logs Insights for OOM stack traces; test repartitioning + pushdown predicate locally or in a smaller job; only then increase DPU/worker type if needed.
3. Inspect Firehose buffer settings (BufferIntervalInSeconds) and DeliveryToS3.DataFreshness; reduce buffer interval for critical SLOs and validate S3 write permissions/KMS.
4. Configure CloudWatch composite alarms combining IteratorAgeMilliseconds, Glue job error rate, and Firehose DataFreshness; deliver alerts to an on-call SNS topic and trigger a runbook via EventBridge.
5. Enable CloudTrail S3 data events and correlate GetObject/PutObject events to Glue job start times and DMS applied changes to detect unauthorized or delayed access.

This approach follows AWS best practices: monitor the right service metrics at the correct granularity, prefer targeted code and configuration fixes before scaling resources, and ensure audit-level logging is explicitly enabled to enable fast root-cause analysis and automated remediation.

← Data Security · All domains · Cost Optimization for Data Workloads →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Amazon →