Amazon DEA-C01: Data Ingestion and Collection — Study Guide

By Laurynas Brazys · 2 August 2026 · #amazon #dea-c01 #certification #exam-prep #study-guide

Part of the Amazon Data Engineer Associate DEA-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

This domain covers the patterns, AWS services, and operational details used to bring raw data into a data platform reliably and at scale. Data engineers must choose between batch and streaming entry points, ensure data cataloging and discoverability, and design for throughput, replayability, and failure modes. Key AWS building blocks are S3 and Glue for batch, Kinesis and Firehose for streaming, DMS for database migration and CDC, and API/Event-driven components (API Gateway, Lambda, SNS, SQS, S3 events) for ad-hoc and push-based ingestion.

Batch ingestion with AWS Glue and S3

Glue is the primary managed ETL and metadata solution for batch ingestion into S3 and your data catalog. Typical pattern: land raw files into S3 (separate raw/zone prefixes), run a Glue crawler to infer schema and populate the Glue Data Catalog, then run Glue ETL jobs (Spark) to transform, partition, convert to columnar formats (Parquet/ORC) and write optimized data back to S3. Configure crawlers with appropriate classifiers (built-in CSV/JSON/Parquet or custom grok/regex) and give the crawler an IAM role that has s3:GetObject/s3:ListBucket and glue:catalog permissions—missing those is a common operational fault.

When configuring Glue jobs and crawlers use these console/CLI patterns and toggles:

• Create crawler: aws glue create-crawler –name my-crawler –role GlueRole –database-name raw_db –targets ‘{“S3Targets”:[{“Path”:“s3://bucket/raw/”}]}’ and start with aws glue start-crawler –name my-crawler.
• Glue job: aws glue create-job –name etl-job –role GlueRole –command ‘{“Name”:“glueetl”,“ScriptLocation”:“s3://bucket/scripts/job.py”}’; enable job bookmarks to avoid reprocessing. Decision criteria for Glue vs alternatives:
• Use Glue when you want managed Spark ETL, schema discovery, and catalog integration with Athena/Redshift Spectrum.
• Use EMR when you need specialized cluster tuning, custom libraries, or long-running clusters.
• Use simple Lambda or Glue on-demand for lightweight transforms on small files.

Streaming ingestion with Kinesis Data Streams and Firehose

Kinesis Data Streams (KDS) is for real-time ingestion with replay, consumer control, and fine-grained scaling. A Kinesis shard provides 1 MB/sec or 1,000 records/sec write capacity and 2 MB/sec read capacity; use aws kinesis create-stream –stream-name my-stream –shard-count 4 and put data with aws kinesis put-record –stream-name my-stream –partition-key key –data fileb://payload. Partition keys determine shard assignment; low partition-key cardinality causes hot shards — avoid by increasing key entropy or suffixing with a hash. Scale shards using aws kinesis update-shard-count or enable On-Demand mode for automatic scaling.

Firehose is a delivery-stream service optimized for near-real-time delivery (S3, Redshift, OpenSearch, Splunk) with built-in buffering, compression and optional Lambda transformation. Configure buffering with BufferingHints: buffer_size (MB) and buffer_interval (seconds) to tune delivery latency vs cost; enable compression (GZIP, Snappy) and set a processing Lambda for record-level transforms. Key differences:

• Kinesis Data Streams:
  • Real-time, supports multiple consumers, replay of retained data, explicit shard management
  • Per-shard throughput (1MB/1k writes), must design partition keys
• Kinesis Data Firehose:
  • Managed delivery to destinations, automatic retry/backoff, no replay of delivered records
  • Supports buffering (size/time), compression, transform via Lambda, S3 staging for Redshift loads

Choose KDS when you need replay, strong consumer control, or multiple downstream consumers; choose Firehose when you need simple delivery and transformation into S3/Redshift/OpenSearch with minimal operational overhead.

Database migration and CDC with DMS

AWS DMS is used for homogeneous/heterogeneous migrations and continuous replication (CDC). Deploy a replication instance (aws dms create-replication-instance –replication-instance-class dms.r5.large –allocated-storage 100) sized for throughput, with sizing decisions driven by change rate, full-load volume, and task parallelism. DMS task types:

• full-load: copy existing data only
• cdc: stream ongoing changes
• full-load + cdc: initial load then continue streaming changes Configure endpoints with appropriate engine settings (JDBC/connection-string), enable supplemental logging or plugins on the source, and supply a JSON table-mapping to filter/include tables. For MySQL-based sources DMS CDC requires binary logging (binlog) enabled and appropriate binlog_format (ROW recommended) on the source; for PostgreSQL you must enable logical replication and a plugin like wal2json or use replication slots. Monitor tasks via CloudWatch metrics and task logs; tune batchApplyEnabled and maxFullLoadSubTasks for throughput.

Decision criteria between full-load and CDC: use full-load+CDC when you need minimal downtime migration; use CDC-only for ongoing replication after an initial load is completed by another mechanism. Always validate schema mapping and run test migrations on representative data volumes.

API-based and event-driven ingestion patterns

APIs and events are for push-based ingestion and orchestration. Common patterns:

• API Gateway -> Lambda -> Firehose/Kinesis: suitable when clients push JSON events. Use API Gateway throttling and Lambda concurrency controls to provide backpressure and enforce idempotency headers.
• S3 event notifications: configure bucket notifications to send object-created events to Lambda, SQS, or SNS via the console or aws s3api put-bucket-notification-configuration; use prefix/suffix filters to limit triggers. For fan-out, route S3 -> SNS topic -> multiple SQS queues/Lambda subscribers to deliver the same event to multiple consumers without coupling.
• SQS and SNS for durable, decoupled ingestion: SQS for pull-based worker processing with visibility timeout, SNS for push fan-out.

Operational concerns and CLI patterns:

• Use DLQs for Lambda/SQS failures; configure retry policy on SNS subscriptions.
• For high-throughput streaming from APIs, prefer batching into Kinesis or Firehose rather than synchronous downstream writes to avoid blocking API clients.

Common Pitfalls and Decision Criteria

• Confusing Kinesis Data Streams (replay-capable, shard-managed) with Firehose (managed delivery, no replay): pick KDS when you need replay or multiple consumers; pick Firehose for straightforward delivery pipelines.
• Forgetting Glue crawler IAM permissions: always attach an IAM role that grants s3:GetObject/s3:ListBucket and glue:CreateTable/UpdateTable/DeleteTable for crawlers to populate the Data Catalog.
• Missing binary logging/logical replication for DMS CDC: enable binlog on MySQL (ROW format) or logical replication and wal2json on PostgreSQL before starting CDC tasks.
• Low partition-key cardinality causing hot shards: increase partition key cardinality via hashing, include high-cardinality attributes, or increase shard count; monitor Put/Get throttling metrics.
• Overbuffering Firehose or misconfigured buffering leading to high latency: tune buffer_size and buffer_interval based on acceptable latency and request volume.
• Relying on S3 event notifications without DLQ or retry: use SNS/SQS fan-out or Lambda with DLQ to avoid missed events and ensure durable fan-out.

Practical Problem: Use-Case Scenario

RetailCo collects mobile clickstreams (high-volume real-time) and nightly product catalog files; they need real-time dashboards and a consolidated analytics lake.

1. Ingest clickstreams into Kinesis Data Streams with partition keys derived from user session + hashed shard suffix; create consumers using Kinesis Data Analytics or Lambda/Kinesis Client Library for real-time processing.
2. Use Kinesis Data Firehose with a transformation Lambda to persist enriched streaming outputs to S3 (Parquet), compress with Snappy, and optionally load to Redshift Spectrum for analytics.
3. Place nightly catalog files in S3 raw/ and run a scheduled Glue crawler to update the Glue Data Catalog, then run Glue ETL jobs to convert to partitioned Parquet in the curated zone.
4. Use S3 event notifications -> SNS -> Lambda to trigger lightweight metadata updates or invalidate caches; route delivery to SQS for durable downstream processing.
5. Monitor Kinesis shard metrics (IncomingBytes, IncomingRecords, PutRecords.Success) and use UpdateShardCount or On-Demand streams to handle growth; enable CloudWatch alarms.

AWS best practice rationale: separate real-time and batch paths, use Kinesis Data Streams when replay and consumer isolation are required, use Firehose for managed delivery into S3/destinations, and maintain a Glue Data Catalog for discovery and query integration with Athena/Redshift.

All domains · Data Storage and Lake Architecture →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Amazon →