Amazon DOP-C02: Event-Driven Architectures and Automation — Study Guide
Part of the AWS DevOps Engineer Professional DOP-C02 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.
Overview
Event-driven architectures decouple producers from consumers, emphasize asynchronous communication, and make systems resilient to spikes and failures. The core tenets are loose coupling via events/messages, consumer-driven scaling, idempotent handlers, and explicit failure handling and observability. AWS provides building blocks for durable queues, pub/sub, event buses, stream processing, orchestration, and operational automation. Mastering the interplay among Amazon SQS, Amazon SNS, Amazon EventBridge, AWS Step Functions, Lambda event source mappings, AWS Systems Manager Automation and OpsCenter, plus Kinesis Data Streams and Firehose, lets you build scalable, fault-tolerant, and auditable automation and data pipelines.
Messaging and Ingestion: SQS, SNS, Kinesis, and Firehose
Amazon SQS provides durable, scalable queues for decoupling. Standard queues offer at-least-once delivery and best-effort ordering with virtually unlimited throughput; they suit parallel workers that tolerate duplicate and out-of-order messages through idempotence keys and conditional writes. FIFO queues enforce ordered delivery per message group and exactly-once processing semantics with de-duplication (within a five-minute window). They guarantee order and single processing but trade absolute throughput: use multiple message groups to parallelize within FIFO, or enable high-throughput FIFO for thousands of messages per second. Configure visibility timeout per queue or per message to exceed the maximum processing time; with Lambda consumers, set visibility timeout to at least six times your function timeout to allow retries before a message reappears. Use long polling to reduce empty receives. Dead-letter queues (DLQs) capture poison messages when a message exceeds maxReceiveCount; later redrive from the DLQ to a source queue for reprocessing with fixes. Monitor ApproximateAgeOfOldestMessage to detect backlogs and drive concurrency/autoscaling changes.
Amazon SNS provides managed, high-throughput pub/sub. Publishers push once to a topic; SNS fans out to multiple subscriptions (SQS, Lambda, HTTP/S, email, mobile). Subscription filter policies use message attributes to route only relevant notifications per subscriber, shrinking downstream cost and load; express predicates such as exact match, prefix, numeric ranges, anything-but, and exists. Use SNS for fan-out, decoupled notifications, and mobile push. Enable retries and consider per-subscription DLQs for undeliverable deliveries. For ordered fan-out requirements, SNS FIFO topics with FIFO SQS subscriptions enforce ordering and de-duplication.
Kinesis Data Streams provides ordered, low-latency streams with shard-level parallelism for real-time analytics and event processing. Producers write records with partition keys to shards; consumers (Lambda, KCL, Enhanced Fan-Out consumers, Kinesis Data Analytics) read in order per shard with checkpointing. Use on-demand capacity for unpredictable load or provisioned shards with resharding for predictable throughput. Tune partition keys to balance hot shards and monitor IteratorAge for consumer lag. Enhanced Fan-Out offers dedicated 2 MB/s per consumer stream throughput with low-latency push over HTTP/2.
Kinesis Data Firehose is a fully managed delivery service for near-real-time ingestion into S3, Amazon OpenSearch Service, Splunk, or HTTP endpoints, with optional Lambda transformation, buffering (size/time), compression, and encryption. It sources from direct PutRecord/PutRecordBatch, Kinesis Data Streams, or CloudWatch Logs/Events subscriptions. Use Firehose when you need managed delivery with transformation and batching and do not need to build and operate consumer code. Dynamic partitioning lets you route records to S3 prefixes by keys for efficient downstream processing.
Routing, Orchestration, and Scheduling: EventBridge and Step Functions
Amazon EventBridge is the central event bus for routing, governance, and cross-account/event-domain integration. Use the default bus for AWS service events, custom buses to segment domains and apply permissions, and partner buses for SaaS integrations. Rules match events via content-based patterns and target 200+ AWS services. Use input transformers to shape payloads and archive/replay to reprocess historical events during recovery or new consumer onboarding. Resource policies enable cross-account event routing to consolidate governance in a platform account. EventBridge Pipes provide point-to-point, configurable flows from event sources (SQS, Kinesis, DynamoDB Streams, self-managed Apache Kafka on Amazon MSK, and others) to targets with built-in filtering, batching, transformation, and enrichment through Lambda or Step Functions—ideal when you do not want to manage a full consumer application but need lightweight mediation. EventBridge Scheduler provides one-time and cron schedules to invoke targets with an execution role, time zone support, and optional flexible time windows to reduce thundering herds.
AWS Step Functions orchestrates distributed workflows defined in Amazon States Language with states for Task, Choice, Parallel, Map (including distributed Map), Wait, Pass, Succeed/Fail, and robust Retry/Catch patterns. Deep service integrations remove the need for glue code to call AWS APIs, including synchronous (.sync) and callback patterns with task tokens. Choose Standard Workflows for long-running, audit-heavy orchestration with exactly-once state progression, up to one year duration, and per-state-transition pricing; execution history is retained, with strong visibility and built-in X-Ray traces. Choose Express Workflows for high-throughput, short-lived orchestrations (seconds to minutes) where you can trade per-request+duration pricing and at-least-once execution semantics for massive scale; use for streaming ingest enrichments, event routers, and micro-orchestrations where you can make tasks idempotent. Apply patterns like saga with compensating tasks and centralized error handling; externalize retries/timeouts in the state machine to simplify task code.
Compute Triggers and Backpressure: Lambda Event Source Mappings
Lambda event source mappings (ESMs) connect poll-based sources to Lambda and control concurrency, batching, and error handling.
SQS: Lambda scales horizontally by polling the queue and invoking your function with batches (up to 10 messages; max batching window up to 300 seconds). With Standard queues, scaling is aggressive with depth and message throughput; with FIFO, Lambda preserves per-message-group order and processes one batch per group at a time. Configure maximum concurrency on the ESM to cap worker scale and protect downstream systems; combine with reserved/provisioned concurrency to guarantee capacity. Use partial batch response to acknowledge only successful records and re-queue failed ones, avoiding replay of the entire batch. Set queue visibility timeout to exceed total worst-case retries. DLQs and redrive policies isolate poison messages.
Kinesis Data Streams: One concurrent Lambda invocation per shard by default ensures per-shard ordering. Increase ParallelizationFactor up to 10 to process multiple batches per shard concurrently where ordering across sub-sequences is acceptable. Batch size up to 10,000 records (6 MB) and maximum batching window up to 5 minutes let you amortize costs and increase throughput. Use bisect on function error to binary-search bad records within a batch and on-failure destinations or maximum retry/record age to discard or route unprocessable records. Monitor IteratorAge to detect consumer lag and reshard or increase parallelization accordingly.
DynamoDB Streams: Similar to Kinesis in semantics; batch size up to 1,000 records (6 MB) and shard-per-partition-key model. Consumers receive ordered item-level mutations (INSERT, MODIFY, REMOVE). Apply the same failure handling (bisect on error, maximum retry attempts, record age) and filtering. Use stream view type that includes the attributes your consumer needs (NewImage, OldImage, NewAndOldImages, or KeysOnly) to optimize payload size.
Event filtering in ESMs reduces invocations by dropping non-relevant events at the poller. Use tumbling window aggregation on streams with Lambda to aggregate records over time for mini-batch processing patterns.
Operations Automation and Remediation: Systems Manager Automation and OpsCenter
AWS Systems Manager Automation provides runbooks (documents of type Automation) written in JSON/YAML with steps such as aws:runCommand, aws:executeScript, aws:invokeLambda, aws:approve, aws:createStack, and aws:executeAutomation. Automations accept parameters, emit outputs, version with change history, and run with a dedicated AutomationAssumeRole for least privilege and cross-account/Region operations. Control concurrency and error thresholds across fleets, require approvals and Change Calendar windows, and integrate with notifications via SNS. Invoke automations on a schedule, from EventBridge rules (for near-real-time remediation on AWS Health, CloudWatch, or API events), from AWS Config remediations to enforce policy (for example, applying default tags to EBS volumes or attaching a default instance profile to EC2 instances), and from OpsCenter.
OpsCenter aggregates operational issues into OpsItems from CloudWatch alarms, AWS Config, Health events, or custom sources. Each OpsItem tracks status, priority, deduplication, related resources, and runbook links. Associate one-click runbooks for standard fixes and enable automatic remediation by wiring EventBridge or Config rules to start a specific Automation when an OpsItem is created or updated to a matching condition (for example, security group allowing 0.0.0.0/0 on SSH, drifted patch compliance, or failed backups). Use Systems Manager Explorer to visualize fleet health and open OpsItems across accounts and Regions. This pairing—OpsItems as durable records plus Automation runbooks as codified fixes—enables auditable, consistent operations at scale.
Design and Operational Guidance
Design for idempotency across all consumers because at-least-once delivery is common. Prefer event-driven fan-out via SNS or EventBridge rules for low-latency parallel reactions; prefer SQS to buffer workloads and shield producers from consumer slowness; prefer Kinesis when strict per-shard ordering and replayable streams are required for analytics. Use EventBridge Pipes for lightweight, managed integration between sources and targets when a bespoke consumer is overkill, and EventBridge Scheduler for time-based triggers without maintaining cron infrastructure.
Right-size timeouts and retries. For SQS, visibility timeout must exceed the maximum processing plus retries; for streams, cap retries and set MaximumRecordAgeInSeconds to avoid endlessly replaying bad records. Use DLQs or on-failure destinations systematically and add dashboards and alarms on SQS ApproximateAgeOfOldestMessage, Lambda ConcurrentExecutions/Throttles/Errors, IteratorAge, Step Functions ExecutionFailed/TimedOut, and EventBridge FailedInvocations. When spiky traffic is unavoidable yet latency SLAs are strict, use Lambda provisioned concurrency to pre-warm capacity. For governance, prefer EventBridge with resource policies for cross-account routing and archive/replay to support consumer evolution and incident recovery.
Practical Problem Scenario
Shopify needs to modernize order processing for flash-sale events while adding real-time analytics and automated remediation when downstream services slow or fail.
- Ingest and fan-out order events
- Use an SNS FIFO topic to publish OrderPlaced events from checkout, guaranteeing ordered, de-duplicated notifications per OrderId. Subscriptions include:
- SQS FIFO queue (Order-Workers) for fulfillment, preserving order.
- EventBridge custom event bus (CommerceBus) for governance and additional routing.
- Kinesis Data Firehose delivery stream for near-real-time delivery of Orders to S3 with GZIP compression for analytics. Why SNS FIFO: It provides ordered, exactly-once semantics with scalable fan-out to multiple subscribers without coupling publishers to consumers.
- Buffer and process fulfillment
- Lambda consumes from the SQS FIFO queue via an event source mapping with batch size 10, partial batch response enabled, and maximum concurrency capped to protect downstream warehouse APIs. Queue visibility timeout is set to six times the Lambda timeout to accommodate retries. A DLQ captures poison messages with maxReceiveCount=3; a redrive workflow later reprocesses fixed messages. Why SQS FIFO + Lambda ESM: Enforces per-order sequencing, isolates downstream slowness with buffering, and gives fine-grained error handling.
- Orchestrate multi-step saga
- A Step Functions Standard workflow orchestrates payment capture, inventory reservation, fraud screening, and shipment booking with retries, timeouts, and compensating tasks (refund, restock) on failure paths. The first Task is triggered by a Lambda invoked from the SQS consumer. Why Standard: Long-running, auditable, exactly-once state progression across external systems with rich error handling.
- Route domain events to capabilities
- The CommerceBus receives Order* events via PutEvents from producer services and from the SNS subscription. EventBridge rules:
- Match OrderPlaced to notify Marketing (Lambda) and create a support case (AWS Support API integration) when high-value customers order.
- Forward OrderFailed to a central operations account’s event bus using a resource policy for cross-account governance. Why EventBridge: Centralized routing, filtering, cross-account delivery, and the ability to add new consumers without changing producers.
- Pipe partner feed to enrichment
- EventBridge Pipes connects a partner’s SQS Standard queue (backordered SKUs) to a Step Functions Express workflow that enriches items via a Lambda function and pushes results to an internal SQS queue for restocking. Why Pipes + Express: Managed, low-overhead integration with lightweight enrichment at high throughput and low cost.
- Real-time analytics and search
- A Kinesis Data Stream collects clickstream and operational events. Lambda (Enhanced Fan-Out consumer) performs sessionization, and Kinesis Data Analytics aggregates KPIs. Firehose delivers transformed orders and analytics to S3 data lake and Amazon OpenSearch Service with dynamic partitioning by date/market for efficient queries. Why Streams + Firehose: Ordered, low-latency processing for analytics, with managed delivery and transformation to storage and search.
- Time-based automation
- EventBridge Scheduler runs a cron every minute to publish InventorySnapshotRequested to CommerceBus, triggering a Step Functions Express workflow that compiles snapshots across warehouses for near-real-time stock accuracy. Why Scheduler: Native, resilient cron without custom infrastructure.
- Automated remediation and operations
- AWS Config rules detect open SSH or public S3 ACLs in fulfillment VPCs. Managed remediations invoke Systems Manager Automation runbooks to correct drift. CloudWatch alarms on SQS ApproximateAgeOfOldestMessage and Lambda IteratorAge create OpsItems in OpsCenter; associated runbooks scale provisioned concurrency on specific Lambdas, increase Step Functions reserved capacity, or temporarily widen batch windows. An EventBridge rule on AWS Health EC2 maintenance events targets an SSM Automation document to restart impacted instances gracefully during maintenance windows. Why OpsCenter + Automation: Centralized, auditable issue tracking with one-click or automatic, least-privilege remediations that run safely across accounts/Regions.
This design sustains flash-sale spikes through buffering and fan-out, preserves business invariants via orchestration, delivers analytics within seconds, and closes the loop with policy-driven, automated remediation.
← High Availability · All domains · Storage →
Practice these questions → · Timed practice on ExamRoll.io →
Pass the whole exam — not just this question
You found this answer. Get every verified question and explanation in one place, and save hours of prep. Free to start.
Pass your exam →