Amazon DOP-C02: Monitoring, Logging and Observability — Study Guide

By Laurynas Brazys · 4 August 2026 · #amazon #dop-c02 #certification #exam-prep #study-guide

Part of the AWS DevOps Engineer Professional DOP-C02 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

Overview

Monitoring, logging, and observability on AWS require combining metrics, logs, traces, events, and health telemetry into actionable signals. Effective architectures use Amazon CloudWatch for metrics, alarms, and dashboards; CloudWatch Logs and Logs Insights for log ingestion and analytics; AWS X-Ray for distributed tracing; AWS CloudTrail for auditing and integrity; Amazon EventBridge for event-driven detection and automation; AWS Health for account-specific service events; and centralized pipelines (Kinesis Data Firehose and OpenSearch) for search and correlation at scale. The patterns below emphasize noise reduction, precise signal routing, automation, and multi-account/multi-Region operations.

CloudWatch Metrics, Alarms, Dashboards, and Composite Alarms

CloudWatch metrics are the foundation for SLOs, scaling, and alerting. Publish custom metrics with fine-grained dimensions to isolate signals (for example, apiOperation, appVersion, statusCode). Use the CloudWatch Embedded Metric Format (EMF) with structured logs to emit high-cardinality dimensions efficiently from Lambda, containers, and EC2, avoiding PutMetricData API overhead.

Configure alarms with robust evaluation:

Choose periods aligned to data granularity and SLO windows.
Set datapointsToAlarm (m out of n) for resiliency against transient noise.
Use TreatMissingData to avoid false positives during deployment or pauses.
Leverage anomaly detection bands when baselines vary with seasonality, and metric math for derived indicators (p95 latency, error percentages, saturation ratios).
Attach actions to alarms: notify via SNS, create OpsCenter OpsItems, execute SSM Automation, or recover EC2 instances. Scaling policies can reference alarm states for action, but composite alarms cannot directly trigger scaling.

Composite alarms reduce alarm fatigue by combining multiple underlying alarms with AND/OR logic. For example, alert only when p95 latency is high AND 5xx rate exceeds threshold AND CPU saturation persists, thereby aligning to user impact. Composite alarms accept state updates from child alarms across Regions/accounts via cross-account observability or metric streams into a central account.

Dashboards visualize key indicators across services. Use widgets for metrics, Logs Insights query results, and Alarm Status. Standardize dashboard conventions (naming, time ranges, SLO overlays) and leverage cross-Region/cross-account views with CloudWatch Observability Access Manager (OAM). For ad hoc correlation, pin Logs Insights and X-Ray ServiceLens widgets side-by-side with service map widgets and Kinesis Firehose error rates.

CloudWatch Logs: Log Groups, Metric Filters, Subscription Filters, and Logs Insights

Structure log groups per application/component and lifecycle stage. Set explicit retention policies (do not rely on “Never Expire”) and enable KMS encryption where required. Use resource policies and fine-grained IAM to control producers and subscribers. For high-throughput ingestion, ensure adequate log stream concurrency and batching.

Metric filters turn log patterns into metrics. Define a filter pattern with extracted tokens (JSON or space-delimited) and map tokens to metric dimensions. This supports use cases such as per-API, per-version, per-response-code metrics published directly from logs without modifying producers. Ensure units and default values are correct; prefer 1 per event and derive rates via metric math. Use these metrics for SLO alerting and dashboards.

Subscription filters stream logs in near real time to:

Kinesis Data Firehose for transformation and delivery to S3/OpenSearch.
Kinesis Data Streams for custom consumers.
Lambda for custom routing, PII redaction, or event-driven notifications. Use a CloudWatch Logs destination with an IAM role for cross-account subscriptions. Plan for retry and backpressure; Lambda and Firehose provide built-in retries and DLQs/error S3 buckets, respectively.

CloudWatch Logs Insights provides interactive, serverless query over logs. Core operators include fields, filter, parse, stats, sort, limit, dedup, and bin for time bucketing. Parse JSON fields or use grok-like parsing for text logs. Examples:

filter status >= 500 | stats count() by apiOperation, appVersion
parse @message /duration=(?<ms>\d+)/ | stats pct(@ms,95) by service Save frequently used queries with QueryDefinition for team reuse, and embed them into dashboards as query widgets. For automation, schedule a Lambda via EventBridge to run StartQuery/GetQueryResults and publish summaries to SNS or OpsCenter. Restrict query scope to specific log groups and time windows to control cost.

AWS X-Ray: Tracing, Sampling Rules, Service Maps, and Annotations

X-Ray captures distributed traces across services to find latency contributors and fault boundaries. Instrument services with the AWS Distro for OpenTelemetry (ADOT) or X-Ray SDKs, propagate the trace header (e.g., X-Amzn-Trace-Id), and run the X-Ray daemon/agent where needed (ECS/EKS/EC2). Many managed services integrate natively (API Gateway, ALB via access logs proxying traces, Lambda with active tracing, Step Functions via subsegments).

Sampling rules control data volume and signal fidelity. Use a central sampling rule set with:

Fixed reservoir per second for baseline traces per service.
Rate-based sampling percentage to scale with throughput.
Rule priority and service/URL matching for hot paths and error scenarios. Increase sampling during incidents and for canary traffic to guard observability while managing cost.

Service maps visualize the call graph, showing edges with latency, error rates, and throttle indicators. Drill into traces to examine segments and subsegments for downstream dependencies. Use annotations (indexed key-value pairs) for high-cardinality filtering such as customerTier, apiOperation, appVersion, or AWS request IDs. Use metadata for verbose, non-indexed context to avoid index blowout. Combine X-Ray trace groups with CloudWatch ServiceLens to correlate logs, metrics, and traces in a single view. Create filter expressions (e.g., annotation.appVersion = “2.3.1” and fault = true) to isolate regressions and export trace IDs for targeted log search.

Governance and Events: CloudTrail, EventBridge, and AWS Health

CloudTrail records API activity for governance and forensic analysis. Enable an organization trail across all accounts and all Regions, deliver to a centralized S3 bucket with SSE-KMS, enable log file validation, and integrate with CloudWatch Logs for near-real-time detection. Distinguish event classes:

Management events: control plane (e.g., CreateUser, RunInstances). Configure to include read-only and write-only as needed.
Data events: high-volume data plane operations such as S3 object-level access, Lambda Invoke, DynamoDB item APIs, EKS API server calls. Scope data events selectively (by bucket/function/table) to control cost. Use CloudTrail Insights to detect unusual API spikes, and feed CloudTrail events into EventBridge for auto-remediation. Validate log integrity using digest files and the AWS CLI cloudtrail validate-logs command during audits.

EventBridge provides an event fabric for detection and automation. Use the default event bus for AWS service events and create custom buses for application-domain events. Define event patterns matching source, detail-type, detail fields, prefixes, numeric ranges, and “anything-but”. Apply input transformers to reshape events, attach resource-based policies for cross-account publishing, and configure retry/DLQ on targets. Common targets include Lambda (remediation), Step Functions (orchestration), SQS (decoupling), Systems Manager Automation (ops actions), CodePipeline (CI triggers), and SNS (notifications). Archive and replay events to recover from consumer outages, and use the schema registry to generate strongly typed event models.

AWS Health surfaces account-specific service events, scheduled changes, and operational issues. Integrate via EventBridge with source aws.health and detail-type AWS Health Event to route to incident channels, open OpsCenter OpsItems, or trigger safe shutdown/scale actions for maintenance windows. Use the Organizational View with a delegated admin account to aggregate Health events across all accounts, and consider AWS Health API or the AWS Health Aware solution to push curated notifications into on-call systems.

Centralized Logging with Kinesis Data Firehose and OpenSearch

A multi-account, multi-Region logging strategy standardizes ingestion and search. In each producer account, configure CloudWatch Logs subscription filters to a cross-account Logs destination backed by a central Kinesis Data Firehose. Enable Firehose features:

Data transformation via Lambda for normalization (JSON), PII redaction, and enriching with AWS account, Region, VPC, and service metadata.
Compression (GZIP) and dynamic partitioning when delivering to S3 to optimize query performance in Athena.
Encryption with KMS and VPC delivery for private endpoints. Deliver to Amazon OpenSearch Service for low-latency search and Kibana/OpenSearch Dashboards visualization. Use index templates, ILM/ISM policies for rollover and retention, and fine-grained access policies mapping users to index patterns (e.g., account/team/service). Configure error output to S3 for failed documents and monitor Firehose delivery and OpenSearch ingestion metrics (DeliveryToElasticsearch.Success, ElasticsearchFailedRequests). For very high volume, consider landing all logs in S3 via Firehose and using a subset streamed to OpenSearch, with on-demand Athena queries over S3 for long-tail investigations to control cost.

Combine this pipeline with CloudWatch metric filters for fast, low-cost counters and with Logs Insights for ad hoc deep queries. Use EventBridge rules triggered by Firehose/OpenSearch anomalies or CloudWatch alarms to kick off remediations or to raise incidents.

Practical Problem Scenario

Airbnb experiences intermittent spikes in API errors and latency across microservices deployed on EKS and Lambda, with multiple mobile app versions in the wild. Operations needs near-real-time detection by API operation, response code, and app version; rapid root cause analysis across traces and logs; automated remediation for known failure patterns; and governance-grade audit trails.

Standardize structured logging

Implement EMF-structured JSON logs in services (EKS, Lambda) including fields for apiOperation, statusCode, appVersion, tenantId, and latencyMs.
Why: EMF enables direct metric extraction in CloudWatch with low overhead and high-cardinality dimensions for precise alarms.

Create CloudWatch Logs metric filters

For each service log group, define metric filters that increment counters by apiOperation, statusCode, and appVersion.
Why: Produces per-dimension metrics without extra code paths, enabling dashboards and actionable alarms per API and client version.

Build layered CloudWatch alarms and a composite alarm

Alarm on p95 latency, 5xx rate, and saturation (CPU, memory, concurrency/throttle). Create a composite alarm: LatencyHigh AND ErrorsHigh for 2 of 3 consecutive periods.
Why: Reduces noise and focuses on user-impacting incidents.

Deploy X-Ray tracing with targeted sampling

Use ADOT collectors on EKS and active tracing for Lambda. Define sampling rules to capture all error traces and a representative sample of successful calls, with higher sampling on new app versions.
Why: Guarantees visibility into failures and enough coverage for performance hotspots while controlling cost.

Correlate with ServiceLens and Logs Insights

Create dashboards combining metric widgets, X-Ray service map, and Logs Insights queries (e.g., filter status >= 500 | stats count() by apiOperation, appVersion).
Why: Single-pane correlation accelerates diagnosis of which operation and client version regressed.

Centralize logs via Firehose to OpenSearch and S3

Configure subscription filters to a central Firehose with Lambda transform to normalize, redact PII, and enrich with account/Region. Deliver to OpenSearch for 7-day hot search and S3 for durable retention and Athena queries.
Why: Fast, cross-team search on current issues with low-cost historical analysis.

Automate detection and remediation with EventBridge

Create EventBridge rules for CloudWatch alarm state changes and selected CloudTrail write API events (e.g., security group modifications). Targets: Lambda for safe rollbacks (e.g., revert feature flags) and Step Functions for multi-step remediation.
Why: Event-driven control loops shorten MTTR and enforce guardrails.

Integrate AWS Health and maintenance handling

Add EventBridge rules for aws.health events affecting EC2, EKS, or networking. Target SSM Automation to cordon/drain nodes or shift traffic.
Why: Proactive mitigation of scheduled or operational issues reduces downtime.

Harden governance with CloudTrail org trail and integrity

Enable an organization, multi-Region trail with data events for S3 and Lambda, SSE-KMS encryption, and log file validation. Stream to CloudWatch Logs and OpenSearch for anomaly spotting and investigations.
Why: Complete, tamper-evident audit meets compliance and accelerates RCA.

Notifications and Ops integration

Route critical events to SNS and on-call systems, open OpsCenter OpsItems with runbooks attached, and attach alarm tags for ownership and severity.
Why: Clear ownership and automated runbooks improve response quality and speed.

This design was chosen to combine low-latency, dimension-rich metrics (CloudWatch + EMF), deep trace correlation (X-Ray + ServiceLens), search-at-scale (OpenSearch + S3/Athena), event-driven remediation (EventBridge + Lambda/SSM/Step Functions), and auditable governance (CloudTrail with integrity). It balances cost and fidelity with sampling, retention tiers, and targeted alarms that reflect real user impact.

← Infrastructure as Code and Configuration Management · All domains · Security →

Practice these questions → · Timed practice on ExamRoll.io →