Amazon DOP-C02: High Availability, Resilience and Disaster Recovery — Study Guide
Part of the AWS DevOps Engineer Professional DOP-C02 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.
Overview
High availability and disaster recovery on AWS focus on reducing downtime (RTO) and data loss (RPO) under component, Availability Zone (AZ), or Regional failures. Multi-AZ designs absorb AZ failure without data loss and minimal service impact; multi-Region designs address Regional disruptions and large-scale events. Selecting between active/active, active/passive (warm standby), and pilot-light strategies is driven by business RTO/RPO targets, consistency requirements, and cost. Achieving these objectives requires coherent design across DNS routing, compute elasticity, load balancing, database replication/failover, durable object storage with replication/versioning, centralized backup, and continuous resilience verification via fault injection.
Architectures for RTO/RPO and intelligent routing
Multi-AZ and multi-Region:
- Multi-AZ: Place redundant instances across at least two subnets in different AZs behind a load balancer. Use managed databases with synchronous replication (RDS Multi-AZ, Aurora Multi-AZ/cluster). RPO is typically zero for synchronous storage; RTO targets range from sub-minute (Aurora) to a few minutes (RDS Single-Instance Multi-AZ failover).
- Multi-Region: Choose active/active for the lowest RTO with regional isolation and low latency, or warm standby/pilot light for cost-optimized DR. Data replication must meet RPO: asynchronous DB replicas, Aurora Global Database (<1 s typical RPO), DynamoDB global tables (multi-Region, multi-active), and S3 Cross-Region Replication (CRR) with optional Replication Time Control (RTC) for SLA-backed replication.
Route 53 routing policies and health checks:
- Failover routing: Create two records for the same name: Primary and Secondary. Associate a health check with the Primary (or use “Evaluate Target Health” for alias to ALB/NLB). On failure, traffic shifts to the Secondary. Keep TTL low (e.g., 60 s) to reduce DNS cache lag, and monitor health-check status with CloudWatch Alarms.
- Latency-based routing: Route users to the Region with the lowest measured latency. Associate health checks with each record to ensure only healthy endpoints receive traffic. Pair with multi-Region stacks and regional data stores that support eventual/strong consistency as required.
- Weighted routing: Split traffic by percentage to support canary releases, A/B testing, or “trickle” DR readiness (for example, 1% to secondary continuously). Combine with health checks so unhealthy weights are excluded. Use gradually shifting weights to migrate traffic during a Regional evacuation.
- Health checks: Probe HTTP(S)/TCP endpoints or CloudWatch Alarms. For ALB/NLB aliases, enable “Evaluate Target Health” to inherit target group health. Design health endpoints to reflect true readiness (dependencies reachable, migrations applied). For stateful apps, include dependency checks (database, cache) to avoid routing to partially healthy instances.
Resilient patterns by objective:
- Low RPO, sub-minute RTO globally: Active/active with Route 53 latency-based routing and health checks, region-local stateless compute, DynamoDB global tables or Aurora Global Database, S3 CRR with RTC for critical objects.
- Moderate RPO (≤15 min), RTO ≤4 hours: Warm standby with scaled-down secondary, asynchronous DB replica (RDS cross-Region read replica or Aurora Global), Route 53 failover routing, runbooks or automation to scale up and promote on failover.
- Cost-optimized DR: Pilot light for core data services only, infrastructure-as-code to scale out app tier on invocation, RPO determined by replication frequency, RTO by provisioning time and data catch-up.
Elastic Load Balancing and Auto Scaling
Elastic Load Balancing:
- Application Load Balancer (ALB): Layer 7, host/path routing, WebSocket/HTTP/2, integrated WAF, stickiness via target group cookies, and request-based health checks. Use cross-zone load balancing and deregistration delay (connection draining) to gracefully drain targets during scale-in or deployments. Configure slow start and outlier detection for uneven backend warm-up.
- Network Load Balancer (NLB): Layer 4, ultra-low latency, static IPs/Elastic IPs, TLS pass-through/termination, preserves source IP, and supports long-lived connections. Use for TCP/UDP protocols, high-throughput workloads, or where client IP visibility is mandatory. Health checks are TCP/HTTP/HTTPS at Layer 4/7 as configured.
- Connection draining (deregistration delay): Set an appropriate delay (for example, 60–300 s) to allow in-flight requests to complete. Ensure deployment and Auto Scaling termination events honor this delay to avoid user interruption.
Auto Scaling groups:
- Scaling policies:
- Target tracking scaling: Maintain a metric (CPUUtilization, ALB RequestCountPerTarget) at a target value. This is the simplest and most adaptive policy for web/API fleets.
- Step scaling: Scale by defined steps when metrics cross thresholds. Useful for bursty traffic with predictable patterns.
- Scheduled scaling: Pre-scale for known events (sales, launches) to avoid cold capacity.
- Predictive scaling: Optionally forecast demand using ML for daily/weekly patterns.
- Lifecycle hooks: Launching:Wait and Terminating:Wait allow you to gate instance readiness and teardown. Use hooks to:
- Bootstrap instances (SSM Automation, user data completion, AMI hydration) before they enter service.
- Collect logs and artifacts prior to termination for root cause analysis.
- Coordinate blue/green or in-place deployments that must confirm readiness signals (e.g., cfn-signal).
- Warm pools: Keep instances pre-initialized in Stopped or Running state attached to the ASG to slash scale-out latency. Warm pools pair well with long bootstrap steps (large package install, model downloads). Configure minimum warmed capacity and reuse policies. Combine with lifecycle Launching:Wait to only complete the hook when app readiness checks pass, ensuring consistent cutover times.
- Resilience settings: Enable Capacity Rebalance for Spot, multiple instance types/allocations, health checks tied to target group health, and instance refresh for safe rolling updates with health guardrails.
Data layer resilience, replication, and backups
Relational databases:
- RDS Multi-AZ: Synchronous replication to a standby in a different AZ; automated failover updates the DNS endpoint to the standby. This protects against AZ and instance failure with RPO ≈ 0 and RTO typically minutes for Single-AZ DB instances with Multi-AZ standby. The newer Multi-AZ DB cluster for MySQL/PostgreSQL offers faster failover with multiple readable standbys.
- Read replicas: Asynchronous replication for read scaling and DR. Use cross-Region read replicas for DR; promotion is manual (or automated with runbooks/Serverless functions) and incurs RPO > 0. Ensure binlog or logical replication is correctly configured and that replication lag is monitored.
- Aurora: Aurora Multi-AZ (cluster) uses shared storage with replicas in multiple AZs; failover is generally sub-minute. Aurora Global Database provides storage-based physical replication to secondary Regions with typical RPO < 1 s and RTO < 1 min. Use managed planned failover for zero data loss migrations, or unplanned failover for disaster events. Writer/reader endpoints abstract topology; applications should implement retry with backoff.
Object storage and DR:
- S3 Versioning: Enable versioning to protect against overwrites/deletes and to support CRR. Configure lifecycle policies to transition older versions to cheaper storage and set appropriate retention.
- S3 Cross-Region Replication (CRR): Requires versioning on source and destination. Use a replication IAM role; if buckets are cross-account, add a destination bucket policy allowing the source role s3:ObjectOwnerOverrideToBucketOwner and put permissions. For KMS-encrypted objects, grant the role kms:Decrypt on the source key and kms:Encrypt on the destination key. Consider:
- Replication Time Control (RTC) for 99.9% of objects replicated within 15 minutes with metrics/alerts.
- Replication of delete markers and ownership controls as needed.
- S3 Batch Replication for existing objects.
- Replication metrics and notifications for SLAs.
- DynamoDB: Use global tables for multi-Region/multi-writer low-latency and HA. Alternatively, enable PITR and on-demand backups for recovery.
Centralized backups with AWS Backup:
- Backup plans: Define schedules (CRON), backup windows, lifecycle (transition to cold storage, retention), and copy actions to other Regions/accounts. Assign resources by tag or ARN for policy-driven coverage.
- Backup vaults: Logical containers with independent KMS encryption keys and access policies. Enable AWS Backup Vault Lock for WORM immutability and ransomware-resistant posture. Use cross-account vault copies for blast-radius reduction.
- Cross-account backup: In Organizations, apply backup policies to member accounts for consistent governance. Configure vault access policies to allow copy/restore from a central backup account. Regularly perform automated restore drills to measure RTO and validate runbooks.
- Integrations: Protect EBS, EC2, RDS/Aurora, DynamoDB, EFS, FSx, and more. Align schedule and retention to regulatory RPO/RTO, and coordinate with application-consistent quiescing where required (SSM pre/post scripts).
Chaos engineering and AWS Fault Injection Simulator (FIS)
Chaos experiments validate that HA and DR mechanisms behave as designed. AWS FIS orchestrates controlled faults with guardrails:
- Experiment templates define actions (for example, stop or reboot a percentage of EC2 instances in an ASG, inject CPU or memory stress via SSM, add network latency/packet loss on instances, kill EKS pods, stop ECS tasks, trigger RDS/Aurora failover) and targets (resource tags, ARNs).
- Safety controls: Specify CloudWatch alarm stop conditions, time limits, blast radius constraints via tags/filters, and pre-checks. Run in non-prod first, then in production with tight guardrails and business approval.
- Observability: Instrument KPIs (error rate, tail latency, queue age, replica lag) and verify automated responses, including Auto Scaling reactions, load balancer health convergence, Route 53 failover, database promotion, and circuit breaker behavior.
- Continuous resilience: Integrate experiments into pipelines/gamedays to prevent configuration drift from eroding resilience. Use Parameter Store or AppConfig for toggles and to coordinate safe rollouts.
Practical Problem Scenario
Expedia Group operates a global trip-search API that must provide sub-60-second RTO and near-zero RPO for critical booking data, while maintaining low latency for users in North America and Europe. The team experiences occasional Regional brownouts and deployment-induced instability, and auditors require cross-account immutable backups and documented DR drills.
Step-by-step approach:
- Establish multi-Region, active/active stacks
- Deploy stateless API stacks in us-east-1 and eu-west-1 across multiple AZs behind ALBs. Use Route 53 latency-based routing with health checks and Evaluate Target Health on alias records. This delivers low-latency routing and automatic Regional evasion if an endpoint is unhealthy.
- Global, low-RPO datastore
- Migrate booking and session data to Amazon Aurora Global Database (MySQL-compatible) with us-east-1 as primary and eu-west-1 as secondary. Typical RPO <1 s and RTO <1 min meets the outage target. Use the cluster and reader endpoints in application config with retry/backoff to tolerate failovers.
- Resilient scaling and graceful transitions
- Configure Auto Scaling target tracking on ALB RequestCountPerTarget with minimum capacity in both Regions. Add warm pools sized to absorb the 10× traffic surge during major events and lifecycle Launching:Wait hooks to delay registration until app readiness checks pass. Enable ALB deregistration delay of 120 seconds to preserve in-flight requests during scale-in and deployments.
- Durable object DR
- Enable S3 Versioning and CRR with RTC for itinerary documents from us-east-1 to eu-west-1. Use a dedicated replication IAM role and KMS keys in both Regions, granting kms:Decrypt on source and kms:Encrypt on destination. RTC metrics and alerts provide confidence in replication SLAs.
- DNS controls for canary and failover
- Add weighted Route 53 records (1% constant flow to eu-west-1) to exercise the secondary path continuously. Combined with health checks, this ensures the standby is production-ready and catches drift before a crisis.
- Centralized, immutable backups
- In a security-owned backup account, create AWS Backup vaults with Vault Lock and KMS CMKs. Define organization-level backup policies to schedule daily backups and cross-account copies for RDS, DynamoDB, EFS, and EBS. Assign resources by Backup_Frequency tag. This achieves ransomware resistance and separation of duties.
- Automated failover orchestration
- Implement an EventBridge rule to detect Aurora primary failure signals and invoke a Lambda function that promotes the secondary Region and updates an application endpoint stored in Parameter Store. Applications load the endpoint on startup and refresh on connection errors, minimizing manual steps.
- Chaos validation with AWS FIS
- Create FIS experiment templates to: terminate 10% of ASG instances, inject 150 ms latency and 1% packet loss on EC2 via SSM, and trigger Aurora failover. Guard with CloudWatch alarm stop conditions on p95 latency and error rate. Run monthly gamedays to validate Route 53 failover speed, ASG recovery, ALB health convergence, and Aurora promotion RTO.
Why these services:
- Route 53 latency-based and weighted routing provide both optimal user latency and controlled traffic shaping for DR readiness.
- ALB plus ASG with warm pools and lifecycle hooks ensure quick, graceful scaling without cold-start penalties or user disruption.
- Aurora Global Database uniquely meets near-zero RPO and sub-minute RTO across Regions with minimal app changes.
- S3 Versioning and CRR with RTC deliver auditable, SLA-backed replication for critical artifacts.
- AWS Backup with cross-account vaults and Vault Lock creates immutable, centrally governed backups aligned to compliance needs.
- AWS FIS supplies safe, automated fault injection to continuously prove the resilience posture and prevent configuration drift from undermining the DR plan.
← Containers and Serverless Operations · All domains · Event-Driven Architectures and Automation →
Practice these questions → · Timed practice on ExamRoll.io →
Pass the whole exam — not just this question
You found this answer. Get every verified question and explanation in one place, and save hours of prep. Free to start.
Pass your exam →