Google ACE: Reliability, Backup and Disaster Recovery — Study Guide

By Laurynas Brazys · 31 August 2026 · #google #ACE #certification #exam-prep #study-guide

Part of the Google Associate Cloud Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Reliability, backup, and disaster recovery on Google Cloud require intentional design across failure domains, data protection mechanisms, traffic management, and operational readiness. This section explains how to structure services across zones and regions, how to protect and restore stateful data, and how to validate recovery objectives with disciplined runbooks and continuous resilience testing. It also outlines DR strategy trade-offs and capacity planning to ensure the platform can recover within defined recovery time (RTO) and recovery point (RPO) objectives.

Failure Domains and Regional Design

• Zones, regions, and multi-region services

  • Zones are the smallest independent failure domains. A single zonal failure should not interrupt a regional service.
  • Regions group independent zones with low-latency links. Regional designs survive zonal failures but not necessarily full regional events.
  • Multi-region services replicate across regions, protecting against a regional loss at higher cost and potentially higher write latency.
  • Design principle: avoid single points of failure at the smallest failure domain you care about. If your RTO/RPO requires surviving zonal failure, deploy across at least two zones. For regional survivability, deploy active components in multiple regions or consume multi-region services.

• Managed instance groups (MIG) and self-healing

  • Prefer regional MIGs to distribute instances across zones in one region. This mitigates zonal outages without separate tooling.
  • Load balancer health checks remove unhealthy VMs from traffic. MIG autohealing replaces unhealthy or unresponsive VMs. Use both.
  • Use application-level HTTP(S) health checks that verify readiness endpoints and dependencies. A TCP check only validates port reachability.
  • Autohealing misconfiguration failure mode: using only a load balancer health check prevents traffic to a bad instance but does not recreate it. Configure the MIG’s own health check for replacement and an initial delay to avoid premature restarts during boot.

Example:

• Create an application health check with 10-second intervals and 3 unhealthy thresholds to trigger self-heal after ~30 seconds: gcloud compute health-checks create http hc-app
  –check-interval=10s –timeout=5s
  –unhealthy-threshold=3 –healthy-threshold=1 –request-path=/healthz

• Attach the health check to a regional MIG for autohealing: gcloud compute instance-groups managed update my-rmig
  –region=us-central1 –health-check=hc-app –initial-delay=60

• Multi-region considerations

  • Global external HTTP(S) Load Balancing supports backends in multiple regions with automatic failover based on health checks.
  • State synchronization across regions is the critical trade-off. Active-active throughput is high, but consistency and conflict resolution must be designed. Active-passive is simpler but has slower failover and potentially larger RPO.

Data Protection: Databases, Storage, and Compute

• Cloud SQL high availability and replicas
  • High availability configuration places a standby in a different zone with synchronous replication. Automatic failover occurs on primary failure. RTO is typically minutes; RPO ≈ 0 within a region but consider in-flight transactions.
  • Read replicas offload reads and can be cross-region for DR. They are asynchronous; expect replication lag and non-zero RPO.
  • Backups and PITR
    • Enable automated backups and point-in-time recovery. For MySQL, enable binary logging; for PostgreSQL, enable PITR retention.
    • Restore flows: for corruption or user error, restore to a new instance at a specific timestamp, point applications or replicas to the restored instance, and validate data.
  • Failure modes and trade-offs
    • HA does not protect against logical data corruption; backups and PITR do.
    • Cross-region replicas protect against regional loss but can lag; test your acceptable RPO.

Example:

• Enable binary logging (MySQL) and automated backups: gcloud sql instances patch my-mysql
  –enable-bin-log
  –backup-start-time=03:00 –retained-backups=7

• Persistent disk (PD) snapshots and machine images

  • PD snapshots are incremental, crash-consistent backups of disks. They are storage-location configurable and can be used to create new disks in any zone within the snapshot’s location scope.
  • To achieve application-consistent backups, quiesce the filesystem and application, or coordinate snapshots with database-specific backup mechanisms.
  • Machine images capture disks and instance metadata (boot disks, attached disks, instance properties). Use machine images for faster fleet recovery or cloning a golden server configuration.
  • Snapshot schedule policies automate backups; enforce retention; tag critical disks accordingly.
  • Restore workflow: create a disk from snapshot, attach to a new instance, update startup scripts and service accounts, then validate application integrity before reintroducing traffic.

Example:

• Create a snapshot and restore to a new disk: gcloud compute disks snapshot vm-boot –snapshot-names=boot-2024-09-01 gcloud compute disks create restored-boot –source-snapshot=boot-2024-09-01 –zone=us-central1-a

• Cloud Storage replication, versioning, retention

  • Storage class choices: Standard for hot data; Nearline for infrequent monthly access; Coldline for quarterly access (backups and DR recommended); Archive for long-term, rarely accessed data.
  • Regional, Dual-Region, and Multi-Region buckets offer durability via replication. Dual-Region with turbo replication can constrain replication RPO for newly written objects; Multi-Region offers wide geographic resilience.
  • Object versioning protects against accidental deletions and overwrites by keeping noncurrent versions. Combine with retention policies and bucket lock to enforce WORM retention.
  • Accidental-deletion protection: enable Object Versioning, use retention policy with a lock, implement event-based holds for legal or processing gates, restrict deletes through IAM and uniform bucket-level access. For external time-bound access, use signed URLs with strict expirations.

Example lifecycle to move to Coldline after 90 days and delete after 365 days:

• lifecycle.json { “rule”: [ { “action”: { “type”: “SetStorageClass”, “storageClass”: “COLDLINE” }, “condition”: { “age”: 90 } }, { “action”: { “type”: “Delete” }, “condition”: { “age”: 365 } } ] }
• Apply: gsutil lifecycle set lifecycle.json gs://my-bucket

Traffic, Capacity, and Dependency Resilience

• DNS and traffic-management failover

  • Prefer global external HTTP(S) Load Balancing for internet-facing services; it uses anycast IPs and performs health-based failover across regions.
  • For private services, use internal HTTP(S) or TCP/UDP load balancers. Design for zonal independence with multiple backends.
  • DNS TTL trade-offs: low TTLs enable faster failover but increase query load and can be ignored by some resolvers due to caching behavior. Health-check-based LB failover is faster and more deterministic than DNS-only failover.
  • Weighted or failover DNS policies can be a last-resort control plane for region evacuation but rely on cache expiry.

• Capacity planning and quota design

  • Identify minimum healthy capacity per zone and region. Apply headroom buffers for failover (“N+1 zone” capacity).
  • Use regional reservations for critical Compute Engine shapes to guarantee capacity during scale events or failover.
  • Pre-provision IP addresses, forwarding rules, Cloud NAT capacity, connection tracking, and SSL certificates to avoid control-plane delays during recovery.
  • Request quota increases well before you need them; validate quotas in secondary regions and for all dependencies (e.g., Cloud SQL instances per region, forwarding rules per VPC, Pub/Sub throughput, Cloud KMS QPS).
  • Autoscaling considerations: configure cooldowns, predictive autoscaling if needed, and set min/max bounds to keep exactly one instance when required by policy.

• Dependency resilience

  • Inventory upstream and downstream services. For each, define failure behavior and fallbacks: cached configuration, degraded modes, circuit breakers, queues with dead-letter topics, and backpressure.
  • Validate IAM and service account scoping in recovery regions. Missing roles commonly cause silent failures during DR events.
  • Encryptions keys: ensure Cloud KMS key replicas or multi-region keys align with data location. Plan for key-ring locality and IAM in secondary regions.

Resilience Operations and Continuous Improvement

• RTO, RPO, recovery plans, and runbooks

  • RTO defines how quickly service must resume; RPO defines acceptable data loss. Derive these from business impact analysis.
  • Map each system component to specific mechanisms that satisfy RTO/RPO: HA for zonal failures, cross-region replication for regional failures, backups for corruption, and storage class/replication for durability.
  • Maintain runbooks: precise steps, commands, credentials access, health validation checks, and decision trees. Store in a versioned, access-controlled repository and rehearse regularly.
  • Recovery validation: schedule drills to measure actual RTO/RPO, validate data integrity, and collect improvement actions. Test both small-scope restores (table, disk) and full site recovery.

• DR strategies and trade-offs

  • Active-active: all regions serve traffic; minimal RTO and low RPO if data synchronization is designed correctly. Higher complexity and cost; requires conflict resolution and global load balancing.
  • Active-passive: primary region is active; secondary is warm and receives replicated data. Moderate cost; RTO minutes to tens of minutes; non-zero RPO depending on replication.
  • Pilot-light: minimal critical services run in secondary (database replication, minimal app footprint). RTO hours; cost-efficient; careful orchestration needed to scale up compute during failover.
  • Cold-standby: infrastructure defined as code but not provisioned. RTO days; lowest cost; risk of surprises from drift, quotas, and capacity scarcity.

• Chaos testing and failure simulations

  • Regularly simulate instance crashes, process hangs, health check failures, disk full conditions, and dependency outages. Use tools or scripts to terminate instances, block egress to backends, or inject latency at the proxy layer.
  • Validate MIG autohealing and LB removal by intentionally failing the health endpoint. Confirm replacement behavior and recovery timelines.
  • Practice regional evacuation: drain backends in one region, observe global load balancer failover, and verify stateful dependencies in the secondary region.
  • Continuous improvement: record metrics for mean time to detect, failover time, and data loss during tests. Prioritize fixes that reduce RTO/RPO and remove manual steps.

Practical Problem Scenario

Brightlane Retail operates an e-commerce platform in us-central1 with strict availability and a four-hour RPO for order data. Leadership requires surviving a zonal outage without downtime and a regional outage with minimal customer impact.

Approach:

1. Implement a regional MIG with HTTP autohealing and a global HTTP(S) Load Balancer

   • Rationale: A regional MIG spreads instances across zones, and application-level HTTP health checks enable self-healing after three failed checks of 10 seconds each. The global load balancer automatically removes unhealthy VMs and fails traffic to healthy zones.
   • Commands: gcloud compute health-checks create http app-hc –check-interval=10s –timeout=5s –unhealthy-threshold=3 –request-path=/healthz gcloud compute instance-groups managed update web-rmig –region=us-central1 –health-check=app-hc –initial-delay=60

2. Enable Cloud SQL HA with cross-region read replica and PITR

   • Rationale: Regional HA provides zonal survivability with automatic failover. A read replica in us-east1 provides regional DR with non-zero but bounded RPO. Enabling PITR (binary logging for MySQL) addresses logical corruption by allowing restore to a point in time.
   • Commands: gcloud sql instances patch orders-mysql –enable-bin-log –backup-start-time=03:00 gcloud sql instances create orders-replica –master-instance-name=orders-mysql –region=us-east1

3. Protect object assets with Dual-Region Cloud Storage and lifecycle policies

   • Rationale: Product images and static assets are stored in a dual-region bucket for regional resilience. Lifecycle transitions move older artifacts to Coldline to optimize cost, and versioning plus retention policies prevent accidental deletion of critical assets.
   • Steps: Enable Object Versioning, set a 30-day retention policy for critical buckets, and apply lifecycle rules to transition to Coldline after 90 days and delete after one year for non-critical build artifacts.

4. Schedule PD snapshots and create machine images for stateful services

   • Rationale: Incremental snapshots of VM disks provide fast crash-consistent restore options. Machine images capture boot and config to accelerate rehydration of app servers during a region failover. Snapshot schedules ensure consistent, policy-driven backups.
   • Commands: gcloud compute resource-policies create snapshot-schedule daily-2am –max-retention-days=14 –on-source-disk-delete=apply-retention-policy –start-time=02:00 gcloud compute disks add-resource-policies app-disk-1 –resource-policies=daily-2am gcloud compute machine-images create app-mi-2024-09-01 –source-instance=app-vm-template

5. Define RTO/RPO and codify DR runbooks and IaC

   • Rationale: Set service-level RTO of 15 minutes for web/API and four-hour RPO for orders. Runbooks specify traffic failover procedures, Cloud SQL promotion of read replica, DNS contingencies, and verification steps. Infrastructure as code (Terraform/Deployment Manager) ensures deterministic rebuilds and reduces manual error.

6. Provision capacity and quotas in the secondary region

   • Rationale: Create reservations for critical VM shapes, pre-provision a standby load balancer backend, SSL certificates, NAT capacity, and verify quotas for Compute, SQL, forwarding rules, and KMS in us-east1. This prevents capacity starvation during failover.

7. Validate recovery through chaos drills and document improvements

   • Rationale: Quarterly zonal-failure drills validate MIG and LB behavior; semiannual regional evacuation promotes the read replica in us-east1, points the global LB to us-east1 backends, and measures RTO/RPO. Findings drive improvements such as reducing manual steps or increasing replica capacity.

By following these steps, Brightlane Retail achieves zonal high availability with automated self-healing and a regional DR posture with defined, tested runbooks, ensuring that order data meets a four-hour RPO and application services recover within targeted RTO bounds.

← Security · All domains · Cost Management →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →