Google PCA: Reliability, Disaster Recovery and Business Continuity — Study Guide

By Laurynas Brazys · 5 September 2026 · #google #PCA #certification #exam-prep #study-guide

Part of the Google Professional Cloud Architect — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Reliability, disaster recovery (DR), and business continuity ensure that services continue to meet agreed objectives despite component, zone, or regional failures. On Google Cloud, reliability is engineered by understanding failure domains (zonal, regional, and global), defining recovery objectives (RTO/RPO), selecting resilient service architectures (e.g., active-active), and rigorously testing recovery plans. Your design must map service criticality to explicit availability targets, durability guarantees, and validated recovery paths, balancing availability, consistency, cost, and operational complexity. Key themes include isolating single points of failure, using managed replication where possible, automating failover decisions, and continuously validating that assumptions hold in production-like conditions.

Failure Domains, Locations, and Multi-Regional Services

• Availability zones and regions:
  • Zones are independent failure domains within a region. Zonal failures are the most common large-scale events you design for.
  • Regions are collections of zones with low-latency links. Regional failures are rarer but must be considered for high-criticality systems.
  • Design patterns:
    • Intra-region: Place stateless compute across at least two zones via regional managed instance groups (MIGs).
    • Inter-region: Replicate state and failover traffic for critical services that cannot tolerate regional loss.
• Multi-regional and global services:
  • Global control planes: VPC networks, Cloud DNS, global external HTTP(S) load balancing, and Cloud IAM are global-scope services used to reduce regional coupling.
  • Data-plane placement matters:
    • Cloud Storage: choose regional, dual-region, or multi-region buckets aligned to access patterns and DR needs.
    • BigQuery: datasets live in a region or multi-region; multi-region improves availability of the analytics surface but consider data residency and egress for external joins.
    • Spanner: instance configuration (regional or multi-regional) defines replica topology and consistency behavior.
• Failure-domain analysis:
  • Map every component to its blast radius. Examples:
    • Zonal: single VM, zonal GKE node pool, zonal SSD PD.
    • Regional: Cloud SQL HA primary+standby are regional; some maintenance events can impact a region.
    • Global: misconfigured IAM or Cloud DNS affects all regions.
  • Identify correlated failures such as shared dependencies (e.g., one NAT gateway, one Memorystore instance) or human-induced risk (shared service account, single Terraform state).
  • Consider quotas as failure domains; an autoscaler that hits a regional quota cap is functionally down.

Trade-offs:

• Cross-zone replication reduces downtime but adds cross-zone traffic and cost.
• Cross-region designs reduce RTO but increase latency, complexity, and spend.
• Global anycast load balancing simplifies failover but masks unhealthy backends only if health checks are precise.

Objectives, Dependency Mapping, and DR Validation

• RTO and RPO:
  • Recovery Time Objective (RTO): target time to restore service. Drives automation depth, standby posture, and runbook detail.
  • Recovery Point Objective (RPO): acceptable data loss window. Drives choice of replication and backup cadence.
• Service criticality and tiering:
  • Define tiers (e.g., Tier 0: safety/financial impact; Tier 1: revenue; Tier 2: internal tools) with target SLOs, RTO/RPO, and testing frequency.
  • Tie spend and complexity to tier; not every service needs cross-region.
• Dependency mapping:
  • Inventory upstream and downstream dependencies: identity (Cloud IAM, SAML IdP), secrets (Secret Manager, KMS), networking (DNS, Cloud Interconnect/VPN), storage and DBs, observability, CI/CD, and third-party APIs.
  • Document region, zone, and SLA for each dependency; define compensating controls for weaker links.
• Recovery plans:
  • Create runbooks and automation for failover/failback, data restores, and configuration promotion (DNS, load balancer backends, firewall).
  • Pre-provision permissions and service accounts; stage infrastructure definitions to eliminate manual gating.
  • Maintain break-glass access with auditable elevation.
• Recovery testing:
  • Schedule routine failovers for stateful systems (e.g., Cloud SQL HA) to validate promotion and connection renegotiation.
  • Conduct game days that simulate zonal or regional outage; include upstream providers and IAM/KMS failures.
  • Use fault injection to validate circuit breakers, timeouts, and retries; verify that autoscaling and backpressure work as intended.
  • Continuously measure RTO/RPO during tests; adjust architecture when objectives are missed.

Resilient Compute, Databases, and Storage Patterns

• Self-healing compute with regional MIGs and load balancing:
  • Use regional MIGs to distribute instances across zones with autoscaling and autohealing.
  • Front-end with a global external HTTP(S) load balancer and a backend service health check aligned to true readiness (e.g., /healthz checks dependencies).
  • Allow health checks through firewalls to avoid constant VM recycling:

    gcloud compute firewall-rules create allow-lb-health-checks \
      --network=prod-vpc --action=ALLOW --direction=INGRESS \
      --rules=tcp:80,tcp:443 \
      --source-ranges=130.211.0.0/22,35.191.0.0/16 \
      --target-tags=web-backend
    

  • Avoid local state; externalize sessions to Memorystore or databases; use connection draining on backends to preserve in-flight requests during scale-in.
  • Common failure modes: misaligned health checks (checking too much or too little), missing firewall rules, and bootstrapping that depends on unhealthy downstreams.
• Cloud SQL resilience:
  • High availability: primary and standby in separate zones with synchronous disk replication and automatic failover; select a maintenance window and test failovers.
  • Read replicas: add same-region or cross-region read replicas to offload reads and reduce RTO for regional events; promote replicas during DR.
  • Backups and PITR:
    • Enable automated daily backups and point-in-time recovery (PITR) via binary/transaction logs with sufficient retention for compliance and RPO.
    • Validate restores into non-production, and rehearse promotion procedures and application connection string updates.
  • Network: prefer private IP for production; ensure failover tests validate DNS/connection pooling behavior.
  • Operational tip: periodically execute a controlled failover to verify that application pools reconnect cleanly.

    gcloud sql instances failover prod-sql
    

• Spanner configuration and resilience:
  • Regional instances provide low-latency, strongly consistent reads/writes within a region using Paxos across zones.
  • Multi-regional instances replicate across regions with synchronous quorum writes (strong global consistency) and optional read-only replicas; choose a leader region close to writers.
  • Trade-offs: multi-regional improves RTO/RPO and read availability but increases write latency and cost. Use for globally distributed, write-intensive workloads needing strong consistency; otherwise consider regional Spanner or Cloud SQL with replicas.
• Cloud Storage durability and recovery patterns:
  • Location strategy: regional for compute-locality, dual-region for active-active across two regions, multi-region for broad availability to global users.
  • Versioning: enable object versioning to recover from deletion or corruption; combine with lifecycle rules to manage costs.
  • Retention: apply bucket-level retention policies and, if required, retention locks for compliance; use event-based holds for records management.
  • Backup patterns: cross-project, separate-admin buckets mitigate accidental deletion and privilege escalation. For databases, export logical backups to Cloud Storage in a distinct project.
  • Example lifecycle rule to delete versions older than 90 days:

    {
      "rule": [
        {
          "action": { "type": "Delete" },
          "condition": { "age": 90, "isLive": false }
        }
      ]
    }
    

    Apply with:

    gsutil lifecycle set lifecycle.json gs://prod-backups
    

  • Recovery: maintain catalogs of critical objects and test restores; for large datasets, stage restores to temporary buckets to avoid name collisions and validate integrity.

Traffic Management, Multi-Site Strategies, and Continuous Resilience

• Multi-site strategies:
  • Active-active: serve traffic from multiple regions simultaneously; requires symmetric data replication and conflict-free writes. Best RTO/RPO; highest cost and complexity.
  • Active-passive: a hot primary and ready secondary; data replicated continuously, traffic switched during failure. Good balance of cost and RTO.
  • Warm standby: scaled-down secondary with pre-synced data; requires scale-up at failover; moderate RTO and cost.
  • Pilot light: minimal critical data replication and infrastructure definitions; most components provisioned at failover; long RTO, low steady cost.
  • Cold standby: periodic backups only; rehydrate on failure; longest RTO, lowest cost.
• DNS and traffic-management failover:
  • Prefer health-based routing at Layer 7 with the global external HTTP(S) load balancer. It performs per-backend health checks and shifts traffic away from unhealthy zones or regions without DNS changes.
  • Use low-TTL DNS records only as a coarse failover control or to switch between disjoint load balancer VIPs; understand DNS caching means failover is not instantaneous.
  • For private services, use Internal HTTP(S) Load Balancing with regional failover patterns plus private DNS that you can update programmatically if needed.
• Graceful degradation patterns:
  • Implement feature flags to disable non-critical functionality under stress.
  • Use circuit breakers, timeouts, retries with jitter, and bulkheads to localize failures.
  • Provide read-only mode when write paths are impaired; queue writes for later reconciliation.
  • Rate-limit clients and apply backpressure to prevent cascading failures.
• Chaos testing and continuous improvement:
  • Fault injection at the network (latency, packet loss) and application layers validates that resiliency controls trigger as designed.
  • Game days operationalize recovery across teams; include paging, runbook execution, and postmortems with concrete corrective actions.
  • Track error budgets and SLOs; adjust capacity, retry strategies, and replication configurations as data warrants.
• Trade-offs across availability, consistency, cost, and complexity:
  • Availability vs. consistency: strong global consistency (e.g., Spanner multi-regional) may add write latency; eventual consistency (e.g., asynchronous replicas) may improve latency but risk stale reads.
  • Cost vs. RTO/RPO: dual-region storage and multi-regional databases increase spend but minimize data loss and downtime.
  • Complexity vs. reliability: every failover mechanism, replication stream, and routing rule must be operated and tested; keep designs as simple as necessary to meet objectives.

Practical Problem Scenario

Acme Tickets, a fast-growing online ticketing company, must ensure continuous operations during regional outages for its purchase API and event catalog while maintaining stringent RTO/RPO (RTO ≤ 5 minutes, RPO ≤ 1 minute). The stack includes stateless microservices, a relational order database, an analytics pipeline, and static media assets.

1. Define service tiers, SLOs, and recovery objectives

• Rationale: Classify purchase API and order DB as Tier 0 (RTO 5m, RPO 1m), catalog as Tier 1 (RTO 15m, RPO 5m), and analytics as Tier 2 (best effort). This aligns cost and complexity to business impact.

2. Choose regional placement and multi-site strategy

• Rationale: Deploy active-active across us-central1 and us-east1 for stateless services to minimize RTO; use active-passive for the order database to balance write latency and cost.

3. Implement regional MIGs and global HTTP(S) load balancing

• Rationale: Two regional MIGs (one per region) spread across at least two zones each. A single global anycast VIP routes traffic via health-checked backend services, failing unhealthy regions out automatically.

4. Externalize state and configure self-healing

• Rationale: Store sessions in Memorystore with cross-region read replicas for the catalog, and keep services stateless so MIG autohealing and rolling updates are safe. Health checks point to /healthz that validates critical downstreams.

5. Provision Cloud SQL for PostgreSQL with HA and cross-region read replica

• Rationale: Use HA in primary region for zonal resilience and enable PITR with sufficient retention. Create a cross-region read replica in the secondary region and a tested runbook to promote on regional failure, meeting RPO ≤ 1 minute with minimal write loss.

6. Schedule routine database failover tests

• Rationale: Run monthly controlled failovers to validate application reconnection behavior and replica promotion. This addresses a common failure mode where replicas are never promoted during real incidents.

7. Place static media in a dual-region Cloud Storage bucket with versioning and retention

• Rationale: Dual-region ensures object availability across two regions; versioning protects against accidental overwrites/deletes. Apply lifecycle rules to expire old versions and control cost.

8. Protect health checks and egress with firewall and quotas

• Rationale: Create explicit firewall rules for load balancer health checks and monitor regional instance quotas to prevent autoscaler stalls during failover.

9. Implement DNS as a coarse control with low TTL

• Rationale: While the global load balancer handles health-based routing, maintain a low-TTL A record to a standby VIP for emergency manual cutover, understanding DNS cache limitations.

10. Automate DR runbooks and validate via game days

• Rationale: Use Cloud Scheduler to trigger synthetic traffic and Cloud Monitoring SLOs to confirm behavior during quarterly game days with injected faults (e.g., blocking inter-region traffic, terminating nodes). Capture RTO/RPO metrics and refine procedures.

11. Secure and separate backups

• Rationale: Export daily logical backups of the order DB to a Cloud Storage bucket in a separate project with retention locks; periodically restore to a staging instance to validate integrity and timing.

12. Implement graceful degradation

• Rationale: If the order DB is degraded, switch the catalog to read-only, queue writes for later reconciliation, and shed non-critical features. This prevents cascading failures and maintains partial service.

This architecture delivers automated regional failover for stateless services, controlled and tested failover for stateful components, and verified recovery processes that meet Acme Tickets’ business continuity objectives.

← Security · All domains · Migration →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →