Google PCD: Performance, Scalability and Resilience Engineering — Study Guide

By Laurynas Brazys · 3 September 2026 · #google #PCD #certification #exam-prep #study-guide

Part of the Google Professional Cloud Developer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Performance, scalability, and resilience engineering on Google Cloud focus on maintaining low-latency, cost-effective service under variable load while tolerating faults without violating SLOs. Design must align autoscaling signals with workload characteristics, place data and compute to minimize tail latency, and implement overload controls, retries, and failover to avoid cascading failures. This section details the patterns, controls, and trade-offs that matter for application developers across compute, networking, and data layers.

Scaling and Load Distribution

Horizontal vs vertical scaling

Horizontal scaling adds instances or pods to increase capacity and resiliency. Prefer for stateless services and when you need rapid elasticity. Use Managed Instance Groups (MIGs), Cloud Run revisions, or GKE Deployments.
Vertical scaling increases machine size. Useful for single-threaded or memory-bound workloads, or to reduce inter-node coordination, but offers limited headroom and longer restart times.
Concurrency: Tune request concurrency to match CPU-bound vs I/O-bound profiles. Cloud Run supports per-revision concurrency; GKE pods can serve multiple requests if your runtime is non-blocking; for strict isolation, set concurrency to 1.

Autoscaling signals and warm capacity

MIG autoscaling supports CPU utilization, load balancing utilization, and custom metrics via Cloud Monitoring. For bursty traffic, base scaling on request metrics (rps, queue depth) rather than CPU.
GKE Horizontal Pod Autoscaler (HPA) can scale on CPU, memory, or custom/external metrics (e.g., Pub/Sub queue length). Use Vertical Pod Autoscaler (VPA) for right-sizing, but avoid VPA live updates on rapidly scaling frontends to prevent churn.
Cloud Run scales on concurrent request load and optionally custom metrics. Avoid cold starts by maintaining warm capacity: configure minimum instances, keep idle concurrency low, and pre-warm via synthetic health pings if needed.
Predictive autoscaling in MIGs and setting min replicas on Deployment/Revision help mask provisioning latency during diurnal peaks.

Load balancing, global traffic distribution, health checks, and failover

Use the global external Application Load Balancer for worldwide anycast VIP, HTTP/2 and HTTP/3, and edge termination with Cloud CDN. Backends can be instance groups, zonal/regional NEGs, serverless NEGs (Cloud Run/Functions), or GKE Ingress.
Health checks drive traffic away from unhealthy backends. Ensure your health endpoints validate dependencies narrowly (e.g., process and critical local resources) to avoid circular failures on downstream outages.
Firewall allowlists must permit health checkers. If checks to port 80 fail, allow Google’s ranges: gcloud compute firewall-rules create allow-lb –network load-balancer –allow tcp –source-ranges 130.211.0.0/22,35.191.0.0/16 –direction INGRESS
Failover: Configure primary/backup backend services or traffic policies that steer to alternate regions on health failure. For DNS-level failover, use Cloud DNS policies with health checks for non-HTTP endpoints.

Latency and Efficiency

Latency budgets

Allocate end-to-end latency budget per tier (client, edge, app, data). Monitor p95/p99, not averages. Use Cloud Trace to find cross-service contributors and head-of-line blocking. Apply deadlines to RPCs so upstream cancelation frees capacity.

Caching and CDN use

Layer caches: client/browser cache, CDN edge (Cloud CDN), and regional/memory caches (Memorystore or in-process). Choose cache keys and vary headers carefully. Set TTLs based on data freshness and risk of staleness; consider negative caching for 404s when safe.
Serve static assets from Cloud Storage behind Cloud CDN to reduce origin load and tail latency. Use signed URLs/headers for controlled access.

Connection reuse

Prefer HTTP/2 or gRPC for multiplexing and header compression. Enable keep-alives and connection pooling to reduce handshake overhead. Watch NAT port exhaustion; tune client connection pools and idle timeouts, and size Cloud NAT ports per VM if applicable.

Payload efficiency

Use binary encodings (e.g., protobuf) and compress text payloads (gzip/brotli) above a size threshold. Design request/response fields carefully; paginate, filter server-side, and avoid over-fetching. Use ETags and conditional requests (If-None-Match) to avoid redundant transfers. For Cloud Storage, use generation preconditions and Range reads for partial content.

Overload and Resilience Patterns

Rate limiting, backpressure, queueing, and batching

Enforce rate limits at the edge (Cloud Armor for IP/geo/service-based rate limiting) and at the API layer (Apigee quotas, per-API client tokens). Implement token-bucket or leaky-bucket algorithms server-side for fair sharing.
Backpressure: Don’t outpace downstreams. Use queues (Pub/Sub for at-least-once eventing; Cloud Tasks for per-queue and per-target throttles with scheduling and retries). Propagate 429 Too Many Requests or 503 with Retry-After to push back clients.
Batching can increase throughput and reduce per-call overhead (e.g., batch mutations to databases or batch Pub/Sub acks), trading increased latency for efficiency. Tune batch size and max wait.

Overload protection

Apply timeouts and deadlines to every RPC. Use circuit breakers to stop sending work to failing dependencies and enable fast fallbacks. Implement load shedding based on queue depth, CPU, or latency SLO breach to protect core functionality.

Resilient retries, exponential backoff, jitter, idempotency, and duplicate handling

Retry only when safe: network timeouts, 5xx, or documented retryable codes (e.g., Cloud Storage 429/5xx). Never retry on 4xx like 400/401/403 unless specified.
Use truncated exponential backoff with jitter to avoid synchronized retries. Prefer full jitter. Example: for attempt in range(max_attempts): sleep = min(base * 2**attempt, cap) * random.uniform(0.5, 1.5) try_request_or_break()
Ensure idempotency. Use idempotency keys (e.g., a unique operation ID) and upserts/conditional writes to tolerate duplicates. For Pub/Sub, de-duplicate using messageId or application keys; design handlers to be safe for at-least-once delivery. For Cloud Storage writes, use generation-match preconditions to avoid overwrites.

Dormant-resource ramp-up

Some services enforce adaptive limits. For Cloud Storage, ramp request rates gradually on previously idle buckets to reduce transient 429/5xx during sudden spikes. Throttle producers and warm buckets with controlled traffic before full load.

High Availability, Data, DR, and Testing

Multi-zone, regional, multi-region; active-active vs active-passive

Deploy across failure domains. Use regional MIGs or regional GKE clusters for zone failure tolerance. For global services, use multiple regions with the global load balancer.
Active-active reduces RTO and latency but demands conflict-free data and careful consistency management. Active-passive simplifies write semantics but incurs higher RTO and potential cold capacity.

RTO, RPO, backup, restore, and disaster-recovery testing

Define RTO (time to recover service) and RPO (tolerable data loss) per workload. Map to platform capabilities:
- Cloud Spanner: multi-region with five 9s availability and synchronous replication for near-zero RPO.
- Cloud SQL: High Availability within region; use cross-region replicas for DR, enable PITR, and validate failover/failback runbooks.
- Firestore and Bigtable offer regional and multi-regional options; choose to meet RTO/RPO.
- Cloud Storage dual- or multi-region buckets provide geo-redundancy; verify restore procedures and signed URL re-issuance.
Test DR: Run failover drills regularly. Validate backups by restoring into an isolated environment, rehearse DNS/traffic failover, and measure actual RTO/RPO.

Database and storage performance, index design, hot keys, and contention

Cloud Spanner: Avoid monotonically increasing primary keys that hotspot. Use interleaved tables for locality, secondary indexes for read patterns, and bounded transactions to reduce lock conflicts. Size nodes for QPS and storage; keep at least three nodes for production quorum and headroom.
Cloud SQL: Analyze queries, add covering indexes, avoid long transactions, and use connection pools. Tune InnoDB or Postgres settings judiciously; scale read replicas for read-heavy workloads.
Bigtable: Design row keys to evenly distribute load (salting or field reversal). Use multi-cluster routing for high availability across regions if available.
Firestore: Use composite indexes for multi-field queries; be aware of hotspotting when many writes target the same document path.
Cloud Storage: Strong read-after-write for new objects; use parallel uploads and chunking for throughput. Ramp traffic on idle buckets; prefer CDN edge for hot reads. For many VMs needing the same large read-only dataset, attach a persistent disk in read-only mode to multiple instances for fast local access at low cost.

Load tests, chaos experiments, fault injection, and capacity planning

Load testing: Simulate realistic traffic shapes and data distributions. Warm caches and autoscalers; test p95/p99 latency under load and during scaling events. Mirror a small fraction of live traffic to shadow stacks to validate behavior at production complexity.
Chaos and fault injection: Kill pods/VMs, cordon a zone, inject latency/errors at the service mesh (e.g., Envoy/Istio) to observe blast radius and resilience. Verify circuit breakers and retries behave as intended.
Capacity planning: Forecast using historical demand and planned events. Maintain headroom for N+1 failures and rebalancing. Align autoscaler cooldowns and max rates with expected spikes; pre-provision during predictable peaks.

Availability trade-offs among managed services and custom architectures

Compute: Cloud Run offers rapid scale-to-zero and low ops overhead but has cold starts and request concurrency constraints. GKE provides fine-grained control and portability at higher operational cost. Compute Engine VMs provide maximum control with the highest ops burden.
Data: Cloud Spanner provides global consistency and high availability at higher cost and schema rigor. Cloud SQL fits traditional RDBMS with simpler ops but limited HA/scalability. Bigtable excels at low-latency, massive scale key-value/time-series. Firestore provides flexible schemas with strong consistency and global options.
Networking: Global load balancers and Cloud CDN are highly available and operate at Google’s edge; DIY proxies offer customization but create operational and failure risk.
Prefer managed services for higher baseline availability and DDoS resistance, but account for quotas, cold starts, and service-specific semantics in your design.

Practical Problem Scenario

NimbusMart, a global ecommerce company, needs a low-latency product catalog API with five 9s availability and minimized read latency for users in North America, Europe, and Asia-Pacific. Writes must be globally consistent. Traffic is spiky during flash sales, and historical incidents include cascading retries and origin overload.

Approach:

Provision a multi-regional Cloud Spanner instance using nam-asia-eur1 with at least three nodes.

Rationale: Delivers globally consistent reads/writes with five 9s availability and places replicas near users to reduce read latency. A minimum of three nodes provides quorum robustness and headroom for rebalancing.

Implement a stateless API layer in multiple regions behind the global external Application Load Balancer.

Rationale: Anycast VIP and global routing reduce connection setup and steer users to the nearest healthy region. Stateless services ease horizontal scaling and failover.

Configure health checks and firewall rules for load balancer reachability.

Rationale: Health checks prevent routing to unhealthy backends. Allow Google health check IP ranges so checks succeed: gcloud compute firewall-rules create allow-lb –network load-balancer –allow tcp –source-ranges 130.211.0.0/22,35.191.0.0/16 –direction INGRESS

Implement autoscaling based on request metrics with warm capacity.

Rationale: Scale MIGs or GKE HPA on QPS/latency rather than CPU to react to flash-sale traffic. Maintain a minimum replica count per region to avoid cold starts and enable predictive autoscaling before known events.

Add Cloud CDN for static product media stored in Cloud Storage.

Rationale: Edge caching offloads origin, reduces tail latency, and mitigates burst amplification on the application and storage layers. Use signed URLs and appropriate cache keys/TTLs.

Enforce overload protection and rate limiting at the edge and service.

Rationale: Configure Cloud Armor rate limits to absorb abusive spikes. In the service, use token-bucket limits per client and shed low-priority requests when latency SLOs are threatened. Apply deadlines to every downstream call.

Use resilient retries with truncated exponential backoff and full jitter; ensure idempotency with operation IDs.

Rationale: Prevent thundering herds and duplicate writes during partial failures. Idempotency keys ensure safe replays; for storage operations, use conditional preconditions.

Introduce a write queue for burst smoothing and asynchrony where acceptable.

Rationale: Pub/Sub buffers sudden spikes of non-critical writes (e.g., analytics events), decoupling producers from Spanner and protecting primary write paths from overload.

Define SLOs and latency budgets; instrument tracing and dashboards.

Rationale: Per-tier budgets guide optimization. Cloud Monitoring SLOs with error budgets and Cloud Trace reveal cross-region and data-layer contributors to p99 latency.

Establish DR runbooks and test failover.

Rationale: With multi-region Spanner and multi-regional compute, practice region evacuation drills. Verify RTO with traffic drain and ramp-up timelines, and validate that autoscalers and CDN behave correctly during failover.

This design satisfies global availability and low-latency goals by aligning compute and data topology, enforcing overload controls, and using managed services that provide proven scalability and resilience.

← Observability · All domains · Testing →

Practice these questions → · Timed practice on ExamRoll.io →