Google PCD: Observability, Debugging and Site Reliability Operations — Study Guide

By Laurynas Brazys · 3 September 2026 · #google #PCD #certification #exam-prep #study-guide

Part of the Google Professional Cloud Developer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Observability, debugging, and site reliability operations in Google Cloud center on making systems measurable, diagnosable, and resilient. Strong observability requires consistent logging and metrics, distributed tracing, actionable alerting, and disciplined incident response. Reliability demands clarity on service-level indicators and objectives, rigorous health signaling, and a feedback loop that turns production insights into engineering improvements. This section outlines how to build these capabilities end to end in Google Cloud and how to reason about trade-offs and common failure modes.

Logging and Monitoring Foundations

Cloud Logging

• Emit structured logs. Prefer JSON with stable field names so queries and log-based metrics remain robust across releases. Include severity, service name, version, location, and a request ID or trace context for correlation.
• Correlate logs to traces by setting fields:
  • logging.googleapis.com/trace: projects/PROJECT_ID/traces/TRACE_ID
  • logging.googleapis.com/spanId: SPAN_ID
  • logging.googleapis.com/trace_sampled: true
• Buckets and retention. The _Default bucket typically has 30-day retention (configurable). The _Required bucket contains certain audit logs with longer, fixed retention. Create regional buckets to control data residency and set custom retention per bucket.
• Sinks. Route logs to BigQuery for analytics, to Pub/Sub for streaming consumers, or to Storage for archive. Use aggregated sinks at the folder or organization to capture children projects.
• Queries. Use the Logging query language to filter by resource.type, labels, jsonPayload, httpRequest, or textPayload.

Examples:

• Structured log entry (abbreviated): { “severity”: “ERROR”, “message”: “Checkout failed”, “service”: “payments”, “version”: “2026-09-01”, “user_id_hash”: “9c1…”, “labels”: {“tenant”:“gold”}, “logging.googleapis.com/trace”: “projects/myproj/traces/4bf92f3577b34da6a3ce929d0e0e4736”, “logging.googleapis.com/spanId”: “00f067aa0ba902b7” }
• Update retention: gcloud logging buckets update _Default –location=global –retention-days=180
• Create a BigQuery sink for error logs: gcloud logging sinks create bq-errors
  bigquery.googleapis.com/projects/myproj/datasets/log_analytics
  –log-filter=‘severity>=ERROR’
• Read recent 5xx for Cloud Run: gcloud logging read ‘resource.type=“cloud_run_revision” AND httpRequest.status>=500’ –limit=20

Cloud Monitoring

• Metrics. Use Google Cloud metrics, custom metrics, and logs-based metrics. Favor low-cardinality labels; exploding label cardinality causes cost and query latency issues.
• Dashboards. Curate dashboards per service and per dependency (database, cache, queues). Visualize RED (requests, errors, duration) and USE (utilization, saturation, errors) signals.
• Alert policies. Trigger on thresholds, metric absence, ratios, SLO burn rates, or uptime check failures. Configure notification channels (email, SMS, PagerDuty, Pub/Sub, webhooks). Suppress flapping via windowing and aligners.
• Uptime checks. Probe from multiple regions; use private uptime for internal endpoints or run synthetics from within the VPC.

Logs-based metrics

• Counters summarize occurrences (for example, count of /api/alpha/* requests).
• Distributions capture latency or payload sizes.
• Example: gcloud logging metrics create api_alpha_count
  –description=“Count of /api/alpha/* requests”
  –log-filter=‘httpRequest.requestUrl=~"/api/alpha/.*" AND resource.type=“cloud_run_revision”’

Operational analytics

• For ad hoc analysis, route to BigQuery via a sink; design schemas and partitioning by timestamp to control cost.
• Use Log Analytics in Logging buckets to aggregate without exporting where applicable.
• Build capacity signals from metrics such as CPU, memory, queue depth, Cloud SQL connections, Spanner high-priority CPU, Pub/Sub unacked messages, and Cloud Storage 429/5xx rates.

Common pitfalls and trade-offs

• Over-logging increases ingestion cost and obscures signal; prefer sampling and severity discipline.
• Missing correlation IDs hinders incident triage; propagate trace IDs end to end.
• Long retention in hot buckets increases cost; export to archive or BigQuery for long-term needs.

Tracing, Errors, and Deep Diagnostics

Distributed tracing

• Trace context. Prefer W3C trace-context (traceparent, tracestate). For Cloud Trace interoperability, continue supporting x-cloud-trace-context: x-cloud-trace-context: TRACE_ID/SPAN_ID;o=1
• Propagation. Forward trace headers across services, message queues, and async boundaries; capture new child spans when making RPCs or SQL calls. Loss of context breaks service maps and inflates “unknown service” nodes.
• Sampling. Balance cost and fidelity; dynamic head-based sampling on ingress and tail-based sampling for rare slow requests can improve usefulness.

Cloud Trace

• Provides latency histograms, span waterfalls, and service maps. Use annotations for critical sub-operations (RPCs, DB queries).
• Diagnose outliers using p95/p99 traces; watch for fan-out, N+1 queries, or lock contention.
• Enable trace-log correlation so clicking a trace reveals its logs.

Error Reporting

• Auto-aggregates exceptions by stack signature per service/version. Configure service context to avoid cross-service conflation. Suppress noisy known errors or channel them to lower-priority notifications.
• Redact PII from exception messages; log stable error codes and correlation IDs instead.

Cloud Profiler

• Low-overhead continuous CPU/heap profiling for supported runtimes. Compare profiles between versions and traffic levels to catch regressions. Avoid interpreting sampling artifacts as exact counts.

Cloud Debugger

• Snapshots capture variables at a code location without pausing the process. Logpoints inject temporary logging statements. Restrict access, redact sensitive variables, and scope to non-PII expressions.

Short example: add W3C traceparent and correlate a log

• HTTP propagation: traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
• Log field to link to Cloud Trace: “logging.googleapis.com/trace”: “projects/myproj/traces/4bf92f3577b34da6a3ce929d0e0e4736”

Reliability Engineering, Alerting, and Health Signals

SLIs, SLOs, SLAs, and error budgets

• SLIs measure user happiness: availability, latency, correctness. Define per critical endpoint and user journey.
• SLOs set targets, e.g., 99.9% of requests under 300 ms over 30 days.
• Error budgets quantify allowable unreliability. Spend budgets on releases, experiments, or migrations; freeze changes if burn rate is too high.
• SLAs are external commitments; keep SLO more stringent than SLA to protect margin.

Alert-quality design

• Prefer SLO- and symptom-based alerts over cause-based where possible.
• Use multi-window, multi-burn-rate alerts (for example, 14x over 5 minutes and 2x over 1 hour) to catch fast and slow burns while reducing noise.
• Add metric absence for watchdogs (e.g., long-running job heartbeat).
• Route by severity; rate-limit notifications; provide links to runbooks and dashboards.

Health checks and probes

• Readiness checks gate traffic until dependencies are ready; liveness checks trigger restarts on wedged processes; startup probes protect slow starters from premature restarts.
• For load-balanced VMs, allow health checker source ranges or traffic never reaches backends: gcloud compute firewall-rules create allow-lb
  –network prod –allow tcp
  –source-ranges 130.211.0.0/22,35.191.0.0/16 –direction INGRESS
• Kubernetes example: readinessProbe: httpGet: { path: /ready, port: 8080 } periodSeconds: 5 failureThreshold: 3 livenessProbe: httpGet: { path: /healthz, port: 8080 } initialDelaySeconds: 30 periodSeconds: 10
• Synthetic testing. Use uptime checks and bespoke end-to-end flows via Cloud Scheduler + Cloud Run/Functions to validate login, payments, or other critical paths.
• Dependency monitoring. Track database connection saturation, RPC error ratios, queue backlogs, egress errors, and third-party SLIs. Set retries with truncated exponential backoff for transient 429/5xx, with idempotency keys for safety.

Incident Response, Security-Conscious Debugging, and Root Cause

Incident response lifecycle

• Triage: classify severity, assign incident commander, and page on-call via defined channels.
• Contain: apply known mitigations and traffic controls (roll back, canary, circuit breaker, rate limiters).
• Communicate: maintain an internal war-room, update stakeholders regularly, and publish user-facing status when required.
• Resolve and recover: verify health via SLIs; avoid premature “all clear.”
• Postmortem: blamelessly analyze timeline, detection gaps, contributing factors, and action items with owners and due dates. Track to closure.

Runbooks

• Include triggers, required context, diagnostic commands, safe mitigations, rollback steps, and escalation paths. Link to dashboards, logs, and playbooks for specific failure modes.

Quota and capacity monitoring

• Monitor service quotas via Cloud Monitoring metrics. Automate alerts at 70–80% utilization and pre-request increases for planned load tests or launches.
• Capacity signals to track: CPU, memory, file descriptors, thread pools, DB connections, autoscaler limits, and request queue depth.

Debugging without exposing sensitive information

• Redact secrets and PII at source; centralize secrets in Secret Manager. Use hashing or tokenization for user identifiers. Enable field-level redaction in logging middleware.
• Limit access to logs, traces, and debug tools via IAM; use CMEK and VPC Service Controls where applicable.
• In Debugger, disable collection of large object graphs and add conditions to avoid capturing sensitive frames.

Root-cause analysis across layers

• Runtime: correlate spikes in GC, thread pools, or CPU via Profiler with p99 latency in Trace.
• Network: inspect load balancer logs, VPC Flow Logs, firewall logs, and Connectivity Tests to validate paths. Health-check failures commonly result from missing firewall rules or wrong ports.
• IAM: review Cloud Audit Logs for permission denials or policy changes; confirm service account roles and token scopes.
• Data: use Cloud SQL Insights, Spanner query stats, Bigtable CPU and hot tablets, and Storage error rates to find hotspots. Apply retries with backoff for transient faults and reduce fan-out that amplifies tail latency.
• Tie together with correlated trace IDs and log-based metrics; export to BigQuery to run multi-source joins during post-incident analysis.

Practical Problem Scenario

Fjord Retail migrates a multi-service checkout to Google Cloud using Cloud Run, Cloud SQL, Pub/Sub, and an external tax API. Users report intermittent timeouts and spiky error rates during flash sales, and on-call receives noisy, low-signal alerts.

Approach:

1. Instrument structured logging with trace correlation

• Add W3C traceparent propagation across services and include logging.googleapis.com/trace in all logs. Rationale: end-to-end correlation lets engineers pivot from a slow user request to the exact slow RPC or query and its logs.

2. Create Logging buckets, retention, and exports

• Increase _Default retention to 90 days and create a regional bucket for EU workloads. Add an aggregated sink to BigQuery for ERROR and WARNING logs: gcloud logging sinks create bq-prod
  bigquery.googleapis.com/projects/fjord/datasets/ops_logs
  –log-filter=‘severity>=WARNING’ –include-children Rationale: sufficient hot retention aids debugging; BigQuery enables rapid incident analytics without inflating hot storage costs.

3. Define SLIs/SLOs and SLO-based alerting

• Availability SLI: successful requests / total. Latency SLI: p95 duration for POST /checkout.
• SLOs: 99.9% availability monthly; 95% of checkouts < 300 ms.
• Configure multi-window burn-rate alerts and a metric-absence alert for the checkout heartbeat. Rationale: symptom-based alerts reduce noise and page only for user impact.

4. Set health probes and synthetic checks

• Cloud Run services expose /ready and /healthz. Add a global uptime check for /checkout and a private synthetic job in VPC that performs a full checkout with test credentials. Rationale: readiness prevents cold backends from receiving traffic; synthetics catch end-to-end issues and third-party regressions.

5. Enable Cloud Trace and Profiler, and adopt retries with backoff

• Install Trace/Profiler agents where applicable; enable automatic HTTP client instrumentation and SQL span annotation. Implement truncated exponential backoff with idempotency keys for tax API calls. Rationale: tracing isolates latency contributors; backoff reduces 429/5xx amplification and protects error budgets.

6. Monitor capacity and quotas

• Add dashboards and alerts for Cloud SQL connections, CPU, InnoDB buffer pool, Pub/Sub unacked messages, Cloud Run concurrency, and service quota usage. Rationale: capacity saturation is a common hidden cause of tail latency; early alerts prevent outages.

7. Harden debugging for privacy

• Use hashed user IDs and exclude PII from error messages. Restrict Debugger to production with redaction rules and logpoints only. Rationale: maintain observability while complying with data minimization.

8. Build dependency monitors and circuit breakers

• Track the external tax API’s success rate and latency via custom metrics; trip a circuit breaker to cached tax rates when failures exceed a threshold. Rationale: isolate failures in third-party dependencies and maintain core checkout availability.

9. Prepare runbooks and escalation paths

• Document steps: verify SLO dashboards, check Trace service map for hot edges, inspect Cloud SQL Insights for slow queries, verify firewall and health checks, and evaluate quota headroom. Include rollback and canary procedures. Rationale: consistent, fast response reduces MTTR and avoids ad-hoc risky changes.

10. Post-incident analytics pipeline

• Use BigQuery exports to compute per-tenant error rates and to correlate logs with traces and Cloud SQL insights by trace ID. Rationale: durable, queryable history enables accurate RCAs and prevention measures.

This plan elevates signal quality, shortens time to detect and resolve, protects user experience during spikes, and enforces privacy while debugging in production.

← Continuous Delivery · All domains · Performance →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →