Google PCA: Operations, Observability and Platform Automation — Study Guide

By Laurynas Brazys · 5 September 2026 · #google #PCA #certification #exam-prep #study-guide

Part of the Google Professional Cloud Architect — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Operations, Observability, and Platform Automation on Google Cloud ensure that services are diagnosable, maintainable, and continuously improved while controlling security and cost. A cohesive design spans logging, metrics, tracing, auditability, runbooks, incident response, quota and capacity governance, and automation. The goal is actionable, low-noise signals tied to service-level objectives, coupled with deterministic automation that reduces toil and configuration drift.

Logging and Auditability

Cloud Logging centralizes logs from Google Cloud services, GKE, and VMs. Prefer structured logs (JSON) with consistent keys for request_id, user_id, service, version, latency_ms, and severity; structured data enables precise queries, logs-based metrics, and policy evaluation. On VMs and GKE nodes, install the Ops Agent (preferred) or legacy logging agent to collect system and application logs; ensure parsers emit JSON for your frameworks.

• Log buckets and retention: Use regionally located log buckets for data residency and CMEK. Default buckets include _Default and _Required; the latter stores Admin Activity, System Event, and Policy Denied audit logs with fixed long-term retention. Create dedicated buckets per data class (e.g., app, security, analytics) with tailored retention and CMEK. Longer retention improves forensics but increases cost; export for long-term archiving when retention in Logging is not required.

• Log sinks and exports: Route with the Log Router using sinks to BigQuery (analytics), Pub/Sub (SIEM or pipelines), and Cloud Storage (archival). Use partitioned BigQuery tables to manage volume and cost. Always grant the sink service account least-privilege write access to the destination to avoid silent failures. Avoid routing loops by not re-ingesting exported logs back into Logging.

• Queries and logs-based metrics: Use Logs Explorer with filters on logName, resource.type, severity, labels, and jsonPayload fields. Derive logs-based metrics (counter or distribution) for error-rate SLIs and latency histograms, supporting alerting. Control cardinality by normalizing high-variance fields.

• Cloud Audit Logs: Admin Activity (control-plane writes), Data Access (reads/writes of user data), System Event, and Policy Denied logs provide administrative observability. Data Access logs are high-volume and disabled by default for many services; enable only where needed and route to a bucket with appropriate retention and CMEK. Policy Denied logs help detect permission and org policy violations early. Ensure custodial segregation: security teams typically own and access audit logs, with restricted views for other teams.

• Failure modes and trade-offs:

  • Overly broad sinks explode costs in BigQuery; filter precisely and expire partitions.
  • High-cardinality JSON fields (e.g., full URLs) degrade queries; sanitize and extract normalized labels.
  • Insufficient retention impairs investigations; exports to GCS or BigQuery mitigate.
  • Missing agent or parser misconfigurations cause silent log loss; alert on agent heartbeat and ingest errors.

Example: create a regional log bucket with custom retention and export a filtered audit sink.

• gcloud logging buckets create app-logs –location=us-central1 –retention-days=30
• gcloud logging sinks create bq-audit-sink bigquery.googleapis.com/projects/PROJECT/datasets/audit –log-filter=“logName:cloudaudit.googleapis.com AND protoPayload.serviceName:*” –use-partitioned-tables

Monitoring, Tracing, and Application Diagnostics

Cloud Monitoring collects system and application metrics, supports dashboards, alerting, uptime checks, notification channels, and service-level objectives.

• Metrics and dashboards: Use built-in metrics for Google services and create custom metrics via the Cloud Monitoring API or OpenTelemetry. Emphasize the four golden signals: latency, traffic, errors, saturation. Apply labels judiciously; avoid unbounded label values. Use Metrics Scope to aggregate multi-project views. Use MQL for expressive queries when needed.

• Alerting and notification channels: Implement multi-window, multi-burn-rate alerts for SLOs to balance fast detection and noise reduction. Define notification channels (email, SMS, Pub/Sub, webhooks, third-party incident tools) and include runbook links and context in alert documentation. Use notification rate limits and incident autoclose to prevent alert storms.

• Uptime checks: Probe critical endpoints from multiple regions with TLS verification, DNS, and content match. Tie uptime checks to alerting and service SLOs. Remember that uptime checks do not validate internal dependencies; complement with synthetic transactions and internal health checks.

• SLOs and SLIs: Define SLIs for availability, latency, and correctness. Configure SLOs in Cloud Monitoring Service Monitoring and track error budgets. Alert on budget burn, not raw errors, to align with customer impact. Use release gates or progressive delivery to respect remaining error budget.

• Cloud Trace, Error Reporting, Profiler: Use distributed tracing across services with OpenTelemetry to annotate spans with request and dependency metadata. Adjust sampling dynamically per service and per-path to ensure coverage of critical flows while controlling cost. Error Reporting auto-aggregates exceptions from logs, deduplicates by stack trace, and triggers notifications. Profiler provides continuous CPU, heap, and wall-time profiles in production with low overhead; use it to eliminate hot paths and reduce cost.

• Failure modes and trade-offs:

  • Excess metric cardinality increases cost and slows queries; aggregate before emit.
  • Low trace sampling hides tail latency issues; sample at a higher rate for slow paths.
  • Misaligned SLOs (e.g., too strict) generate alert fatigue; iterate with real traffic data.
  • Uptime checks can pass while internal dependencies fail; use dependency-aware SLOs.

Platform Operations, Runbooks, and Incident Management

Operational rigor reduces mean time to detect, mitigate, and learn.

• Runbooks and escalation: Every alert must link to a deterministic runbook with preconditions, diagnostic steps, rollback, and communication templates. Define clear on-call rotations and escalation policies. Store runbooks in version control and test them.

• Incident management and postmortems: Use standardized severities, roles (incident commander, communications, ops, SME), and channels. Prefer chat ops and status pages for broadcast. Write blameless postmortems capturing timeline, contributing factors, detection gaps, customer impact, and concrete follow-ups tied to owners and dates.

• Quota management and capacity signals: Monitor Service Usage and serviceruntime quota metrics with alerts on usage ratio. Request quota increases proactively and align autoscaling limits with quotas. Use capacity signals such as CPU, memory, file descriptors, connection pools, and queue depth. For GKE, tune HPA/VPA and cluster autoscaler; for GCE MIGs, set cool-downs and predictive autoscaling where appropriate.

• Service health and troubleshooting: Combine Logs Explorer live tail, logs-based metrics, dashboards, and Trace to reduce MTTD. Enable VPC Flow Logs and Firewall Rules Logging for network triage; use VM serial console for boot failures. Keep packet capture and kernel tracing as break-glass procedures in runbooks.

• Failure modes:

  • Quota exhaustion looks like outages; alert on 80 percent usage and rate-limit upstream.
  • Autoscaling without prewarming causes cold starts; use minimum replicas for critical paths.
  • Lack of synthetic checks masks customer-visible failures; implement canary transactions.

Automation, Resource Inventory, Policy, and Drift

Automate repeatable tasks with least privilege and idempotency.

• Tooling: Use Cloud Shell for secure, ephemeral admin with persistent $HOME; place helper binaries in ~/bin for PATH persistence. Automate with gcloud, REST APIs, and client libraries. Use service accounts and workload identity to eliminate long-lived keys.

• Schedulers and orchestrators: Use Cloud Scheduler to trigger HTTP and Pub/Sub jobs on cron schedules. Use Workflows to orchestrate multi-service automation with retries, backoff, compensation, and timeouts. Ensure idempotency and add correlation IDs to logs.

• Routine automation examples:

  • Daily asset export to GCS and BigQuery for inventory and drift reports.
  • Automated SLO burn-rate calculation publishing to a dashboard.
  • Periodic policy evaluation against organization policies and IAM anomalies.

• Resource inventory and policy evaluation: Cloud Asset Inventory provides point-in-time and real-time change feeds of resources, IAM bindings, and org policies. Export to BigQuery for historical analysis and drift detection; subscribe to Pub/Sub for near-real-time policy violation triage. Use Policy Analyzer and Recommender to detect overly broad IAM and unused permissions. Enforce constraints with Organization Policy and validate configurations pre-deploy with policy-as-code.

• Configuration drift: Prevent drift with declarative IaC and continuous validation. On detection, either reconcile automatically or quarantine resources. Tag resources with provenance (e.g., deployment_id) to distinguish managed vs ad hoc.

• Observability architecture for security, reliability, and cost:

  • Security: Route audit logs to CMEK-protected, access-restricted buckets; export to a dedicated security project. Integrate SIEM via Pub/Sub.
  • Reliability: Drive dashboards and alerts from SLIs and traces; rehearse incident automation with workflows.
  • Cost: Control metric cardinality, tune log retention per bucket, partition BigQuery exports, and use Profiler to optimize hot paths.

Example: schedule a daily asset export and execute a workflow.

• gcloud asset export –content-type=resource –output-path=gs://ORG-SEC-BUCKET/daily/resources-$(date +%F).json
• gcloud scheduler jobs create http run-asset-scan –schedule=“0 3 * * *” –uri=“WORKFLOW_EXECUTIONS_API_ENDPOINT” –http-method=POST –oauth-service-account-email=scheduler-sa@PROJECT.iam.gserviceaccount.com

Practical Problem Scenario

Contoso Commerce is launching a multi-region GKE-based checkout platform. Requirements: auditable administration, SLO-driven alerts with minimal noise, end-to-end request tracing, automated nightly compliance inventory, and strong cost controls.

Approach:

1. Establish logging and audit foundations

• Create regional log buckets with CMEK for app, security, and analytics logs; set 30 days for app, 400+ days for security as required. Route Admin Activity, System Event, and Policy Denied logs to the security bucket; enable Data Access logs for payment services only.
• Rationale: Segregation by sensitivity reduces blast radius and cost; CMEK satisfies compliance; scoping Data Access avoids volume spikes.

2. Implement structured application logging and collection

• Deploy Ops Agent on GKE nodes and sidecar/daemonset collectors to ship app logs as structured JSON with correlation IDs (trace_id, span_id) and user/session labels sanitized for PII.
• Rationale: Structured logs enable precise queries, logs-based metrics, and join with traces; correlation IDs support distributed diagnostics.

3. Deploy distributed tracing, error aggregation, and profiling

• Instrument microservices with OpenTelemetry SDKs exporting to Cloud Trace; set higher sampling for checkout and payment paths. Enable Error Reporting for all runtimes and Profiler for CPU/memory-critical services.
• Rationale: Traces localize latency by service hop; Error Reporting groups stack traces to accelerate triage; Profiler reduces compute cost and tail latency.

4. Define SLIs/SLOs and configure alerting and dashboards

• Define SLIs: p90 and p99 checkout latency, availability of the checkout API, and payment success rate. Set SLOs (e.g., 99.9 percent availability, p99 latency under 800 ms). Configure burn-rate alerts (2 percent over 1 hour and 1 percent over 6 hours) with runbook links and PagerDuty channel; build dashboards displaying golden signals and error budget trends.
• Rationale: Error-budget alerts tie to customer impact and reduce noise; dashboards provide operational situational awareness.

5. Add external and internal health checks

• Configure multi-region uptime checks for checkout endpoints with content validation; add synthetic transaction checks for cart-to-payment flow. Integrate GCLB and GKE readiness probes.
• Rationale: Uptime checks verify customer-facing availability; synthetic flows detect dependency breakages.

6. Automate inventory, policy monitoring, and drift detection

• Create a Security project to receive Cloud Asset Inventory exports to GCS and BigQuery; enable real-time Pub/Sub feeds for IAM and org policy changes. Run Workflows nightly to compare desired-state manifests against current assets; open tickets or auto-reconcile low-risk drift.
• Rationale: Centralized inventory supports audits; continuous policy evaluation prevents privilege creep; automation curbs drift.

7. Govern quotas and capacity

• Monitor compute, load balancer, and API quotas with alerts at 70 and 85 percent usage. Pre-request higher quotas for target load; align GKE cluster autoscaler and HPA limits with quotas. Enable predictive autoscaling for MIG-based services where startup is slow.
• Rationale: Quota ceilings can masquerade as outages; proactive adjustments and aligned autoscaling prevent throttling under peak.

8. Optimize cost in observability

• Cap log retention by bucket, exclude verbose debug logs in production via Log Router filters, and export only necessary fields to BigQuery with partition expiration. Constrain metric label cardinality; use Profiler findings to downsize hot services.
• Rationale: Observability should be cost-effective; targeted retention and exports prevent runaway spend.

9. Prepare runbooks and incident practice

• Author versioned runbooks for each alert policy, including diagnostic queries, trace filters, rollback commands, and communications. Run game days to validate escalation and automation.
• Rationale: Deterministic, practiced responses reduce MTTR and improve reliability.

10. Validate and iterate

• Continuously review alert noise, adjust thresholds, and refine SLOs based on real traffic. Track postmortem action items to completion with owners and deadlines.
• Rationale: Observability and operations improve through measured feedback, reducing toil and increasing service quality over time.

← Migration · All domains · DevOps →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →