Google ACE: Monitoring, Logging and Operational Troubleshooting — Study Guide

By Laurynas Brazys · 31 August 2026 · #google #ACE #certification #exam-prep #study-guide

Part of the Google Associate Cloud Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Operational excellence on Google Cloud depends on converting telemetry into action. Monitoring, Logging, and Troubleshooting together provide the signals, guardrails, and workflows that keep services reliable. This section covers core observability services, diagnostic tooling, reliability practices, and incident operations with design rationale, trade-offs, and common failure modes.

Monitoring and Alerting Foundations

Cloud Monitoring ingests time series from Google Cloud services, agents, and custom metrics to provide dashboards, uptime checks, SLOs, and alerting.

• Metrics and cardinality

  • Monitored resource types (for example, gce_instance, aws_ec2_instance, global) scope dimensions like project, region, and instance ID.
  • Minimize unbounded label values (for example, user_id) to prevent high-cardinality explosions that slow queries and increase cost.
  • Prefer distribution metrics for latency (p50/p90/p99) and use alignment windows for consistent rollups.

• Dashboards

  • Use built-in service dashboards for quick starts. Create custom dashboards to group cross-project metrics. Organize panels by symptom (latency, errors, saturation) before cause (CPU, memory).
  • Avoid per-instance charts for fleets; aggregate by service, zone, or MIG to reduce noise and increase signal.

• Alerting policies

  • Design symptom-based alerts tied to user experience: availability SLO burn, latency percentiles, error rates. Use multi-window, multi-burn-rate alerts to catch fast and slow burn (for example, 2% over 1 hour and 5% over 5 minutes).
  • Set reasonable notification rate limits and auto-close behaviors. Use incident auto-mitigation annotations to document runbooks.
  • Use metric absence alerts for critical batch jobs and data pipelines with strict schedules.
  • Log-based metrics support alerts for application or security events (for example, repeated permissionDenied).

• Notification channels

  • Configure channels per severity: paging for SEV1 (on-call, SMS, phone), chat for SEV2/3, email or webhooks for low-priority. Use Pub/Sub for integration with ticketing or automation.
  • Test channels periodically; stale channels are a silent failure mode.

• Uptime checks and synthetic monitoring

  • HTTP(S) and TCP checks from global vantage points verify external reachability. Pair checks with content match to detect partial failures.
  • Use private uptime checks via hybrid connectivity or Private Service Connect for internal services.
  • Failure modes: checks can fail due to DNS TTL lag, TLS cert rotation issues, or region-scoped outages; corroborate with metrics before paging.

• Multi-project monitoring

  • Use a single Monitoring workspace and link all projects for consolidated dashboards and alerts. This simplifies fleet-wide SLOs and reduces duplication.

Logging, Audit, and Application Diagnostics

Cloud Logging is the log router, storage, and query plane for platform and application logs. Complementary APM tools (Error Reporting, Trace, Profiler) accelerate root-cause isolation.

• Log routing, buckets, sinks, and retention

  • Route with sinks to log buckets (default or custom), BigQuery (analytics), Pub/Sub (stream processing), or Cloud Storage (archival).
  • Use regionally scoped log buckets for data residency and performance. Apply CMEK if required by compliance.
  • Set retention per bucket (for example, 30–90 days for ops, multi-year for audit). Longer retention increases cost; filter aggressively to control spend.
  • Exclusions reduce ingestion of verbose logs (for example, health checks). Validate filters to avoid unintentionally dropping critical logs.

  Example: create a BigQuery sink for audit logs

  • gcloud logging sinks create bq-audit-sink bigquery.googleapis.com/projects/PROJECT_ID/datasets/audit_ds –log-filter=‘logName:cloudaudit.googleapis.com AND protoPayload.methodName:*’ –include-children

  Example: create an exclusion

  • gcloud logging exclusions create drop-health-checks –description=“Exclude load balancer health checks” –log-filter=‘resource.type=“gce_instance” AND httpRequest.requestUrl:"/healthz"’

• Queries and Log Explorer

  • Use advanced filters on resource.type, severity, labels, and JSON payloads. Save queries for common triage paths (startup failures, permissionDenied, quotaExceeded).
  • Create distribution and counter log-based metrics for alerting and dashboards.

• Cloud Audit Logs

  • Admin Activity logs: always on, no charge to ingest; records admin write operations (for example, createInstance).
  • Data Access logs: record read and write data-plane operations (for example, storage.objects.get). Disabled by default for many services; enable selectively because volume and cost can be high.
  • System Event logs: Google system actions (for example, autoscaler, maintenance live migration).
  • Policy Denied logs: explicit records of IAM and organization policy denials. Critical for access troubleshooting and security reviews.
  • Route audit logs to BigQuery for retention and investigation; index labels such as authenticationInfo.principalEmail for attributions.

• Error Reporting, Trace, and Profiler

  • Error Reporting automatically groups stack traces by service and version; integrate with notification channels for new error groups and sudden spikes.
  • Cloud Trace collects request latency distributions and spans; set sampling to balance overhead and fidelity (for example, 1 in 1000 for high-QPS services, with tail-based sampling if using OpenTelemetry collectors).
  • Profiler provides low-overhead continuous CPU/heap profiling in production. Limit to hot paths or representative workloads to control data volume and overhead. Use source mapping for readable call graphs.
  • Trade-offs: higher sampling improves diagnostics but increases cost and potential PII exposure; scrub sensitive fields and use tokenization.

Service, Resource, and Network Troubleshooting

Efficient troubleshooting moves from symptoms to system boundaries, then to resources and dependencies.

• Managed-service health, service status, quotas, regional incidents

  • Validate whether an incident is upstream: check service health and recent regional advisories. Look for error bursts, elevated latency, or quota errors.
  • Quotas are per-project and often per-region; quotaExceeded and rateLimitExceeded in logs indicate throttling. Request increases ahead of peak events.
  • Failure mode: partial regional outages can appear as intermittent errors; configure multi-regional failover where possible.

• Resource health and VM diagnostics

  • Use instance operations, maintenance events, and health checks. For MIGs, inspect autohealing restarts and health check failures to isolate bad images or configs.
  • VM serial console for boot and kernel messages:
    • gcloud compute connect-to-serial-port VM_NAME –zone=ZONE
    • Common causes: kernel panics, bad fstab entries blocking boot, incorrect network configs, OS Login misconfiguration causing SSH failures.
  • Ensure OS Login with per-user SSH keys and IAM roles (compute.osLogin or compute.osAdminLogin) for attributable access.

• Logs Explorer for failure analysis

  • Start with symptom logs (5xx, deadlineExceeded), pivot by resource labels, then correlate with deployment changes and quota logs. Use histogram views to locate change points.

• Network observability

  • VPC Flow Logs: per-VNIC sampling of 5-tuple traffic; enable on subnets. Tune sampling (for example, 0.5) and metadata options for performance versus detail.
    • gcloud compute networks subnets update SUBNET –region=REGION –enable-flow-logs –flow-sampling=0.5 –aggregation-interval=interval-5-min –metadata=include-all
  • Firewall logging: capture allow/deny decisions on critical rules to diagnose unexpected blocks or shadowing.
    • gcloud compute firewall-rules update RULE_NAME –enable-logging
  • Connectivity Tests: model and verify reachability across VPCs, peering, Cloud VPN, Cloud Interconnect, and firewall rules. Useful for pre-change validation and incident triage.
    • gcloud network-management connectivity-tests create test-a –source-instance=projects/PRJ/zones/ZONE/instances/VM1 –destination-ip=10.0.3.21 –protocol=TCP –destination-port=443
  • Complementary signals: Cloud NAT and load balancer logs for egress and edge issues. Failure modes include asymmetric routing, missing routes, misordered firewall rules, and policy constraints.

Reliability, SLOs, and Incident Operations

Operational discipline ties telemetry to user-impact objectives and consistent incident execution.

• SLOs, error budgets, and baselines

  • Define SLOs on user-centric SLIs (availability, latency, correctness). Example: 99.9% of read requests complete under 200 ms over 30 days.
  • Track error budgets and design rollup dashboards by service and release version. Gate rollouts on budget consumption.
  • Establish performance baselines before traffic ramps; regressions are detected by deviation, not absolute values.

• Alert-noise reduction

  • Prefer service-level over instance-level alerts. Use rate-of-change and percentile-based conditions. Apply notification throttling, incident auto-close, and alert muting for maintenance windows.
  • Deduplicate alerts via common labels and policies; use dependency-aware routing to avoid paging both on database and application for the same incident.

• Incident response workflow

  • Triage and declare severity; assign roles (incident commander, operations, communications, scribe).
  • Escalation paths: on-call rotations, subject-matter experts, and vendor support (include project ID, request IDs, timestamps, and regions in support tickets).
  • Communication: maintain a single source of truth (chat channel and incident document). Provide periodic stakeholder updates with impact, mitigation, and ETAs.
  • Mitigation playbooks: rollback, failover, feature flag disable, or capacity add. Prefer reversible changes with short blast radius.

• Post-incident review and RCA

  • Evidence-driven: correlate metrics, logs, traces, and change events. Include what detection signals fired, why or why not, and time-to-detect/mitigate.
  • Identify contributing factors, not just the proximate cause. Capture concrete action items with owners and deadlines; update runbooks and alerts accordingly.
  • Blameless culture encourages full disclosure and systemic fixes.

• Operational runbooks

  • Structure: trigger and detection, quick diagnosis tree, safe mitigations, rollback/restore steps, verification, and exit criteria.
  • Keep commands and filters copy-paste ready; verify least-privilege IAM for responders (for example, storage.objectCreator for write-only backups; dedicated service accounts for workload identity).
  • Version runbooks with change management; test during game days.

Practical Problem Scenario

Northwind Outfitters runs a regional e-commerce platform on Google Cloud. After a recent spike in traffic, users intermittently report checkout timeouts and slow product searches. The operations team needs to quickly restore reliability, reduce alert noise, and harden diagnostics across multiple projects.

Approach:

1. Consolidate monitoring across projects

• Action: Create a single Monitoring workspace and link the prod, payments, and search projects. Build a “User Journey” dashboard showing availability, latency, and error rates for checkout and search.
• Rationale: Centralized visibility supports service-level triage and correlates cross-service issues (for example, search latency cascading into checkout timeouts).

2. Implement SLOs and burn-rate alerting

• Action: Define SLOs: 99.9% checkout under 400 ms, 99.95% search under 250 ms. Create multi-window burn alerts (2% over 1 hour and 5% over 5 minutes) on latency and error-rate SLIs. Notify on-call via paging, stakeholders via chat.
• Rationale: Burn-rate alerts catch fast regressions and sustained slow burns without paging on normal variance.

3. Add synthetic uptime checks with content match

• Action: Configure TCP and HTTPS uptime checks for the public entrypoints and a private uptime check to the internal payment API, validating response body contains “ok”.
• Rationale: Detects reachability and partial failures like misrouted backends or degraded upstreams.

4. Tighten logging routes and retention

• Action: Create dedicated log buckets: ops (90 days), security-audit (2 years, CMEK). Route Admin Activity, Data Access, System Event, and Policy Denied logs to BigQuery via sinks for analytics. Add exclusions for chatty health checks.
• Rationale: Right-sized retention controls cost; BigQuery enables rapid forensics. Exclusions cut noise without losing critical evidence.

5. Enable application diagnostics

• Action: Instrument services with OpenTelemetry for Trace; enable Error Reporting for backend and frontend; roll out Profiler to checkout service with CPU and heap profiles at conservative sampling.
• Rationale: Traces identify latency hot paths; Error Reporting highlights new error groups; Profiler reveals CPU contention and memory leaks with low overhead.

6. Strengthen network observability

• Action: Enable VPC Flow Logs on prod subnets (0.5 sampling, include-all metadata) and firewall logging on allow/deny rules for ingress to search and payments. Create Connectivity Tests from web tier to search and from payments to Cloud SQL.
• Rationale: Flow and firewall logs expose drops, retransmits, and shadowed rules; Connectivity Tests validate reachability and identify misconfigurations.

7. Resource-level diagnostics and safe access

• Action: For unstable VMs, inspect boot issues with serial console:
  • gcloud compute connect-to-serial-port checkout-vm –zone=us-central1-a
  • If SSH is required, enforce OS Login and grant compute.osAdminLogin to the “ops-admins” group. Each admin uses their own SSH key.
• Rationale: Serial logs reveal kernel and init failures. OS Login with per-user keys ensures attributable, least-privilege access.

8. Quota and regional health verification

• Action: Review recent quotaExceeded events in logs; raise regional API and IP quota for search autoscaling. Check service health advisories for the impacted region; temporarily shift traffic with load balancer weighting.
• Rationale: Quota throttling and regional incidents are common intermittent failure sources; proactive scaling and traffic steering mitigate impact.

9. Noise reduction and runbook updates

• Action: Replace per-instance CPU alerts with service-level saturation alerts. Add maintenance windows to mute non-actionable alerts. Update runbooks with new triage queries, trace dashboards, and rollback procedures.
• Rationale: Cuts paging fatigue and accelerates consistent, safe responses.

10. Post-incident review and actions

• Action: Conduct a blameless review. Correlate burn-rate incidents with increased search tail latency and a recent index rollout. Action items: add canary releases for search, index size guardrails, and autoscaler headroom; commit to periodic alert channel tests.
• Rationale: Evidence-driven RCA prevents recurrence and improves detection, mitigation, and resilience.

← Deployment · All domains · Security →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →