Google PCNE: Network Observability, Reliability and Troubleshooting — Study Guide

By Laurynas Brazys · 22 August 2026 · #google #PCNE #certification #exam-prep #study-guide

Part of the Google Professional Cloud Network Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Network observability on Google Cloud is the disciplined collection, correlation, and analysis of network signals that describe reachability, performance, and correctness across VPCs, load balancers, hybrid connections, and services. Reliability follows from designing for failure detection and safe remediation: instrument first-class telemetry, validate the control plane before touching the data plane, drill down with packet evidence only when necessary, and automate rollback. This section explains how to use Google Cloud tools and patterns to detect, diagnose, and prevent issues while minimizing risk during change.

Flow Logs, Logging, Monitoring, Metrics, and SLOs

VPC Flow Logs provide sampled, aggregated telemetry at the VM NIC after VPC firewall evaluation. They are not full packet captures and they do not replace firewall rule logging for explicit allow/deny evidence. Key controls:

• Sampling: 0.0–1.0. Higher sampling improves fidelity at the cost of log volume and potential cost.
• Aggregation interval: 5s–30m. Shorter intervals reduce time-to-detection but increase entries.
• Metadata: include or exclude instance and VPC metadata. Include for richer analytics, exclude to limit sensitive attributes.

Typical enablement on a subnet:

• gcloud compute networks subnets update SUBNET –region=REGION –enable-flow-logs –logging-flow-sampling=0.5 –logging-aggregation-interval=interval-5-min –logging-metadata=INCLUDE_ALL_METADATA

Export with log sinks for durable analysis and cross-project sharing:

• BigQuery for SQL analytics and long-term trend analysis.
• Pub/Sub for near-real-time pipelines to SIEM/IDS.
• Cloud Storage for archival.
• Example sink to BigQuery:
  • gcloud logging sinks create flowlogs-to-bq bigquery.googleapis.com/projects/PROJECT/datasets/DATASET –log-filter=‘resource.type=“gce_subnetwork”’

Firewall rule logging complements flow logs by recording allow/deny decisions and matched rules. To observe blocked traffic, add a deny-all rule with logging near the bottom of your rule set:

• gcloud compute firewall-rules create deny-all –network=VPC –priority=65500 –direction=INGRESS –action=DENY –rules=all –enable-logging

Cloud Logging allows structured querying, correlation with request logs, health-check logs, NAT logs, and load-balancer logs. Create log-based metrics for signals such as:

• Sudden increases in denied connections to backend tags (possible misconfigured allowlists).
• High volumes of SYN retransmits to a port (possible saturation or blackholing).
• NAT “no available ports” events (Cloud NAT exhaustion).

Cloud Monitoring aggregates metrics and provides dashboards, alerts, and SLOs:

• Metrics to watch: load balancer 5xx rate, backend latency, instance NIC bytes/packets, Cloud NAT allocated/used/overflow ports, Cloud Router BGP session status, VPN tunnel packet drops, Interconnect link utilization, packet loss and latency.
• Dashboards: build per-service and per-connectivity dashboards with shared templates across projects for consistent operations.
• Alerts: favor symptom alerts (error rate, latency, drop counters) with fast feedback, and cause alerts (BGP flap, link down) with paging when user impact is likely.
• SLOs: define user-centric SLOs (e.g., global HTTP success and latency) and burn-rate alerts to identify fast and slow burns. Use request logs as numerator/denominator via log-based metrics for precise SLO evaluation.

Trade-offs and failure modes:

• Low sampling or long aggregation hides microbursts and short-lived failures.
• Flow logs lack visibility into pre-firewall drops; rely on firewall rule logging for deny evidence.
• Excessive logging without filtering increases cost and can slow investigations; export and partition data smartly.

Network Intelligence Center and Advanced Diagnostics

Network Intelligence Center (NIC) provides proactive and structured diagnostics:

Connectivity Tests:

• Validates control-plane reachability across routes, firewall rules (including hierarchical policies), service accounts/tags, load balancers, Cloud NAT, and hybrid connectivity.
• Route diagnostics compute the selected next hop and report misconfigurations such as missing routes or asymmetric paths.
• Use before and after any network change to detect unintended blast radius. It models the control plane and doesn’t guarantee data-plane quality; pair with packet/metric evidence.

Performance Dashboard:

• Google-managed view of packet loss and latency across regions and to internet vantage points. Useful to detect macro events (regional or path-wide congestion) versus service-local issues.

Network Topology:

• Visualizes inter-project, inter-VPC, and hybrid connections and traffic volumes (leveraging logs) to identify hot spots, unexpected peering paths, and transitive behaviors you might not intend.

Firewall Insights:

• Detects shadowed rules, unused allows, overly permissive sources, and rules missing target tags/service accounts. It recommends tighter rules to reduce attack surface without breaking known flows.

Network Analyzer:

• Static and dynamic configuration checks across projects to surface conditions such as:
  • Health checks blocked by firewall (remember to allow Google health-check source IP ranges).
  • Load balancer backends in wrong regions or missing named ports.
  • Subnets with Private Google Access disabled blocking API access for instances without external IPs.
  • Routes that blackhole important prefixes or asymmetric routing across VPNs/Interconnects.

Operational guidance:

• Bake NIC checks into CI/CD for network changes and run on a scheduled cadence. Treat findings as reliability debt and prioritize fixes with risk-reduction impact.

Packet Mirroring, Load Balancers, and Hybrid Telemetry

Packet Mirroring:

• Mirrors VM traffic to a collector (appliance or managed IDS) for deep inspection. Scope by subnet, network tags, or service accounts; restrict to required protocols to control cost.
• Overheads and trade-offs: mirrored traffic egress incurs cost; excessive mirroring can stress collectors; do not mirror indiscriminately in production. Use time-bounded, narrowly scoped sessions for incidents.
• IDS integration:
  • Cloud IDS provides managed, out-of-band threat detection leveraging Packet Mirroring. Prefer it for rapid activation and maintenance reduction.
  • Third-party IDS appliances remain viable where specific signatures or vendor ecosystems are required.

Load balancer logs and health-check evidence:

• HTTP(S) load balancer request logs include method, URL, backend, response code, latency, and client IPs via X-Forwarded-For. Use these for client-path analysis because traceroute stops at Google Front Ends (GFEs).
• Enable logging on backend services and configure sampling to balance cost versus visibility. Turn on health-check logging to see probe results and failure reasons.
• Restricting client access: enforce at the backends by tagging instances and creating firewall rules that allow only approved client ranges and Google health-check IPs. For L7 threat mitigation and gradual rollout, use Cloud Armor rules in preview mode before enforcing.

VPN and Interconnect telemetry:

• Cloud VPN (HA VPN) metrics: bytes, drops, encryption errors, tunnel uptime, and BGP session state. Alert on packet drops, frequent DPD events, and BGP flaps.
• Throughput scaling: add tunnels to distinct peer IPs and distribute traffic; monitor headroom. If you need active/standby across Cloud Routers, prefer the on-prem MED attribute to bias path selection.
• Interconnect metrics: per-link utilization, CRC errors, and availability; watch for sustained utilization >60–70% and error spikes. Maintain spare capacity and diverse circuits. Investigate latency increases with retransmissions and queueing indicators.
• Saturation signals across edges: rising TCP retransmits, increased 99th percentile latency without code changes, NAT port overflow events, and queue occupancy alerts are early warnings of impending impact.

Troubleshooting Methodology, Incident Handling, and Proactive Reliability

Structured troubleshooting from DNS to app:

1. Identify the failing user journey and time window; pin to region and path (public via LB, private via VPC, or hybrid).
2. DNS:

• Validate resolution with Cloud DNS logs, dig output, and policy behavior. Check split-horizon conflicts and ensure forwarding policies are in effect.
• Confirm TTLs and recent changes; stale caches can mimic outages.

3. Load balancer and edge:

• Review request and health-check logs. Correlate 5xx spikes with backend health and deployment events. For L7, trust request logs over traceroute.
• Verify client allowlists and Google health-check IPs in firewall rules when access is restricted.

4. Routing and firewall:

• Use Connectivity Tests for deterministic control-plane evaluation. Inspect effective routes and hierarchical firewall policies. Look for asymmetric routing and shadowed rules.
• For denied packets, rely on firewall rule logging; consider adding a logged deny-all near the bottom of the priority stack during investigations.

5. Egress to Google APIs:

• If instances have no external IPs, confirm Private Google Access and/or Cloud NAT. Missing either results in intermittent failures and confusing timeouts.

6. Hybrid:

• Examine Cloud Router and VPN/Interconnect metrics. BGP up but routes not installed can reflect attribute preferences; verify MED/local-pref and ASNs. Watch packet drops and path MTU issues.

7. Packet evidence:

• If control-plane looks correct but symptoms persist, use Packet Mirroring narrowly to collect PCAP near the affected VM or tier; check SYN/SYN-ACK timing, retransmissions, and MSS/DF bits for MTU blackholes.

Incident handling:

• Change safety: stage changes by scoping to tags/service accounts, lower precedence, and disabled rules; enable logging and test with canaries. Use Cloud Armor preview for L7 policy changes.
• Rollback: predefine inverse changes, keep prior configs in version control, and use short-lived feature flags at the app edge where applicable.
• Post-incident analysis: construct a timeline from logs and metrics, classify contributing factors (e.g., permissive rule shadowed by deny, NAT exhaustion), record detection gaps, and add safeguards: alerts, NIC checks, and policy hardening.

Capacity planning and proactive reliability:

• Track headroom targets: 30–50% on VPN/Interconnect links, NAT port usage below 60% sustained, LB backend CPU and QPS well below autoscaling triggers.
• Build log-based metrics for key risks: deny spikes, NAT overflow, BGP flap counts, 5xx rates, and LB backend connection errors. Use multi-window burn-rate alerts to catch both fast and slow incidents.
• Synthetic monitoring: use Uptime Checks from multiple regions for public endpoints and private probes from test VMs for internal services.
• Preventive improvements: refine firewall rules using Firewall Insights, resolve Network Analyzer findings, shorten Flow Log aggregation during peak events, and export logs to BigQuery for recurring anomaly detection.

Practical Problem Scenario

Contoso Games operates a global HTTP(S) load-balanced gaming API in us-east1 and europe-west1, with HA VPN to an on-prem data center. Users in Europe report intermittent timeouts and higher latency after a recent firewall change. Instances have no external IPs and must reach Google APIs privately.

Approach:

1. Establish time window and SLO impact

• Rationale: Pinpointing the window confines queries to relevant logs and aligns the investigation with user-impacting SLOs. A burn-rate alert confirms a fast SLO burn in europe-west1.

2. Validate control plane with Connectivity Tests

• Rationale: Create a test from the external HTTP(S) load balancer frontend to the europe-west1 backend service and from affected VMs to a Private Google Access VIP. The test flags a hierarchical firewall policy blocking health checks to some backends and missing Private Google Access for one subnet.

3. Confirm edge and backend health via logs

• Rationale: Filter load balancer request logs for europe-west1 and response_code >= 500 to isolate backend errors. Health-check logs show probe failures from known Google health-check source IPs. This evidences firewall-induced backend flapping rather than application regressions.

4. Restore health and protect safety with scoped changes

• Rationale: Add an allow rule targeted to backend tags that permits health-check source ranges. Enable logging on this rule. Because access is restricted to known clients, verify the allowlist firewall rule includes only the specific client IP ranges and the health-check ranges. Keep the new rule initially disabled, then enable during a low-traffic canary to limit blast radius.

5. Re-establish private egress to Google APIs

• Rationale: Enable Private Google Access on the affected subnet so instances without external IPs reach Google services without hairpinning through VPN or third-party firewalls. This reduces latency and removes a choke point.

6. Check hybrid saturation and MTU

• Rationale: Review HA VPN metrics for packet drops and utilization. One tunnel shows elevated drops. Increase capacity by adding a second tunnel to a different on-prem peer IP and distribute traffic. Verify effective MTU and MSS clamp to prevent PMTU blackholes on the VPN path.

7. Use Cloud Armor preview for suspected abusive clients

• Rationale: From request logs, a small set of client IPs spikes traffic just before probe failures. Add a deny rule in Cloud Armor with preview mode to validate that blocking would reduce backend load before enforcement, avoiding accidental user impact.

8. Narrowly use Packet Mirroring to confirm data-plane behavior

• Rationale: Mirror traffic from a single unhealthy backend VM to Cloud IDS for 15 minutes. PCAP shows SYN backlog exhaustion during bursts from the abusive IPs, corroborating Cloud Armor policy utility and the need for rate limiting.

9. Close the incident and harden

• Rationale: After enabling the health-check allow rule, confirming Private Google Access, scaling VPN capacity, and enforcing the vetted Cloud Armor rule, error rates return to baseline. Add dashboards for health-check success rate, NAT port usage, VPN drops, and per-region 5xx. Create log-based alerts on firewall denies to backend tags and SLO burn-rate alerts. Record the incident, root causes (hierarchical firewall change; abusive traffic; VPN saturation), and add NIC Analyzer and Firewall Insights checks to the pre-change checklist.

This sequence demonstrates a safe, evidence-driven workflow: confirm control plane, observe data plane, apply minimal, reversible changes, and then institutionalize learnings with alerts and automated checks.

← GKE · All domains · Network Automation →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →