Google PDE: Messaging, Event Ingestion and Real-Time Services — Study Guide

By Laurynas Brazys · 21 August 2026 · #google #PDE #certification #exam-prep #study-guide

Part of the Google Professional Data Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Messaging, event ingestion, and real-time services on Google Cloud center on Cloud Pub/Sub and Eventarc for decoupled, durable transport; Dataflow for stateful stream processing; and sinks such as BigQuery, Cloud Storage, and operational databases. Designing for at-least-once delivery, idempotent consumption, and observability ensures resilient systems that scale elastically while maintaining correctness under failure, backpressure, and schema evolution.

Core Messaging with Pub/Sub

• Topics and subscriptions
  • Publishers send messages to a topic; subscribers attach via subscriptions (multiple subscribers can independently consume the same messages).
  • Subscription types:
    • Pull: clients explicitly pull messages; use streaming pull for highest throughput and fewer round trips.
    • Push: Pub/Sub delivers via HTTPS; your endpoint must return 2xx to acknowledge.
    • Export to BigQuery: a BigQuery subscription delivers messages to a BigQuery table without code; best when payloads match declared schema and low-latency ingestion into analytics is required.
• Ordering keys
  • Enable message ordering on the topic and subscription to receive in-order delivery per ordering key. Throughput per key is serialized: one in-flight message per key can block subsequent ones; use many keys (for example, hash(device_id)) to scale.
• Fan-out and replay
  • Create separate subscriptions for different consumers to isolate workloads and retention.
  • Seek or snapshot to replay from a timestamp or snapshot for recovery and backfills.

Trade-offs:

• Ordering reduces parallelism and throughput per key; disable ordering unless strictly needed.
• Push simplifies client code but introduces HTTP endpoint scaling, security, and backoff concerns; pull gives more control and stability at high throughput.

Delivery Semantics, Acknowledgment, Retention, and Dead Lettering

• Acknowledgment and deadlines
  • At-least-once delivery: duplicates can occur.
  • Each delivery has an ack deadline (default 10 seconds). Extend (ModifyAckDeadline) while processing long-running work; failure to ack before the deadline is the most common cause of duplicate push deliveries.
  • Nack or deadline expiry makes the message eligible for redelivery.
• Retention
  • Unacknowledged messages are retained for the subscription’s ack deadline and retried; acknowledged messages can be retained up to the topic’s message retention duration for replay. Configure retention to cover your maximum outage plus recovery time.
• Retries
  • Pull: redelivery happens after ack deadline expiry; control concurrency with flow control limits.
  • Push: exponential backoff; only HTTP 2xx is success. 3xx/4xx/5xx trigger retries. Implement idempotent handlers to tolerate repeats.
• Dead-letter topics (DLTs)
  • Configure a DL topic and max delivery attempts per subscription to quarantine poison messages.
  • Monitor DLQ volume; create triage workflows and re-publish to the main topic after correction.

Example: gcloud pubsub subscriptions create orders-sub
–topic=orders
–dead-letter-topic=orders-dlt
–max-delivery-attempts=10

Delivery semantics summary:

• Pub/Sub: at-least-once, best-effort ordering within an ordering key if enabled.
• Sinks: BigQuery insert APIs provide duplicate mitigation (insertId or Storage Write API stream offsets), but still design consumers and writers to be idempotent.

Schemas, Compatibility, and Validation

• Pub/Sub schemas
  • Native support for Avro and Protocol Buffers with schemas stored centrally.
  • Topic-level schema settings: encoding (Avro or Protobuf) and enforcement (none, validate-only, or require).
  • Producer publishes encoded payloads; Pub/Sub validates against the current schema when enforcement is enabled.
• Evolution and compatibility
  • Use backward-compatible changes (add optional fields, add fields with defaults in Avro, never reuse tags in Protobuf, avoid removing or renaming fields).
  • Version schemas explicitly. For breaking changes, dual-publish to v1 and v2 topics, or add a version field and route accordingly.
• Producer–consumer contracts
  • Consumers should ignore unknown fields and default missing ones.
  • Test schema compatibility across all consumers before promotion; validate in staging subscriptions with the same schema enforcement as prod.

Short Avro example (excerpt): { “type”:“record”,“name”:“Order”, “fields”:[ {“name”:“order_id”,“type”:“string”}, {“name”:“ts”,“type”:{“type”:“long”,“logicalType”:“timestamp-micros”}}, {“name”:“amount”,“type”:[“null”,“double”],“default”:null} ] }

Event-Driven Integration, Eventarc, and Kafka Interoperability

• Eventarc and CloudEvents
  • Eventarc routes events from Google Cloud services (and custom sources via Pub/Sub) to Cloud Run, GKE, or Workflows using the CloudEvents spec. Attributes like type, source, subject enable fine-grained filtering and auditability.
  • Use attribute filters to minimize fan-out and reduce downstream load.
  • Delivery is at-least-once; make handlers idempotent and stateless where possible.
• Eventarc trigger example: gcloud eventarc triggers create gcs-finalize-to-run
  –destination-run-service=ingestor
  –event-filters=“type=google.cloud.storage.object.v1.finalized”
  –event-filters=“bucket=my-data-bucket”
  –service-account=eventarc-sa@PROJECT_ID.iam.gserviceaccount.com
• Kafka interoperability and managed migration
  • Dataflow templates connect Kafka <-> Pub/Sub for phased migration. Mirror topics with preserved keys; cut over consumers first, then producers, or dual-write during transition.
  • Pub/Sub Lite offers partitioned, capacity-provisioned streaming with key-based routing and lower cost; it is regional/zonal and suitable for Kafka-like workloads where predictable capacity and per-partition ordering are primary concerns.
  • Migration considerations:
    • Ordering: map Kafka keys to Pub/Sub ordering keys or Lite partitions.
    • Offsets: carry offsets as message attributes for diagnostics; consumers cannot rely on Kafka offsets after migration.
    • Delivery: accept at-least-once; enforce idempotency downstream.
    • Schemas: migrate Confluent Schema Registry definitions to Pub/Sub schemas or standardize on Protobuf/Avro with compatible evolution rules.

Streaming Ingestion Patterns, Throughput, Scaling, Security, and Operations

• Real-time ingestion patterns
  • Pub/Sub -> Dataflow -> BigQuery: use the BigQuery Storage Write API sink for high throughput and idempotency with stream offsets; route failures to a dead-letter table for inspection.
  • Pub/Sub -> Dataflow -> Cloud Storage: archive raw events for reprocessing; use windowed, compressed writes to balance cost and latency.
  • Pub/Sub -> operational stores: write to Bigtable for low-latency lookups, Spanner for strongly consistent transactions, or Cloud SQL/Firestore based on workload needs. Ensure idempotent upserts keyed by a unique event ID.
• At-least-once, duplicate prevention, and idempotency
  • Carry a unique event_id and event_time in every message; enforce producer-side UUIDs.
  • BigQuery streaming de-dup: set insertId or use Storage Write API with ordered streams; still guard queries with dedup logic.
  • Query-time dedup example: WITH ranked AS ( SELECT t.*, ROW_NUMBER() OVER (PARTITION BY event_id ORDER BY event_time DESC) AS rn FROM dataset.events t ) SELECT * EXCEPT(rn) FROM ranked WHERE rn = 1;
  • For push endpoints, return 2xx only after successful processing; otherwise expect redelivery.
• Message throughput, quotas, and scaling
  • Publishers: batch messages and reuse connections; parallelize across multiple clients. Use many ordering keys to scale ordered workloads.
  • Subscribers: prefer streaming pull with flow control (max outstanding bytes/messages). Size ack deadlines to processing time and extend when needed.
  • Monitor and request quota increases for publish and subscribe throughput as volumes grow; design for headroom (for example, 2x expected peak) to absorb bursts.
• Consistency and availability
  • BigQuery streaming is eventually consistent for query visibility; for interactive queries that must include streamed rows, wait based on observed latency (for example, 2x the P50 availability delay) or design using watermark-aligned aggregations in Dataflow and query materialized results.
• Security
  • IAM: grant least-privilege roles (pubsub.publisher to producers on the topic; pubsub.subscriber to consumers on the subscription). Use dedicated service accounts for each workload.
  • Push auth: configure push subscriptions to attach OIDC tokens from a service account; enforce audience validation on the endpoint. Prefer Cloud Run private endpoints for built-in auth and TLS.
  • Encryption: Pub/Sub encrypts in transit and at rest; use CMEK on topics for customer-managed keys. Apply VPC Service Controls to reduce data exfiltration risk. Use client-side encryption for sensitive payload fields if needed.
• Operational diagnosis of lag, redelivery, and subscriber failure
  • Monitor with Cloud Monitoring:
    • subscription/num_undelivered_messages and oldest_unacked_message_age for backlog.
    • expired_ack_deadline_count to detect missed acks causing duplicates.
    • publish_request_count and pull_request_count for throughput.
  • Investigate missing-dashboard events by replaying a known dataset through the pipeline and comparing stage-by-stage outputs to isolate the faulty transform or sink.
  • For Dataflow streaming:
    • Use autoscaling with an appropriate maxWorkers to absorb load from many sources.
    • Drain pipelines for incompatible updates to allow in-flight work to complete and prevent data loss.
  • For BigQuery insert notifications, route Cloud Logging audit entries via a sink filtered to specific tables to a Pub/Sub topic for alerting.

Practical Problem Scenario

Contoso Freight needs a global, real-time eventing platform to ingest 10,000 IoT telemetry messages per minute from trucks, enrich events, power interactive analytics, and trigger workflows on file drops from external partners. Some partner CSVs contain malformed rows, and the analytics team must inspect errors without blocking the stream.

1. Create the core messaging and schema layer

• Action: Define an Avro schema for telemetry and attach it to a Pub/Sub topic telemetry with schema enforcement set to require. Enable message ordering and publish with ordering_key = hash(device_id).
• Rationale: Topic-level schema enforcement rejects malformed events early. Per-device ordering supports ordered processing when needed, while hashing spreads keys to preserve throughput.

2. Provision subscriptions with isolation and dead-lettering

• Action: Create a pull subscription telemetry-stream-sub for Dataflow with a dead-letter topic telemetry-dlt and max_delivery_attempts=10. Add a BigQuery subscription telemetry-raw-bq to land raw events in a time-partitioned table for lineage and replay.
• Rationale: DLQ isolates poison messages for investigation. A separate BigQuery subscription provides a low-ops export path for raw event preservation independent of the processing pipeline.

3. Build a Dataflow streaming pipeline for enrichment and sinks

• Action: Ingest from telemetry-stream-sub using streaming pull with flow control. Validate against the schema, enrich with reference data, and compute windowed aggregates. Write to BigQuery using the Storage Write API with a named stream and insertId = event_id; write raw backups to Cloud Storage hourly; redirect bad/failed records to a dead-letter BigQuery table.
• Rationale: Storage Write API yields high-throughput, low-latency writes with idempotency via insertId/stream offsets. A dead-letter table supports inspection without blocking the stream, and Cloud Storage archives enable replay.

4. Handle duplicates and eventual consistency in analytics

• Action: For interactive queries that must exclude duplicates, publish event_id and event_time in every record and use a dedup view: CREATE OR REPLACE VIEW analytics.latest_events AS SELECT * EXCEPT(rn) FROM ( SELECT e.*, ROW_NUMBER() OVER (PARTITION BY event_id ORDER BY event_time DESC) rn FROM analytics.events e ) WHERE rn = 1; Introduce a brief query delay based on observed BigQuery streaming availability (for example, twice the median latency).
• Rationale: At-least-once delivery requires idempotent writes and query-time dedup. Waiting reduces misses from in-flight rows given streaming visibility latency.

5. Integrate partner file drops with Eventarc

• Action: Configure Eventarc to route Cloud Storage object.finalized events for bucket partner-drops to a Cloud Run service that launches a batch Dataflow job to load CSVs into BigQuery, sending parse errors to a dead-letter table.
• Rationale: Eventarc provides event-driven orchestration with CloudEvents filtering on bucket and object prefix. A batch Dataflow job separates malformed rows for analysis while loading good data promptly.

6. Secure the platform

• Action: Use distinct service accounts: producers get pubsub.publisher on telemetry; Dataflow worker SA gets pubsub.subscriber on telemetry-stream-sub and write access to target BigQuery datasets and Cloud Storage; Eventarc trigger uses a dedicated SA with invoker on Cloud Run. Enable CMEK on the telemetry topic and BigQuery datasets. Configure push endpoints, if any, with OIDC and audience checks.
• Rationale: Least-privilege IAM and CMEK meet security and compliance requirements; authenticated delivery prevents spoofing.

7. Operate and scale reliably

• Action: Set Dataflow autoscaling with a generous maxWorkers to absorb peaks. Monitor subscription/oldest_unacked_message_age and expired_ack_deadline_count; alert when thresholds are exceeded. For pipeline changes that break compatibility, deploy with drain to avoid message loss. If lag grows, increase subscriber parallelism and extend ack deadlines proportionally to processing time.
• Rationale: Proactive monitoring detects lag and redeliveries early. Autoscaling and tuned ack deadlines prevent duplicate storms. Draining preserves in-flight messages during upgrades.

This design delivers resilient, secure, and observable real-time ingestion with event-driven batch integration, supports duplicate tolerance and schema evolution, and offers fast analytics while isolating bad data for targeted remediation.

← Stream Processing with Dataflow and Apache Beam · All domains · Spark →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →