Google PDE: Stream Processing with Dataflow and Apache Beam — Study Guide

By Laurynas Brazys · 21 August 2026 · #google #PDE #certification #exam-prep #study-guide

Part of the Google Professional Data Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Stream processing on Google Cloud centers on Apache Beam’s unified programming model executed by the Dataflow runner. Beam provides a logical abstraction—pipelines of transforms over PCollections—that decouples your code from execution details such as parallelism, autoscaling, and fault tolerance. In streaming, correctness hinges on time semantics (event time vs processing time), windowing (fixed, sliding, session, global), watermarks, triggers, and handling late data. Operational excellence on Dataflow requires the right worker sizing, autoscaling policy, streaming engine, shuffle choices, idempotent sink design, dead-letter handling, and robust observability.

Apache Beam model and time semantics

• Pipelines, transforms, PCollections, runners:

  • A Beam pipeline applies a directed acyclic graph of PTransforms to PCollections (bounded or unbounded).
  • Runners (Dataflow, Spark, Flink, Direct) execute the pipeline; Dataflow provides managed autoscaling, checkpointing, and operational visibility.
  • Transforms include element-wise (ParDo), grouping and combining (GroupByKey, Combine), joins (CoGroupByKey), and IOs (PubSubIO, BigQueryIO, FileIO).

• Windows:

  • Fixed windows: non-overlapping slices (e.g., 1-minute tumbling windows) for periodic aggregates.
  • Sliding windows: overlapping windows for smooth rolling metrics (e.g., 5-minute windows sliding every 1 minute).
  • Session windows: dynamic windows that close after a gap of inactivity, ideal for user sessions or device bursts.
  • Global window: the default unwindowed view of the entire unbounded stream; often paired with triggers for periodic materialization.

• Event time vs processing time:

  • Event time: when the event occurred at the source; enables logically consistent aggregations despite variable transport latencies.
  • Processing time: when the event is observed by the pipeline; useful for operational triggers but not for semantic correctness.

• Watermarks:

  • A watermark estimates event-time completeness (the runner’s guess that it has seen all events up to time T).
  • Watermarks can advance irregularly or stall under backpressure or source delays; late data is anything arriving with timestamp < watermark.

• Triggers and lateness:

  • Default: AfterWatermark trigger that fires when the watermark passes window end; with allowed lateness = 0, late data is dropped.
  • Early firings (processing-time or count-based) give low-latency preliminary results.
  • Late firings allow corrections when late data arrives; accumulation mode governs whether panes accumulate results or discard prior output.
  • Choose allowed lateness based on business tolerance and storage/compute trade-offs; more lateness increases state retention and cost.

• Stateful processing, timers, sessionization, deduplication:

  • Stateful DoFns hold per-key state (e.g., last seen event, running aggregates) and set timers to emit or clear state.
  • Sessionization is naturally expressed via SessionWindows; for custom logic, use keyed state and processing/event-time timers.
  • Deduplication: use a stable id per event and either Distinct/Combine per window, or per-key state (e.g., Bloom filter or set with TTL). Trade-off memory and false positives vs strict accuracy.

Failure modes and trade-offs:

• Using processing time windows for business metrics causes drift under spikes or retries; prefer event-time windows.
• Too-small windows with frequent early triggers cause excessive pane emissions and sink write amplification.
• Unlimited allowed lateness can bloat state; always bound state TTL and set timers to clear dormant keys.

Operating Dataflow for streaming workloads

• Worker sizing and autoscaling:

  • Horizontal autoscaling adds/removes workers based on backlog, watermark lag, CPU, and throughput; set sensible maxWorkers to absorb spikes.
  • Pick machine types for bottlenecks: CPU-bound (more vCPUs), memory-bound (high-memory types), network-bound (larger VMs reduce shuffle overhead).
  • Increase boot disk for heavy shuffles or file-based sinks. Monitor system lag and backlog seconds.

• Streaming Engine and shuffle:

  • Streaming Engine externalizes state and shuffle to the service backend, improving elasticity, reducing worker memory pressure, and enabling faster updates.
  • For batch-heavy stages or massive key-grouping, use Dataflow Shuffle to offload shuffle I/O from workers. Both reduce hot-worker failures and disk thrash.

• Backpressure, hot keys, and skew:

  • Dataflow manages backpressure via dynamic work rebalancing; nonetheless, tune source flow control (e.g., Pub/Sub message/byte outstanding) when applicable.
  • Hot keys (e.g., popular ids) create stragglers. Mitigate with key sharding (key#N), partial pre-aggregation then re-keying, or sketch-based approximations.
  • Skew from outlier records (huge payloads) or bursty publishers may require per-publisher partitions, batching, or compression.

• Pub/Sub integration:

  • Use Pub/Sub topics for ingestion; enable message attributes for metadata (e.g., deviceId, event timestamp).
  • Ingest with PubSubIO; extract event timestamps from attributes or from payload, else fallback to publish time.
  • Ordering keys provide per-key ordering; Dataflow still needs idempotent downstream behavior due to at-least-once delivery.

• Streaming to BigQuery patterns:

  • Prefer BigQueryIO with the Storage Write API for high throughput, low-latency “exactly-once” semantics within a stream via stream offsets and automatic retries.
  • For low-rate simple pipelines, streaming inserts are acceptable; set insertId to dedupe client retries.
  • Queries over streaming buffers are eventually consistent; for time-critical analytics, query after a buffer delay (e.g., wait ~2x observed availability latency), or materialize via micro-batch windows and Storage Write API committed mode.

• Exactly-once effects, idempotency, replay, and sinks:

  • Beam guarantees at-least-once processing; “exactly-once” must be achieved at the sink using idempotent writes, transactions, or deduplication keys.
  • BigQuery: use Storage Write API default streams or committed streams for exactly-once within a stream; with streaming inserts, set a stable insertId.
  • Files: write temp files with unique names, finalize on window completion, and ensure atomic renames; avoid overwriting to prevent partial duplicates.
  • External databases: use upserts keyed by a stable id or implement dedupe windows.
  • Design for replay: maintain deterministic transforms; ensure sinks de-duplicate on retry.

• Dead-letter handling, error routing, observability:

  • Wrap risky parsing/enrichment in try/catch within ParDo and emit failures to a dead-letter PCollection via TupleTag; include payload, error code, and context.
  • Route DLQs to BigQuery or Cloud Storage for analysis; consider a separate Pub/Sub topic for reprocessing.
  • Observability: use Dataflow job metrics (watermark lag, system lag, throughput), custom counters, distribution metrics, and per-step logs in Cloud Logging. Create alerting on lag and error rates in Cloud Monitoring. Use Error Reporting to aggregate exceptions.

• Performance tuning patterns:

  • Read efficiently: for BigQuery sources, prefer Storage Read API or query-based reads that select only needed fields and filters.
  • Combine lifting: use combiners to reduce shuffle volume before GroupByKey.
  • Side inputs: cache small reference data in memory; watch for fanout and update cadence.
  • Serialization: use compact schemas (Avro/Proto), and avoid excessive JSON parsing on hot paths.

Deployment, templates, and upgrade strategies

• Flex Templates:

  • Package pipelines into containerized, parameterized templates for reproducible deployments. Flex Templates support custom dependencies, GPU images, and environment isolation.
  • Externalize runtime parameters (e.g., input subscription, output table, dead-letter sink, maxWorkers) to enable environment-specific deployments.

• Pipeline updates and compatibility:

  • Dataflow supports in-place update for many streaming pipelines if transform names, state specs, and output types remain compatible. Use stable PTransform names.
  • For incompatible graph or state changes, perform a controlled cutover: start the new job, then drain the old job to finish in-flight work and stop reading new elements.

• Draining and snapshots:

  • Drain gracefully completes processing, writes remaining output, and terminates; coordinate with Pub/Sub retention or snapshots to avoid gaps.
  • To ensure continuity, you can create a Pub/Sub snapshot, start the new pipeline seeking to the snapshot or an appropriate timestamp, verify output, then drain the old job.

• Configuration examples:

  • Example windowing with early/late triggers and accumulation: window .into(FixedWindows.of(Duration.standardMinutes(1))) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings(AfterProcessingTime .pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(30))) .withLateFirings(AfterPane.elementCountAtLeast(1)) ) .withAllowedLateness(Duration.standardMinutes(10)) .accumulatingFiredPanes();

  • Example BigQueryIO with Storage Write API: BigQueryIO.writeTableRows() .to(“project:dataset.table”) .withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API) .withWriteDisposition(WriteDisposition.WRITE_APPEND);

• Common pitfalls:

  • Writing to file-based sinks in streaming without windowed writes can stall finalization; enable windowed writes and triggers.
  • Unbounded Growth: forgetting to bound state or allowed lateness can cause memory leaks and scaling failures.
  • Missing timestamps: not assigning event timestamps causes the pipeline to default to processing time and lose correctness under variable delays.

Practical Problem Scenario

NovaTrack Inc. ingests global IoT telemetry from 50,000 temperature sensors and must deliver minute-level aggregates, persist raw data, and surface a real-time dashboard. Occasional malformed messages and out-of-order delivery are expected. The solution must auto-scale, surface bad records for inspection, and support zero-downtime upgrades.

Approach:

1. Ingestion and time semantics

   • Create a regional Pub/Sub topic and per-region publishers with attributes deviceId and eventTs (RFC3339). Enable ordering keys by deviceId when feasible.
   • Rationale: Pub/Sub provides durable, elastic ingress with at-least-once delivery. Attaching event timestamps at the edge preserves true event time; ordering per device reduces intra-device reordering without central bottlenecks.

2. Dataflow streaming pipeline with event-time windows

   • Read from a dedicated subscription via PubSubIO, extracting eventTs as the Beam timestamp, falling back to publishTime if missing.
   • Apply FixedWindows of 1 minute with an early trigger at 30 seconds and late firings on each late element; set allowed lateness to 10 minutes and accumulating panes.
   • Rationale: Event-time windows ensure accurate minute aggregates; early firings feed the dashboard with sub-minute freshness; late firings correct aggregates as delayed data arrives. The lateness bound caps state size and cost.

3. Validation, enrichment, and dead-letter routing

   • Implement a ParDo that parses JSON, validates schema and ranges, and enriches with small static reference data via a side input loaded from BigQuery at job start.
   • Use TupleTags to emit valid records to the main output and failures to a dead-letter PCollection containing payload, error, deviceId, and parse timestamp; write DLQ to a partitioned BigQuery table.
   • Rationale: Side inputs keep reference data in-memory for low latency. Dead-letter capture allows inspection and targeted reprocessing of bad rows without blocking the main flow.

4. Aggregation and hot-key mitigation

   • Key by deviceId and compute per-minute avg/min/max with CombineFns. For top-N regional metrics, shard by region#N to avoid hot keys, then re-aggregate.
   • Rationale: Combiners minimize shuffle volume and cost; key sharding prevents single-key bottlenecks during regional fan-in.

5. Sinks and exactly-once effects

   • Write raw validated events and minute aggregates to BigQuery using BigQueryIO with the Storage Write API. Set a stable insert id based on deviceId + eventTs for idempotency in any custom retries.
   • Rationale: Storage Write API provides high-throughput, low-latency ingestion with exactly-once semantics within a stream. Stable ids ensure downstream dedup if replays occur.

6. Dashboard consistency strategy

   • The dashboard queries partitioned aggregate tables with a lookback of 2 minutes relative to the watermark or a fixed delay of 2x observed availability latency for streaming data.
   • Rationale: BigQuery streaming visibility is eventually consistent; deferring reads slightly prevents missing in-flight rows while retaining near-real-time behavior.

7. Operations: autoscaling and streaming engine

   • Enable Streaming Engine; set maxWorkers based on expected peak (e.g., 3x average), select a machine type sized for CPU-bound parsing and encryption, and increase boot disk to accommodate transient shuffle.
   • Monitor watermark lag, backlog seconds, CPU, and per-step throughput; alert on sustained lag and DLQ rate spikes.
   • Rationale: Streaming Engine externalizes state/shuffle for elasticity and simpler upgrades; right-sizing and monitoring prevent silent SLO breaches.

8. Deployment and upgrades with Flex Templates

   • Package the pipeline as a Flex Template with parameters: input subscription, output tables, DLQ table, maxWorkers, and region. For an incompatible change, start the new pipeline targeting the same topic with a new subscription, verify outputs, then drain the old job. Optionally create a Pub/Sub snapshot and seek the new subscription to the snapshot to guarantee no gaps.
   • Rationale: Flex Templates enable repeatable, parameterized deployments. A verified blue/green cutover with drain achieves zero data loss and minimal downtime.

9. Reprocessing and batch backfills

   • Store compressed Avro files of raw events in Cloud Storage via a side output; run a batch Dataflow pipeline to backfill or reprocess into BigQuery when models or schemas change.
   • Rationale: Durable raw archives support reproducibility and schema evolution without impacting the hot path.

This design yields correct, low-latency aggregates with bounded cost, clear error isolation, strong observability, and safe upgrade paths while handling out-of-order and late data at global scale.

← BigQuery Analytics and Warehouse Engineering · All domains · Messaging →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →