Google PDE: Data Engineering Architecture and Design — Study Guide

By Laurynas Brazys · 21 August 2026 · #google #PDE #certification #exam-prep #study-guide

Part of the Google Professional Data Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Data engineering architecture and design on Google Cloud balances domain boundaries, processing patterns, and service capabilities to deliver reliable, scalable, and cost-efficient data platforms. Effective designs make storage, compute, orchestration, and serving independently scalable; codify contracts so domains interoperate; and validate risks early with measurable service-level objectives (SLOs). This section summarizes canonical architectural styles (data mesh, lake, warehouse, lakehouse, operational stores), processing modes (batch, micro-batch, streaming, event-driven, lambda), and the trade-offs among scalability, latency, availability, consistency, and cost. It also covers regional and multi-cloud placement, schema evolution, the end-to-end data lifecycle, workload-based service selection, and risk-driven validation practices tailored to Google Cloud.

Architectural Paradigms and Processing Patterns

• Data mesh, domains, and data products:
  • Empower domain teams to publish “data products” with clear ownership, SLOs, access policies, and documentation. Use Dataplex to define domains, govern metadata, and apply consistent policies across BigQuery and Cloud Storage. Products may expose BigQuery datasets, Pub/Sub topics, or Cloud Storage paths with contracts enforced via Pub/Sub schemas and BigQuery table schemas.
• Data lake:
  • Raw, open-format storage (Parquet/Avro) in Cloud Storage with lifecycle and versioning. Suits heterogeneous workloads (Spark on Dataproc, Dataflow, Presto/Trino) and multi-cloud portability. Trade-off: eventual consistency semantics in object stores; design for idempotency and metadata-driven deduplication.
• Data warehouse:
  • Curated, governed analytics in BigQuery. Optimized for ANSI SQL, separation of storage/compute, and fine-grained security. Trade-offs: streaming inserts exhibit brief query-time staleness; prefer batch-loads or insert with buffered queries for strict freshness SLAs.
• Lakehouse:
  • Blend open data lake storage with warehouse capabilities. On Google Cloud, store Parquet/Avro in Cloud Storage; use BigQuery external tables for economics and BigQuery managed tables for performance and governance. Dataflow or Dataproc maintains ACID-like merge semantics with partitioned/clustering strategies.
• Operational-store architecture:
  • Low-latency transactional or key-value stores backing applications. Choose Cloud SQL for traditional OLTP, Cloud Spanner for globally consistent SQL with horizontal scale, and Bigtable for very high throughput, wide-column access patterns. Separate operational stores from analytics; use CDC (Datastream) to capture changes into Pub/Sub, Cloud Storage, or BigQuery.

Processing patterns and when to use them:

• Batch: Periodic, large-scale transformations (e.g., nightly feature generation). Tools: Dataflow batch, Dataproc. Failure modes: long-running job timeouts, skew; mitigate with autoscaling and repartitioning.
• Micro-batch: Small, frequent batches (e.g., every minute) to balance freshness with stability and cost. In BigQuery, use scheduled queries or Dataflow with fixed windows.
• Streaming: Millisecond-to-seconds latency on unbounded data. Use Pub/Sub + Dataflow. Handle late/out-of-order events with event-time windows and watermarks; ensure idempotency to prevent duplicates.
• Event-driven: Triggered by changes (GCS finalize, Pub/Sub messages). Use Cloud Functions or Cloud Run for stateless reactions and Dataflow for stateful processing. Trade-off: per-event costs vs. throughput.
• Lambda pattern: Maintain both streaming and batch paths for accuracy and reprocessing. Complexity doubles; consider a Kappa-like simplification where everything is replayable from an immutable log (Pub/Sub to Cloud Storage archival).

Short Dataflow streaming configuration example for late data:

events
  .apply(Window.into(FixedWindows.of(Duration.standardMinutes(5)))
    .withAllowedLateness(Duration.standardMinutes(10))
    .accumulatingFiredPanes());

Domain Ownership, Data Products, and Contracts

• Ownership and SLOs:
  • Each domain team defines and operates its data products with availability, latency, and data-quality SLOs. Publish SLOs via Dataplex catalogs and monitor with Cloud Monitoring SLIs (e.g., on-time partition completeness).
• Contracts and interoperability:
  • Enforce schemas with Pub/Sub Schema Registry (Avro/Proto) and BigQuery table schemas. For CSV ingestion, validate in Dataflow and route malformed rows to a dead-letter table for triage. Interoperate with open formats in Cloud Storage and BigQuery external tables when multiple engines must read the same data.
• Schema evolution:
  • Favor backward-compatible changes: add nullable columns, add optional fields in Avro/Proto, avoid renames/drops without deprecation windows. Communicate changes via versioned contracts and deprecation schedules.
  • BigQuery example (backward-compatible column add):

ALTER TABLE sales.orders
ADD COLUMN coupon_code STRING;

• Consumer impact:
  • Maintain semantic versioning of schemas; publish both v1 and v2 during migration. For streaming, route to versioned topics or include a schema version field. Provide authorized views in BigQuery to insulate consumers from physical changes.
• Governance and lineage:
  • Use Dataplex and Data Catalog for metadata, tags (e.g., PII), and lineage. Apply row- and column-level security in BigQuery. For data loss prevention, integrate Cloud DLP in ingestion (e.g., Cloud Run or Dataflow transforms) to tokenize or redact sensitive fields before storage.

Nonfunctional Trade-offs and Deployment Topology

• Scalability:
  • BigQuery scales elastically for analytics; Bigtable scales linearly with node count but requires careful row-key design (e.g., hashed or rotated prefixes) to avoid hotspotting. Dataflow autoscaling responds to backlogs; design for backpressure by leveraging Pub/Sub flow control.
• Latency:
  • Streaming to BigQuery offers low-latency inserts but queries may see slight lag; design queries with a freshness buffer or watermark-based windows. For sub-100 ms reads at scale, precompute and serve from Bigtable or Memorystore.
• Availability and consistency:
  • Cloud Spanner provides strongly consistent, globally distributed SQL. Bigtable offers high availability with eventual consistency across clusters. BigQuery availability is regional or multi-regional; materialize critical datasets in multi-region for resilience.
• Cost:
  • Optimize BigQuery with partitioning and clustering to reduce scanned bytes. For small files over network-limited links, batch or bundle to reduce RPC overhead. Use BigQuery BI Engine for cached, interactive dashboards where appropriate.
• Regional, multi-regional, hybrid, and multi-cloud:
  • Regional design reduces latency and cost; multi-regional storage (e.g., BigQuery US/EU multi-region, Cloud Storage dual-/multi-region) increases durability and locality options. For DR, define RPO/RTO and replicate critical datasets. In hybrid scenarios, use Datastream for CDC and Transfer Appliances or Storage Transfer Service for bulk migration. For multi-cloud, standardize on open formats in Cloud Storage and use portable compute (Apache Beam/Dataflow, Spark on Dataproc) while acknowledging egress and operational overhead.

Layering, Lifecycle, and Service Selection

• Separation of layers:
  • Storage: Cloud Storage for raw/bronze and archival; BigQuery for curated/serving analytics; Bigtable for low-latency key access; Spanner/Cloud SQL for OLTP.
  • Compute: Dataflow for serverless streaming/batch; Dataproc for Spark/Hadoop ecosystems; BigQuery for in-warehouse ELT; Cloud Run/Functions for event microservices.
  • Orchestration: Cloud Composer (Airflow) or Workflows for DAGs and API choreography; Scheduler for cron-like triggers.
  • Serving: Bigtable or Spanner for online reads; BigQuery for BI; Looker/BI Engine for dashboards; Memorystore for caching.
• Data lifecycle:
  • Ingest: Pub/Sub for streams; Storage Transfer or gsutil for files; Data Transfer Service for SaaS. Validate, deduplicate, and land immutable raw data in Cloud Storage with object versioning.
  • Process: Use Dataflow or BigQuery to transform raw to silver (cleaned, conformed), then to gold (business-ready marts).
  • Serve: Publish BigQuery views/tables for analytics; precompute features or predictions to Bigtable for APIs.
  • Retain and archive: Apply Cloud Storage lifecycle rules to transition to Coldline/Archive tiers; use BigQuery time partitioning with partition expiration for retention. Enable CMEK where required and VPC Service Controls for data exfiltration protection.
• Service selection based on workload characteristics:
  • High-throughput time-series with wide rows and low latency: Bigtable.
  • Strongly consistent global OLTP with ANSI SQL: Cloud Spanner.
  • Traditional relational transactions with modest scale: Cloud SQL.
  • Petabyte-scale analytics with ANSI SQL and separation of storage/compute: BigQuery.
  • Real-time ingestion and processing: Pub/Sub + Dataflow.
  • Batch Spark/Hadoop or library-specific tooling: Dataproc.

Short BigQuery partitioning example:

CREATE TABLE ops.events
PARTITION BY DATE(event_ts)
CLUSTER BY device_id AS
SELECT * FROM staging.events_clean;

Practical Problem Scenario

Contoso Mobility operates a global e-scooter fleet and needs real-time ingestion, processing, storage, and analytics for ride telemetry and billing. They must support millions of events per minute, sub-second fraud rules, up-to-date dashboards, privacy controls, and resilient multi-region operations.

Approach:

1. Establish event ingestion with Cloud Pub/Sub.

• Rationale: Pub/Sub provides a single global endpoint, durable buffering, and horizontal scale for bursty device traffic. Use ordered keys per scooter to preserve intra-device ordering for 1-hour windows.

2. Implement streaming processing with Cloud Dataflow (Apache Beam).

• Rationale: Dataflow autoscaling handles spikes and offers exactly-once sinks when combined with idempotent keys. Use event-time windows and watermarks to handle late/out-of-order telemetry. Emit a main output to curated streams and a side output for dead-letter records.
• Configuration:

.withAllowedLateness(Duration.standardMinutes(15))
.discardingFiredPanes();

3. Persist raw and curated data in Cloud Storage and BigQuery, respectively.

• Rationale: Land raw (bronze) Avro files in a dual-region Cloud Storage bucket for replay and audit. Write curated (silver) streams to BigQuery partitioned tables for analytics, with clustering on scooter_id for efficient point lookups. Apply a small freshness buffer on dashboard queries to avoid transient streaming staleness.

4. Serve operational lookups and fraud checks from Cloud Bigtable.

• Rationale: Sub-100 ms rule evaluation needs low-latency random access. Precompute aggregates (e.g., rides per device per 5-minute window) in Dataflow and write to Bigtable using a hashed prefix row key (e.g., h(prefix)+device_id+window_start) to avoid hotspotting and to parallelize reads across tablets.

5. Manage transactional billing in Cloud Spanner.

• Rationale: Billing requires globally consistent SQL, strong consistency, and high availability. Use a leader in the primary geography with read-only replicas in secondary regions to reduce read latencies for customer portals.

6. Enforce governance with Dataplex, Data Catalog, and Cloud DLP.

• Rationale: Classify PII fields, tag datasets, and apply column-level security in BigQuery. Integrate Cloud DLP in the Dataflow pipeline to tokenize sensitive attributes before storage. Dataplex domains reflect organizational ownership; each domain publishes documented data products with SLOs.

7. Orchestrate and operate with Cloud Composer and Cloud Monitoring.

• Rationale: Composer coordinates batch backfills, compactions, and ML feature materialization. Monitoring observes end-to-end SLIs: Pub/Sub backlog, Dataflow watermark lag, BigQuery partition completeness, and Bigtable tail latencies. Alert on SLO violations; autoscale Dataflow based on backlog growth.

8. Optimize costs and lifecycle with partitioning and tiering.

• Rationale: BigQuery tables are partitioned by event_ts with 90-day retention, and clustered by scooter_id. Cloud Storage uses lifecycle rules to transition raw data to Coldline after 30 days and Archive after 180 days. Scheduled BigQuery jobs compact small micro-batch files into larger parquet objects to reduce file-count overhead for downstream Spark jobs.

9. Validate risks and resiliency.

• Rationale: Conduct load tests at 2x expected peak to validate Pub/Sub quotas and Dataflow autoscaling. Perform a regional failover exercise: BigQuery multi-region datasets and dual-region buckets maintain availability; Spanner’s multi-region instance sustains RPO=0 and configured RTO via automatic failover. Use Infrastructure as Code (Terraform) with policy validation to enforce CMEK and VPC Service Controls.

This architecture cleanly separates concerns: Pub/Sub buffers ingestion, Dataflow computes, Cloud Storage and BigQuery store and serve analytics, Bigtable accelerates operational reads, and Spanner guarantees consistent transactions. It balances scalability and latency while controlling costs through partitioning, clustering, lifecycle policies, and autoscaling, and it embeds governance and reliability through documented data products, contracts, and continuous validation.

All domains · Data Storage →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →