Google PDE: Machine Learning, AI and Data Serving — Study Guide

By Laurynas Brazys · 21 August 2026 · #google #PDE #certification #exam-prep #study-guide

Part of the Google Professional Data Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Building production-grade machine learning and data-serving systems on Google Cloud requires disciplined data modeling, robust pipelines, and operational guardrails. This section covers model development in BigQuery ML, managed lifecycle on Vertex AI (datasets, training, pipelines, endpoints, feature engineering, and monitoring), prediction path design (batch versus online), feature stores and point-in-time correctness, labeling and bias controls, vector search and retrieval-augmented generation patterns, lineage and governance, drift monitoring and retraining triggers, analytics-serving layers, and privacy-aware data use. Emphasis is placed on design decisions, scaling strategies, and common failure modes to avoid.

BigQuery ML and Feature Engineering

BigQuery ML enables training, evaluation, and prediction directly in SQL, eliminating data movement and aligning model development with analytical datasets.

• Model creation: use CREATE MODEL with explicit label columns and feature transforms to avoid leakage and standardize inputs. Example: CREATE OR REPLACE MODEL ds.churn_model OPTIONS( model_type=‘logistic_reg’, input_label_cols=[‘churned’], l1_reg=0.0, l2_reg=1.0, data_split_method=‘AUTO’ ) TRANSFORM( standardize(tenure_months) AS tenure_std, quantile_bucketize(monthly_spend, 10) AS spend_bkt, one_hot_encoder(region) AS region_ohe, ml.feature_cross(struct(bucketize(lat, 60), bucketize(lon, 60))) AS latlon_cross, (xx + yy) AS r2 – add circular decision boundary support when useful ) AS SELECT churned, tenure_months, monthly_spend, region, lat, lon, x, y FROM ds.customer_features;

• Evaluation: use ML.EVALUATE to get metrics appropriate to the model type (e.g., ROC AUC for classification, RMSE for regression). Track baselines and confidence intervals; keep evaluation datasets time-ordered to approximate future performance. SELECT * FROM ML.EVALUATE(MODEL ds.churn_model, TABLE ds.eval_features);

• Prediction: use ML.PREDICT for online-like scoring in BigQuery, or export models for serving elsewhere. Consider model latency budgets when using BigQuery for synchronous scoring; for high-QPS APIs, deploy to managed endpoints. SELECT user_id, predicted_churn FROM ML.PREDICT(MODEL ds.churn_model, TABLE ds.scoring_candidates);

• Feature transformations: prefer declarative TRANSFORM functions (standardize, one_hot_encoder, bucketize, quantile_bucketize, ml.feature_cross) for reproducibility and to lock preprocessing to the model artifact. Keep transformations idempotent and deterministic.

Operational considerations and failure modes:

• Streaming inserts and query freshness: BigQuery streaming has eventual consistency. For real-time aggregations that must include just-written rows, run queries with a time delay that exceeds measured streaming buffer latency. A conservative starting point is to wait roughly 2× the observed average availability delay, or design watermarks and late-data handling in Dataflow before landing to BigQuery.
• Cost and concurrency: if on-demand slot concurrency limits become a bottleneck, switch to flat-rate or flexible reservations and implement workload management (reservation hierarchies and assignments) to ensure predictable capacity.
• Data quality: for GCS batch loads with malformed rows, use Dataflow to parse and validate records, writing good rows to BigQuery and bad rows to a dead-letter table for inspection. Avoid BigQuery rejecting entire files due to a small number of bad rows.

Vertex AI lifecycle, prediction paths, and feature stores

Vertex AI provides end-to-end managed services for training, pipelines, model registry, endpoints, and monitoring.

• Datasets and training: register datasets and metadata; use custom training jobs or AutoML where appropriate. Choose algorithms based on constraints:

  • Resource-constrained single-VM workloads favor simple models (e.g., linear regression or logistic regression) due to low memory/CPU demands.
  • High-dimensional tasks often benefit from feature selection or combining redundant features to speed training with minimal accuracy loss.
  • Unsupervised anomaly detection is suitable when positive examples are rare and future anomalies are expected to resemble known anomalous signatures.

• Pipelines: implement Vertex AI Pipelines to codify data prep, training, evaluation, and deployment gates. Persist parameters, code commit SHAs, container digests, and dataset snapshots to guarantee reproducibility.

• Endpoints and prediction:

  • Online prediction for low-latency workloads. Configure minimum and maximum replicas and autoscaling policies; profile model latency at P95 and set SLOs accordingly. Add canary deployments and traffic splitting for safe rollouts.
  • Batch prediction for throughput-first jobs (e.g., nightly scoring). Batch avoids per-request overhead and is cheaper for large volumes but offers higher latency.

• Feature engineering and feature stores: use Vertex AI Feature Store for:

  • Offline store in BigQuery for training.
  • Online store for low-latency lookups by entity ID. Enforce training-serving consistency by sharing the same transformation logic (e.g., Dataflow library or feature definitions) and by using feature timestamps to prevent leakage. Maintain point-in-time correctness with temporal joins: SELECT f.* FROM ds.labels l JOIN ( SELECT entity_id, feature_ts, feature_val, ROW_NUMBER() OVER (PARTITION BY entity_id ORDER BY feature_ts DESC) AS rn FROM ds.features WHERE feature_ts <= l.event_ts ) f ON f.entity_id = l.entity_id WHERE f.rn = 1;

Design trade-offs:

• Online store latency vs. freshness: Bigtable-backed online stores provide low latency; ensure backfills and streaming upserts are idempotent. Excessive write skew or hot keys degrade performance—design entity IDs to evenly distribute traffic.
• Batch vs. online: batch reduces serving complexity and cost but may provide stale predictions. For dynamic behavior (e.g., recommendations), combine periodic retraining with up-to-date features at serving time.

Data quality, labeling, bias, privacy, and governance

High-quality labels and rigorous governance underpin trustworthy models.

• Labeling and imbalance:

  • Use clear labeling guidelines and QA sampling. Track inter-annotator agreement.
  • Address class imbalance with stratified sampling, reweighting, or resampling; monitor precision/recall per class, not just overall accuracy.
  • Preserve nulls intentionally. If a model requires numeric inputs, encode nulls explicitly (e.g., 0 with a “was_null” indicator) and validate downstream impact; avoid silently dropping informative missingness.

• Overfitting and generalization:

  • Mitigations include more diverse training data, smaller feature sets, and stronger regularization.
  • Early stopping and cross-validation are essential for neural nets; subsampling can reduce training time when architecture or hardware scaling is not viable.

• Governance and lineage:

  • Track lineage with Vertex ML Metadata, Model Registry, and Data Catalog. Record dataset versions, transformations, hyperparameters, and environment.
  • Approval workflows: require human approval before deployment, using Model Registry states and Cloud Build/Deploy with policy checks. Store artifacts in Artifact Registry; sign containers and enforce Binary Authorization for gated rollouts.

• Monitoring, drift, and retraining:

  • Enable model monitoring for prediction skew, feature drift, and performance degradation. Use distributional metrics (e.g., PSI, KL divergence) and ground-truth lag-aware evaluation where labels arrive later.
  • Establish retraining triggers based on statistically significant drift, SLO breaches, or business event windows. Automate retraining pipelines but gate promotion with evaluations and bias checks.
  • Beware silent data drift from upstream schema changes; enforce schema contracts and alert on missing or shifted features.

• Privacy-aware design:

  • Classify data using Data Catalog policy tags; enforce column- and row-level security in BigQuery, with data masking policies.
  • Minimize data collection; implement data retention and deletion SLAs tied to purpose limitation.
  • Apply DLP for discovery and de-identification; encrypt data with CMEK; isolate services with VPC Service Controls; ensure fine-grained IAM and use dedicated service accounts with least privilege.
  • For monitoring and logging, redact PII and avoid payload logging where not necessary.

Vector search, RAG pipelines, and analytics serving layers

Modern retrieval and serving require both vector-native components and proven analytics stores.

• Vector search and embeddings:

  • Use Vertex AI Vector Search or BigQuery vector search for large-scale, low-latency nearest neighbor retrieval; choose AlloyDB for PostgreSQL with pgvector for app-centric semantics and transactional needs.
  • Batch-generate embeddings with Vertex Pipelines; store vectors alongside dense metadata; partition and index intelligently (e.g., by document domain) to bound latency.

• Retrieval-augmented generation pipelines:

  • Ingest content via Dataflow or Dataproc, extract text, chunk, embed, and index into a vector store. Maintain source-of-truth references for traceability.
  • Implement freshness strategies: periodic re-embedding, invalidation on source updates, and canary indexing to validate quality before swapping indexes.
  • Monitor retrieval quality (hit rate, MRR, nDCG) and content safety; enforce guardrails and access controls for restricted data.

• Analytics serving layers and data products:

  • Curate bronze/silver/gold data products in BigQuery; use partitioning and clustering to minimize scan costs. Materialized views can accelerate common queries.
  • For low-latency key-value or high-QPS counters, use Bigtable with well-distributed row keys; avoid hot-spotting by salting or hashing prefixes.
  • For OLTP workloads and strong consistency, use Cloud SQL or Spanner; offload analytics to BigQuery via scheduled ELT.
  • Streaming design: Pub/Sub → Dataflow → BigQuery/Bigtable with autoscaling. Monitor backlog and watermark metrics; default autoscaling suffices for elastic loads while controlling costs.

Operational tip:

• To trigger notifications on specific BigQuery table insert jobs, export relevant Cloud Logging entries to Pub/Sub using an advanced filter, then wire alerts from the subscription: resource.type=“bigquery_resource” protoPayload.methodName=“jobservice.jobcompleted” protoPayload.serviceData.jobCompletedEvent.job.jobConfiguration.load.destinationTable.tableId=“target_table”

Practical Problem Scenario

AcmeStyle, a fashion marketplace, wants to keep on-site recommendations current as user preferences shift hourly. They stream click and purchase behavior and need to blend this with catalog context to refresh recommendations with low latency and controlled cost.

Approach:

1. Stream ingestion and quality gates

• Use Pub/Sub for event ingestion from web and mobile. A Dataflow streaming job validates schemas, enriches with catalog data, and writes:
  • Clean events to BigQuery partitioned tables (event_date) for offline analytics and training.
  • Aggregated user-feature updates to Vertex AI Feature Store (online store) keyed by user_id. Rationale: Pub/Sub decouples producers and consumers; Dataflow provides exactly-once semantics with idempotent upserts; partitioned BigQuery manages cost and retention; the online store enables millisecond lookups.

2. Feature definitions with point-in-time correctness

• Define features such as rolling CTR, brand affinity, and recency with explicit event_time. Materialize to:
  • Offline store in BigQuery for training with temporal joins constrained to feature_ts <= label_ts.
  • Online store for serving with TTLs to prevent stale values. Rationale: Clear timestamps prevent label leakage; consistent definitions across offline/online ensure training-serving parity.

3. Model training and lineage

• Implement a Vertex AI Pipeline that:
  • Extracts training data from BigQuery using time windows (e.g., last 30 days).
  • Applies the same transformations used in serving (shared library).
  • Trains a ranking model; logs metadata (dataset snapshot IDs, code commit SHA, hyperparameters) to ML Metadata and registers the model in the Model Registry. Rationale: Pipelines make runs reproducible and auditable; Model Registry centralizes versions and approvals.

4. Batch and online prediction paths

• Nightly batch predictions scoring the full catalog-user matrix into BigQuery for backfill and A/B testing.
• Online predictions via a Vertex endpoint that:
  • Fetches fresh user features from the online store.
  • Scores top-K candidates filtered by inventory and availability.
  • Caches results for short periods to absorb bursts. Rationale: Batch provides breadth and cost efficiency; online captures the latest behavior for high-value sessions. Autoscaling endpoints maintain latency SLOs; caching reduces tail latency and cost.

5. Monitoring, drift detection, and retraining policy

• Enable model monitoring for feature drift and prediction skew; compare distributions against training baselines. Track CTR/CVR SLOs and alert on degradation.
• Retrain continuously using a rolling window combining historical and new data; trigger retraining when drift exceeds thresholds or weekly at minimum. Rationale: Fashion trends shift quickly; blending history with recent signals stabilizes learning while staying current.

6. Privacy and governance

• Tag PII columns with Data Catalog policy tags; enforce column-level security in BigQuery and mask where needed. Run DLP scans on raw events; store only necessary fields.
• Require human approval to promote models from staging to production via Cloud Build triggers integrated with Model Registry approval states. Rationale: Least-privilege access reduces risk; gating deployments ensures compliance and safety.

7. Cost and capacity controls

• Use BigQuery reservations to guarantee predictable slot capacity for training windows.
• Scale Dataflow workers automatically based on backlog; shard hot keys in the feature store by hashing user_id prefixes to prevent hot-spotting. Rationale: Predictable capacity avoids contention; autoscaling matches spend to demand; balanced keys sustain low-latency updates.

This design keeps recommendations fresh by unifying streaming features for serving with regular retraining on recent data, while maintaining correctness, governance, and predictable performance at scale.

← Workflow Orchestration and Pipeline Automation · All domains · Data Governance →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →