Google PDE: Data Governance, Security, Reliability and Cost Operations — Study Guide

By Laurynas Brazys · 21 August 2026 · #google #PDE #certification #exam-prep #study-guide

Part of the Google Professional Data Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

This section summarizes design patterns and operational practices for data governance, security, reliability, and cost operations on Google Cloud. It focuses on BigQuery, Cloud Storage, Dataflow, Dataplex, and supporting services. Emphasis is on least privilege, encryption key management, metadata and classification, policy-driven access, compliance evidence, observability with actionable SLOs, and cost control. Trade-offs, failure modes, and practical configurations are included to enable safe, auditable, and efficient data platforms.

Identity, Access, and Governance

• IAM, service accounts, impersonation, workload identity, least privilege

  • Identity boundaries
    • Users and groups via Cloud Identity or Google Workspace
    • Service accounts for workloads; assign narrowly scoped roles at the lowest practical resource (e.g., dataset instead of project)
  • Least privilege
    • Prefer predefined roles over primitive roles; for BigQuery, use roles like bigquery.dataViewer on datasets instead of project-level viewer
    • Grant permissions to groups; manage membership in an IdP, not per-user IAM
    • Split duties: separate roles for key management, data access, and admin
  • Impersonation and Workload Identity Federation
    • Use Service Account Impersonation (roles/iam.serviceAccountTokenCreator) so CI/CD or automation never stores long-lived keys
    • Use Workload Identity Federation with OIDC/SAML to let external identities obtain short-lived tokens without service account key files
  • Failure modes and mitigations
    • Excessive project-level roles lead to lateral movement; audit with Cloud Asset Inventory
    • Lost service account private keys: disallow key creation; use organization policy constraints to block key download; rotate if found

• Dataplex governance, Data Catalog, business metadata, lineage

  • Dataplex provides lakes, zones, and assets to unify governance over BigQuery and Cloud Storage with centralized policies
  • Data Catalog holds a business glossary, tag templates, and technical metadata; attach business metadata (owner, PII class, RTO/RPO) via tags
  • Lineage captures upstream/downstream relationships; use Dataplex lineage integrations with Dataflow, Dataproc, and BigQuery to trace impact and compliance scope
  • Trade-offs
    • Centralized governance adds initial overhead but reduces long-term risk and accelerates audits

• Policy tags, classification, row-level access, column masking

  • Classification
    • Define a taxonomy (e.g., public, internal, confidential, restricted) in Data Catalog policy tags
    • Attach policy tags to BigQuery columns; bind IAM to tags so access follows classification across tables
  • Column masking
    • Use BigQuery data masking policies to hash or nullify sensitive columns for non-privileged readers
    • Example:
      • ALTER TABLE fin.payments ALTER COLUMN card_number SET POLICY TAGS (‘pii.restricted’);
  • Row-level access
    • Use row access policies to filter records by attributes such as tenant_id or region
    • Example:
      • CREATE ROW ACCESS POLICY tenant_filter ON sales.orders GRANT TO (“group:analysts@acme.com”) FILTER USING (tenant_id = “acme”);
  • Failure modes
    • Policy tag IAM not granted to service accounts used by pipelines causes query failures; include policy tag viewer/accessor for service agents as needed
    • Row policies can degrade performance if highly selective per-user predicates are many; prefer coarse-grained dataset-per-tenant where isolation is strict

• Sensitive-data discovery and de-identification

  • Use Sensitive Data Protection to continuously scan Cloud Storage and BigQuery; create discovery configs per lake/zone with templates
  • Use de-identification transforms: tokenization, deterministic encryption for joinability, or masking
  • Store transformation keys in Cloud KMS; keep re-identification keys separate with dual control
  • Trade-offs
    • Deterministic encryption enables joins but may leak frequency information; add format-preserving encryption or bucketing as needed
    • Sampling reduces cost of discovery scans but can miss low-prevalence PII

Security and Compliance Operations

• Encryption, Cloud KMS, CMEK, and secrets handling

  • Encryption at rest and in transit is default; enable CMEK where regulatory control of keys is required (BigQuery, GCS, Pub/Sub, Dataflow)
  • Key management
    • Place keys in the same region as data; grant the service agent (e.g., BigQuery Service Agent) roles/cloudkms.cryptoKeyEncrypterDecrypter
    • Rotate keys regularly; monitor for disabled or scheduled-for-destruction keys
  • Failure modes
    • Disabling a CMEK key or revoking the service agent breaks loads, queries, and exports; alert on key state changes
    • Cross-region key usage is disallowed; align locations to avoid job creation errors
  • Secrets
    • Use Secret Manager for database credentials, API tokens; grant access via IAM and audit with Secret Manager logs
    • Never embed secrets in code, containers, or notebooks; mount secrets via runtime access; prefer IAM database auth where supported

• Audit logs, access review, compliance evidence, retention

  • Enable Data Access logs organization-wide for BigQuery, GCS, Pub/Sub; export to a dedicated, write-only logging project with CMEK
  • Create aggregated log sinks to BigQuery (analytics) and Cloud Storage (long-term immutable archive with bucket retention lock)
  • Use Cloud Asset Inventory and Policy Analyzer for periodic access review and drift detection
  • Retention
    • Set log retention per compliance needs; use object versioning and retention policies on GCS
    • In BigQuery, set default table expiration and rely on table snapshots/time travel for short-term rollback; archive critical datasets to separate projects
  • Evidence
    • Maintain a control mapping with Dataplex tags (e.g., “SOX-C2: Evidence in project X, sink Y”), automate exports, and run scheduled queries to produce attestations

Reliability, Observability, Quality, and Cost Management

• Data quality dimensions, validation frameworks, and incident response

  • Dimensions: accuracy, completeness, consistency, timeliness, validity, uniqueness, integrity
  • Implement validations at ingestion and transformation
    • Dataplex Data Quality rule sets on BigQuery tables and GCS assets
    • Great Expectations or Deequ in Dataflow/Dataproc for schema and content checks
  • Route failures to dead-letter tables or buckets with rich error context; avoid data loss by quarantining bad records
  • Incident response
    • Declare severities, owners, comms channels, rollback plans, and RACI
    • Automate runbooks to backfill windows and reprocess dead letters; snapshot affected tables before remediation

• Cloud Monitoring, logging, alerting, error budgets, and SLOs

  • Expose metrics: Dataflow backlog, BigQuery slot utilization, query latency, GCS latency/errors, Pub/Sub unacked messages
  • SLOs
    • Example: “99.9% of streaming events available in BigQuery within 5 minutes over 30 days”
    • Track error budget burn rates and page on fast burns; ticket on slow burns
  • Logging-based metrics and alerts
    • Create logs-based metrics on BigQuery job failures, DLP findings, KMS key errors
    • Use advanced log filters to alert on specific table appends or access anomalies

• Cost allocation, budgets, query controls, storage lifecycle, capacity planning

  • Allocation and budgets
    • Use labels and tags on all jobs, datasets, buckets, and reservations; export billing data to BigQuery, and create budgets with Pub/Sub notifications
  • BigQuery cost controls
    • Use partitioning and clustering to reduce scanned bytes
    • Set maximumBytesBilled on query jobs; example client/job config:
      • “jobConfiguration”: { “query”: { “maximumBytesBilled”: “1073741824” } }
    • Reserve slots with BigQuery Reservations for predictable workloads; separate interactive from batch via assignments
  • Storage lifecycle
    • GCS: lifecycle rules to transition to colder storage or delete after N days; enable object versioning where rollback is needed
    • Example (condensed): Delete noncurrent versions after 30 days; set bucket retention policy to 365 days for compliance zones
    • BigQuery: default table expiration for transient datasets; snapshot before destructive changes
  • Capacity planning
    • Dataflow: set max workers and autoscaling; right-size machine types; shard inputs to avoid hot keys
    • Pub/Sub: validate publish/consume quotas and message retention
    • Network: account for egress, inter-region movement, and private service access for databases

• Disaster recovery, backups, multi-region resilience, and runbooks

  • Classify services by RTO/RPO; choose cold/warm/hot patterns accordingly
  • Backups
    • BigQuery: regular table snapshots; export to GCS for off-platform retention if required
    • GCS: dual-region or multi-region buckets for durability; enable bucket lock for WORM compliance
    • Databases: managed backups in Cloud SQL and Bigtable; test restores
  • Multi-region
    • Keep compute and storage in the same multi-region to minimize egress and latency; avoid cross-continental dependencies unless required
  • Runbooks
    • Document failover, key recovery, KMS incident procedures, rehydration from exports, and BigQuery reservation reassignments
    • Test DR via game days; track time-to-recover and update SLOs

Practical Problem Scenario

NovaRetail Analytics partners with multiple brands to ingest daily CSVs containing transaction data into a shared analytics platform. Files arrive in a Cloud Storage landing bucket and occasionally include malformed rows. The platform must enforce least privilege so each client can only access its own data, detect sensitive fields, and provide immediate alerts when rows are appended to a specific audit table. The company also needs cost controls and a recovery plan.

Approach:

1. Isolate tenants and enforce least privilege

   • Create a dedicated BigQuery dataset per client (e.g., client_a_analytics). Grant only the client’s group the appropriate dataset roles (bigquery.dataViewer, bigquery.jobUser) and restrict BigQuery API use to approved users via IAM and VPC-SC if applicable.
   • Rationale: Dataset-per-tenant limits blast radius and simplifies row policy complexity. Least-privilege scoping at dataset level prevents cross-tenant access by default.

2. Govern schema, classification, and masking

   • Define a Data Catalog policy tag taxonomy (public, internal, confidential, restricted) and tag templates for owner, data steward, and RTO/RPO. Attach policy tags to sensitive columns (email, card_suffix) in each client dataset. Apply BigQuery masking policies to restrict views for non-privileged roles.
   • Example: ALTER TABLE client_a_analytics.orders ALTER COLUMN email SET POLICY TAGS (‘restricted.pii’).
   • Rationale: Central tags provide uniform control across tables; masking ensures safe-by-default reads without duplicating data.

3. Discover PII and enforce de-identification where needed

   • Configure Sensitive Data Protection discovery to scan the landing bucket and curated BigQuery tables. Use an inspection template for common PII and a de-identification template to tokenize emails deterministically for join use cases.
   • Rationale: Automated discovery reduces manual errors; deterministic tokenization balances privacy with analytics join requirements.

4. Secure pipeline with service accounts, impersonation, and CMEK

   • Use a Dataflow service account with only the needed roles: storage.objectViewer on the landing bucket, bigquery.dataEditor on target datasets, and access to policy tags if required. Use CMEK for the client datasets and grant the BigQuery and Dataflow service agents the CryptoKey Encrypter/Decrypter role.
   • Rationale: Narrow roles plus CMEK meet least-privilege and key control requirements. Service agent access to keys prevents job failures.

5. Build resilient ingest with error quarantine

   • Run a batch Dataflow job that reads the CSVs, validates schema, and writes valid rows to BigQuery partitioned tables. Route invalid rows to a BigQuery dead-letter table with error details (file name, line, reason).
   • Rationale: Side outputs preserve bad data for analysis without blocking good data; partitioned tables reduce scan cost and speed queries.

6. Create lineage and business metadata

   • Register the landing bucket and datasets as Dataplex assets in a lake. Enable lineage collection for the Dataflow job and tag curated tables with business metadata (data owner, sensitivity, retention).
   • Rationale: Centralized governance enables impact analysis, audit readiness, and standardized stewardship.

7. Monitor, alert, and audit

   • Enable Admin and Data Access audit logs, exported with CMEK to a central logging project and to BigQuery for analytics. Add a log-based alert for new rows appended to the audit table using an advanced filter on BigQuery insert jobs; export that sink to Pub/Sub for the monitoring tool to consume.
   • Rationale: Logs are tamper-resistant evidence; targeted alerts notify only on the required table, reducing noise.

8. Enforce cost controls and query guardrails

   • Require query jobs to set maximumBytesBilled and leverage clustering on high-cardinality columns (e.g., order_id). Apply budgets and set labels (client, environment) on jobs and datasets. Use BigQuery Reservations to separate interactive analysis from scheduled loads.
   • Rationale: Guardrails prevent runaway costs; labels enable chargeback; slot isolation maintains predictable performance.

9. Implement retention and DR

   • In the landing bucket, enable object versioning and a lifecycle rule to delete objects after 30 days; set a retention policy for compliance zones. In BigQuery, set default table expiration for staging tables and take periodic table snapshots of curated tables. Store KMS backup procedures and table restore steps in runbooks and test quarterly.
   • Rationale: Lifecycle management reduces storage cost; snapshots and documented runbooks ensure recoverability. Testing validates RTO/RPO assumptions.

10. Periodic access reviews and quality SLIs/SLOs

• Quarterly, export IAM policies with Cloud Asset Inventory and compare to an approved baseline. Define SLOs such as “99% of daily files processed within 30 minutes of arrival” with alerts on burn rate. Track data quality SLIs (completeness, validity) using Dataplex Data Quality rules and route violations to incident response with backfill automation.
• Rationale: Regular review prevents privilege creep; SLO-driven operations align effort with user experience; automated quality checks catch regressions early.

Technical trade-offs and failure modes addressed:

• CMEK misconfiguration or key disablement will break Dataflow loads and BigQuery queries; monitoring on KMS state and service agent permissions is mandatory.
• Overuse of fine-grained row policies can degrade performance; prefer dataset isolation for tenants.
• Discovery scans can be costly; scope by zone and sample where appropriate, accepting the risk of missing rare PII.
• Alert fatigue reduces responsiveness; build precise filters per table/action and test before rollout.

← Machine Learning · All domains

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →