Google PDE: Data Storage, Lakes and File Formats — Study Guide

By Laurynas Brazys · 21 August 2026 · #google #PDE #certification #exam-prep #study-guide

Part of the Google Professional Data Engineer — Study Guide. Practice with verified answers in the Google exam hub, or take timed practice tests on ExamRoll.io.

Overview

Data storage on Google Cloud spans raw object storage, curated data lakes, and analytics-optimized formats. Building reliable, governed, and performant lakes requires careful choices in storage classes, bucket settings, locations, file formats, table layout, and lifecycle. This section details design trade-offs, failure modes to avoid, and patterns that align with BigQuery, Spark, and streaming pipelines at scale.

Cloud Storage foundations: classes, buckets, consistency, and lifecycle

Cloud Storage is the durable, highly available foundation for raw and curated files.

• Storage classes

  • Standard (hot): frequent access, lowest latency. No minimum storage duration.
  • Nearline (cool): infrequent access (~monthly). 30-day minimum; retrieval fees apply.
  • Coldline (colder): infrequent access (~quarterly). 90-day minimum; higher retrieval fees.
  • Archive (coldest): long-term retention (~annual). 365-day minimum; highest retrieval fees.
  • Autoclass can automatically transition between classes; verify that early-delete fees and access patterns won’t erode savings.

• Bucket locations and replication

  • Region: best for data locality and compliance within a single geography.
  • Dual-region: two paired regions with automatic replication; turbo replication commits replicas quickly with an RPO measured in minutes; ideal for low-RPO DR.
  • Multi-region: geo-distributed within a continent for broad availability, content distribution, and analytics spanning a large area.
  • Choose locations to satisfy data residency laws and to minimize egress/latency to compute (Dataproc, Dataflow, BigQuery external tables).

• Consistency and semantics

  • Cloud Storage provides strong global read-after-write, read-after-metadata-update, and list-after-write consistency.
  • Object writes are atomic and immutable; “rename” is a copy+delete pattern. Design for idempotent copies and verify checksums to prevent partial migrations.

• Access patterns and performance

  • Parallel composite uploads and resumable uploads improve throughput for large files.
  • Range reads enable efficient columnar footers and selective reads.
  • Avoid many tiny files (<8 MB) that inflate metadata/listing overhead; batch or compact into larger objects.
  • GZIP is not splittable for distributed reads; prefer Parquet/ORC/Avro+Snappy for scalable processing.

• Lifecycle, retention, and versioning

  • Bucket-level retention policies and object holds (event-based or temporary) enforce immutability for compliance and to reduce accidental deletions.
  • Object versioning keeps prior generations; useful for recovery from overwrite/deletes. Monitor storage cost growth.
  • Lifecycle rules automate transitions and deletions. Example (JSON) to move older data colder and delete after a year: { “rule”: [ {“action”: {“type”: “SetStorageClass”, “storageClass”: “COLDLINE”}, “condition”: {“age”: 30}}, {“action”: {“type”: “Delete”}, “condition”: {“age”: 365}} ] }
  • Failure modes: early delete charges if you transition too aggressively; retention locks cannot be shortened; versioning without compaction can grow costs.

• Transfers and migration

  • Use Storage Transfer Service for parallelized, checkpointed moves from on-prem or other clouds; Transfer Appliance for offline petabytes.
  • Validate integrity with CRC32C/MD5 and generation-match preconditions to prevent races.
  • Prefer gsutil/gcloud storage with -m (parallel) and checksums; avoid SFTP bottlenecks for high-volume ingress.

Unified lake governance with BigLake and Dataplex

BigLake and Dataplex standardize security and governance across files and tables.

• BigLake

  • Exposes Cloud Storage data as BigQuery-managed tables (external) with uniform fine-grained access controls, including row access policies and column-level policy tags.
  • Enables column pruning and predicate pushdown for Parquet/ORC, reducing scanned bytes and egress to engines like BigQuery, Spark on Dataproc, and Dataflow.
  • Centralizes audit via Cloud Logging and central policy enforcement; a single control plane for lake files and warehouse tables.

• Dataplex

  • Organizes data into lakes, zones (raw, curated, trusted), and assets (buckets, datasets); manages metadata, lineage, and data quality rules.
  • Integrates with policy tags for sensitive columns and supports least privilege via IAM at lake/zone/asset scopes.
  • Encourages standardized naming, partitioning, and schema management across multi-team environments to avoid “junk drawers.”

• Governance patterns

  • Implement dataset-per-tenant and bucket-per-zone patterns; avoid cross-tenant data bleed.
  • Use row access policies and column policy tags for PII. Restrict API access to approved identities.
  • Audit access with Cloud Logging; route filtered logs to Pub/Sub for real-time monitoring.

File formats, compression, and query behavior

Choosing the right format has first-order effects on cost and performance.

• Columnar formats (Parquet, ORC)

  • Strengths: column pruning, predicate pushdown, encoding and compression per column, statistics, and splittable files.
  • Trade-offs: higher write-time CPU; schema evolution must be managed carefully (e.g., ADD columns safe; type changes risky).
  • Compression: Snappy for speed, ZSTD for better ratios where supported. Avoid GZIP for columnar unless interoperable constraints require it.

• Row-oriented Avro

  • Strengths: schema evolution with strong typing, block-level compression, splittable; excellent for streaming landing zones and interchange.
  • Trade-offs: less scan-efficient for analytics than columnar; convert to Parquet/ORC in curated zones.

• CSV and JSON (semi-structured)

  • CSV: human-readable, smallest overhead when values are simple; lacks schema, types, and escape consistency; expensive to parse at scale.
  • JSON: self-describing and flexible; newline-delimited JSON is required for scalable distributed reads; verbose and CPU-heavy to parse.
  • Where possible, land raw CSV/JSON, then validate and convert to Avro/Parquet for analytics.

• BigQuery external and BigLake tables

  • Parquet/ORC external tables benefit from pushdown and column pruning; CSV/JSON typically do not, leading to higher scanned bytes.
  • Compressed CSV (GZIP) external tables cannot be split across workers; expect slower reads.
  • Example: creating a Parquet BigLake table with Hive-style partitions: CREATE EXTERNAL TABLE lake.sales WITH CONNECTION us.biglake_conn OPTIONS ( format = ‘PARQUET’, hive_partitioning_mode = ‘AUTO’, hive_partitioning_source_uri_prefix = ‘gs://corp-raw/sales/’, uris = [‘gs://corp-raw/sales/date=/region=/part-*.parquet’] );

Layout, partitioning, performance engineering, residency, and migration

• Object layout and partitioning

  • Adopt Hive-style paths for partitions and clustering keys: gs://bucket/dataset/table/date=YYYY-MM-DD/hour=HH/region=us/part-00001.parquet
  • Keep individual file sizes in the 128–1024 MB range for balanced parallelism and task overhead. Avoid millions of files per partition.
  • Mitigate small-file problems by:
    • Batching uploads client-side.
    • Using Dataflow/Spark compaction jobs to coalesce small files off-peak.
    • Archiving original small files and exposing only compacted data to analytics.

• BigQuery partitioning and clustering

  • Partition on ingestion time or a high-cardinality filter column (e.g., event_date). Avoid over-partitioning (e.g., per-minute) that explodes metadata.
  • Cluster by commonly filtered/sorted dimensions (up to four). Clustering increases data locality and reduces scanned bytes.
  • Prefer native BigQuery tables for heavy interactive analytics; use BigLake/external for governed lake access, cross-engine sharing, and cost isolation.

• Query performance implications

  • Columnar formats materially reduce external scan costs; CSV/JSON external tables often require full-file scans.
  • Consistency guarantees remove the need for artificial delays with Cloud Storage reads, but downstream systems (e.g., BigQuery streaming inserts) can exhibit short data-visibility lag—design with watermarks or read delays where needed.

• Residency, durability, and recovery

  • Select bucket/dataset locations to meet residency constraints; colocate compute to reduce egress and latency.
  • Use dual-region with turbo replication for low RPO; versioning plus retention policies for recoverability from human error and ransomware.
  • For DR, replicate buckets to a separate project/region using bucket replication and protect with separate IAM boundaries.

• Safe migration and validation

  • Plan multi-phase: seed (bulk transfer), incremental sync (mtime/windowed copy), cutover (read-only source), and post-cutover validation.
  • Validate with checksums, counts, byte totals, and sample decode. For tabular data, compare row counts and hash aggregates: SELECT COUNT(*) c, ANY_VALUE(FARM_FINGERPRINT(TO_JSON_STRING(t))) h FROM proj.dst.dataset.table t;
  • Use preconditions (ifGenerationMatch) to prevent overwrites during parallel copy. Keep a rollback window with versioning or retained source.
  • After migration, enable lifecycle and Autoclass per the new access profile; avoid enabling retention lock until validations pass.

Practical Problem Scenario

Acme Retail receives daily CSV drops from a logistics partner into a regional Cloud Storage bucket. Files occasionally contain malformed rows. Acme must land, validate, convert to an analytics-ready format, and load into BigQuery for near-real-time dashboards, while retaining bad rows for inspection and enforcing governance.

Approach:

1. Land and govern raw data in Dataplex

   • Create a Dataplex lake with a raw zone asset mapped to gs://acme-raw/logistics/.
   • Rationale: Centralized governance, metadata, and lineage. Enforce IAM at the zone level and tag sensitive fields with policy tags for downstream enforcement.

2. Enforce lifecycle and retention

   • Apply a bucket retention policy of 30 days and enable object versioning on acme-raw.
   • Rationale: Protects against accidental overwrite/delete from the partner; short retention balances cost and recoverability. Versioning facilitates rollback of bad deliveries.

3. Validate and ingest with a Dataflow batch pipeline

   • Trigger a daily Dataflow job on object finalize notifications. Read CSV with schema and per-record validation; write valid records to BigQuery staging (partitioned by event_date) and route parse/validation errors to a dead-letter BigQuery table.
   • Rationale: Dataflow provides scalable parallel parsing and robust dead-letter handling so analysts can inspect bad rows. This mirrors best practice for heterogeneous CSV quality.

4. Compact and convert to Parquet in curated zone

   • The same pipeline writes validated data to gs://acme-curated/logistics/date=YYYY-MM-DD/ as Parquet files sized ~256–512 MB.
   • Rationale: Parquet enables column pruning and predicate pushdown in BigQuery and Spark, lowering scanned bytes and improving latency; compaction mitigates small-file overhead from the partner’s delivery pattern.

5. Expose governed analytics via BigLake

   • Create a BigLake external table over the curated Parquet path with Hive auto-partitioning; apply column-level policy tags and row access policies for partner-specific filters.
   • Rationale: Uniform fine-grained access across BigQuery and Spark with centralized audit. Partition pruning reduces scan costs on date filters.

6. Load critical aggregates to native BigQuery

   • For hot dashboards, run a scheduled BigQuery job that ingests the last N days from the curated Parquet external table into a native clustered, partitioned table.
   • Rationale: Native storage accelerates high-concurrency BI while the external BigLake table remains the governed system-of-record for broader access.

7. Monitor and alert with Cloud Logging and Pub/Sub

   • Create a log sink filtering Dataflow and BigQuery load job outcomes to Pub/Sub; integrate with the monitoring tool for instant alerts on failures or elevated bad-row rates.
   • Rationale: Targeted, table-specific operational visibility without polling; supports SRE practices.

8. Optimize storage class and residency

   • Keep curated Parquet in Standard for 14 days, transition to Coldline after 30 days via lifecycle rule; store both raw and curated buckets in the same region as BigQuery datasets to avoid egress.
   • Rationale: Balances hot read performance with cost. Co-location maintains compliance and minimizes latency and egress fees.

9. Validate end-to-end quality

   • After each run, compare counts and hash aggregates between staging, curated external, and native BigQuery tables; quarantine anomalies.
   • Rationale: Early detection of schema drift or ingestion regressions; cryptographic or fingerprint hashes provide lightweight assurance without full re-scans.

This design provides resilient ingestion with dead-letter analysis, analytics-ready Parquet for efficient queries, centralized governance via Dataplex and BigLake, and cost-optimized lifecycle policies, all while adhering to least-privilege access and auditable operations.

← Data Engineering Architecture and Design · All domains · BigQuery Analytics and Warehouse Engineering →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Google →