Amazon DEA-C01: Data Cataloging and Metadata Management — Study Guide

By Laurynas Brazys · 2 August 2026 · #amazon #dea-c01 #certification #exam-prep #study-guide

Part of the Amazon Data Engineer Associate DEA-C01 — Study Guide. Practice with verified answers in the Amazon exam hub, or take timed practice tests on ExamRoll.io.

This domain covers the metadata layer that makes data discoverable, queryable, and governable across an AWS data platform. Effective cataloging and metadata management reduce friction for analytics and ensure downstream consumers can find schemas, partitions, access policies, and lineage. AWS services in this area—Glue Data Catalog, Glue Schema Registry, Lake Formation, Athena integration, and DataBrew—provide complementary tooling for discovery, schema evolution, governance, and profiling. Understanding how these services interact, their configuration details, and typical failure modes is essential for operational reliability and security.

AWS Glue Data Catalog structure and operations

The Glue Data Catalog is the regional, centralized metadata repository for databases, tables, partitions, connections, and user-defined classifiers. Core primitives are:

• Database: a logical container (use aws cli: aws glue create-database –database-input Name=analytics_db).
• Table: describes a dataset (serde, input/output formats, columns, tableType EXTERNAL_TABLE). You can create/update via console, Glue API, or CloudFormation; e.g., aws glue create-table –database-name analytics_db –table-input file://table.json.
• Partition: mapping between partition key values and S3 prefixes; partitions can be managed by Glue crawlers (aws glue start-crawler –name my-crawler) or added explicitly (aws glue batch-create-partition).

Operational patterns:

• Crawlers for discovery: schedule crawlers for evolving S3 layouts, choose classifier order (CSV/JSON/Parquet) and set crawler policy for incremental updates.
• Programmatic control: prefer Glue APIs or Lambda to add partitions when you have event-driven S3 arrivals rather than relying solely on crawlers.
• Catalog replication: the Glue Data Catalog is regional. For multi-region reads consider running crawlers in each region, building automation to replicate metadata, or using resource-linking patterns; design decisions depend on cost, consistency needs, and cross-region query patterns.

Decision criteria:

• Use crawlers when schema detection is required and data formats are heterogeneous; use explicit table creation for strict schemas and for high-volume, predictable datasets.
• Use batch-create-partition or Lambda-based partition management for high-frequency small file arrivals to avoid crawler lag and reduce Glue API costs.

Schema discovery and evolution

Schema Registry and Glue schema support Avro, JSON, and Protobuf for streaming and long-lived producer/consumer contracts. Key capabilities:

• Register schemas via console or CLI (aws glue create-schema –schema-name orders –data-format AVRO –compatibility BACKWARD).
• Compatibility modes: BACKWARD (consumers can read new data), FORWARD (new consumers can read old data), and FULL (both). Choose based on consumer deployment patterns.
• Schema enforcement: for streaming, integrate the registry with Kinesis Data Streams, MSK, or Kafka clients and AWS SDKs to serialize/deserialize with embedded schema versions and validation.

Practical configuration and evolution patterns:

• For Avro with many consumers, use BACKWARD compatibility to allow new fields to be added with defaults; avoid breaking removals.
• For strict contract evolution across teams, require full compatibility and gate schema changes through a CI step that runs schema validation.
• For JSON where fields are optional and schema is fluid, use schema evolution with permissive defaults but versioning metadata in the catalog to prevent silent consumer breakage.

Decision criteria:

• Use Schema Registry for streaming events and when multiple consumers need a canonical schema. Use Glue table schemas for batch datasets where format (Parquet/ORC) provides schema on read.
• Pick compatibility mode by evaluating whether you control all consumers (can coordinate FORWARD) or need safe additive changes (choose BACKWARD).

Data lineage and governance with Lake Formation

Lake Formation builds on the Glue Data Catalog to provide fine-grained access control, auditing, and lineage controls. Core features:

• LF-Tags: tag-based access controls applied to databases, tables, and columns. Create LF-Tags in Lake Formation, assign key:value pairs, and then grant IAM principals permission through tag-based grants rather than resource-based grants.
• Column-level controls: use LF-Tags to mask or restrict columns; configure column-level permissions in the Lake Formation console or with aws lakeformation grant-permissions.
• Lineage and auditing: enable CloudTrail and Glue job metrics to capture lineage of ETL jobs; use Glue job bookmarks and job bookmarks metadata in the catalog to track processed data.

Configuration patterns:

• Define a small, consistent set of LF-Tag keys (e.g., sensitivity:public/private/PII) and automate tagging on table creation or via Glue crawlers using crawler configuration or post-processing code.
• Delegate administration via Lake Formation Delegated Admin roles and grant Lake Formation permissions to analytics teams while restricting IAM-level S3 access.

Decision criteria:

• Use Lake Formation when you need centralized, column-level, and tag-based controls across many consumers and when governance/auditability is mandatory.
• If your access control needs are simple (bucket-level), IAM+S3 policies might suffice; use Lake Formation for fine-grained, catalog-integrated controls.

Athena and Glue catalog integration

Athena depends on the Glue Data Catalog for metadata. Common integration points and operational knobs:

• Partition handling: Athena reads partitions from the Glue catalog. When new S3 partitions are added, you must update the catalog. Options:
  • Run MSCK REPAIR TABLE db.table; from Athena or use aws athena start-query-execution with that SQL to refresh partitions discovered under table’s location.
  • Use aws glue batch-create-partition to add partitions programmatically on S3 PUT events (recommended for event-driven flows).
  • Use partition projection by setting table properties like projection.enabled=true, projection.year.type=integer, projection.month.range=1, and projection.year.range=2018,2026 — this avoids Glue lookups entirely and is essential for very large partition counts.
• Query performance and cost trade-offs:
  • Partition projection removes Glue API calls and drastically reduces latency for many small partitions but requires deterministic partition naming.
  • MSCK REPAIR TABLE is simple for occasional ad-hoc arrivals but can be slow for large datasets.

Glue DataBrew complement:

• Use DataBrew for no-code profiling and transformations; point DataBrew at Glue Catalog tables or S3 paths, run profiling jobs, create recipes, and publish output back to S3 or as new Glue tables.
• Use DataBrew for exploratory quality checks and to generate transformations for later productionization in Glue ETL when complex Spark logic is required.

Decision criteria:

• Use partition projection when partitions are numerous and follow a predictable schema (date-based / numeric).
• Use programmatic Glue partition updates for event-driven, near-real-time ingestion.
• Run MSCK REPAIR TABLE only for occasional backfills or when automation is not available.

Common Pitfalls and Decision Criteria

• Athena queries fail because Glue partitions aren’t updated after S3 arrivals: avoid relying solely on crawlers; either run MSCK REPAIR TABLE for occasional updates, invoke aws glue batch-create-partition on S3 events, or implement partition projection for large predictable partition sets.
• Choosing the wrong schema registry compatibility mode breaks consumers: select BACKWARD for additive changes and consumer stability, FORWARD when producers must remain compatible with older consumers, and FULL when both directions must be safe; validate changes in CI against consumer schemas.
• Assuming Glue Data Catalog is global: the catalog is regional. For cross-region access design replication or run catalogs in target regions; do not assume Glue metadata is automatically available across regions.
• Granting IAM S3 access but not Lake Formation permissions: Athena and Lake Formation enforce catalog-level permissions; always grant Lake Formation permissions (and LF-Tags where used) in addition to any IAM policies.
• Excessive partition granularity: using too many tiny partitions harms query planning and metadata overhead; prefer coarser partitions (daily vs minutely) or use partition projection.
• Neglecting DataBrew role permissions: DataBrew jobs require a service role with Glue and S3 permissions; ensure the role has Glue:GetTable, S3 read/write, and kms:Decrypt if datasets are encrypted.

Practical Problem: Use-Case Scenario

Acme Retail receives hourly sales files into S3 with date/hour partitions and analysts query the data in Athena; after load, users see query failures and stale results because partitions are not visible in the Glue Data Catalog.

1. Implement an S3 PUT event notification to invoke a Lambda function that calls aws glue batch-create-partition to register the new partition immediately.
2. For older data or backfills, schedule an Athena query that runs MSCK REPAIR TABLE db.sales_hourly; or run a targeted aws glue batch-create-partition for known ranges.
3. If partitions follow a strict date/hour naming convention, enable partition projection on the Glue table (set projection.enabled=true and define year/month/day/hour properties) to eliminate catalog refresh costs.
4. Add LF-Tags for sensitivity to the table and grant analysts Lake Formation permissions so Athena queries are permitted and governed.
5. Use Glue DataBrew to profile new hourly files in a staging environment to catch schema drift; if schema changes are found, register new schema versions in Glue Schema Registry and validate compatibility before production rollout.

Rationale: automatic partition registration or projection removes the metadata lag that breaks Athena queries; coupling this with Lake Formation governance ensures secure access and DataBrew-driven profiling catches schema drift early, while Glue Schema Registry protects streaming and batch consumers from incompatible schema changes.

← Data Storage and Lake Architecture · All domains · Data Transformation and Processing →

Practice these questions → · Timed practice on ExamRoll.io →

Browse Amazon →