A company stores semi-structured transactional data in S3. Some files are tiny while others are tens of terabytes. The source sends a full JSON snapshot once per day and also sends changed records into the data lake. A data engineer must perform change data capture (CDC) to identify changes most cost-effectively. Which solution is the MOST cost-efficient?

Using an open-source data lake format like Apache Iceberg, Apache Hudi, or Delta Lake directly on S3 is the most cost-efficient solution. These formats are designed for efficient CDC operations on large datasets in object storage, allowing for upserts (updates and inserts) and deletes without rewriting entire files. This minimizes S3 storage and data transfer costs. Creating an AWS Lambda function for CDC would be inefficient and costly for tens of terabytes of data, as Lambda has execution time and memory limits, and processing such large files would incur significant compute costs. Ingesting data into Amazon RDS for MySQL or Amazon Aurora MySQL DB instances (even Serverless) introduces additional database infrastructure costs, data transfer fees into and out of the database, and the complexity of managing a relational database for what is essentially a data lake operation. While AWS DMS can facilitate CDC from these databases, the overhead of the database itself makes it less cost-efficient than a direct data lake format approach for semi-structured data already in S3.