A company stores datasets in S3 that include PII for ecommerce. An internal analytics app does not need PII and the company must avoid unnecessary sharing of PII. The data engineer must implement dynamic, per-application redaction with minimal operational overhead. Which solution should the engineer choose?

The correct answer is to use an S3 Object Lambda endpoint. S3 Object Lambda allows you to process data retrieved from S3 with a Lambda function before it's returned to the requesting application. This enables dynamic, per-application redaction without creating multiple copies of the data, thus minimizing operational overhead. The Lambda function can inspect the request context (e.g., application ID) and apply specific redaction rules. Creating multiple copies of the dataset with S3 bucket policies or AWS Glue increases storage costs and operational complexity, making it difficult to maintain consistency and apply changes across all copies. Using API Gateway adds an unnecessary layer of complexity and latency for direct S3 data access and redaction, which S3 Object Lambda is specifically designed to handle more efficiently.