A company is migrating a legacy application to an S3-based data lake. The legacy data contains duplicate records. The data engineer must identify and remove duplicates with the LEAST operational overhead. Which solution should the engineer choose?

The correct answer is to use an AWS Glue ETL job with the FindMatches ML transform. AWS Glue is a fully managed ETL service, minimizing operational overhead compared to custom Python jobs that require managing infrastructure. The FindMatches transform is specifically designed for identifying fuzzy matches and duplicates in datasets, even when records lack a common identifier. This built-in functionality makes it highly efficient and effective for deduplication tasks within a data lake context. Custom Python jobs (options 1 and 3) increase operational overhead because you are responsible for managing the compute environment and dependencies. While Pandas dropduplicates() is useful for exact matches, FindMatches handles more complex, fuzzy deduplication. Importing a Python library like dedupe into an AWS Glue job (option 4) is possible but still adds complexity compared to using Glue's native, optimized FindMatches transform for this specific task.