A developer built a microservice that uses AWS Lambda to process messages from an Amazon SQS standard queue. The Lambda function enriches each SQS message by calling external APIs and then writes the data to an Amazon Redshift table. The queue must support up to 1,000 messages per second. During testing, duplicate rows were repeatedly inserted into the Redshift table because duplicate SQS messages were processed; all duplicate messages were enqueued within 1 minute of each other. How should the developer fix this problem?

The problem describes duplicate messages being processed, leading to duplicate Redshift entries, with duplicates arriving within 1 minute. Amazon SQS Standard queues do not guarantee strict message ordering or exactly-once processing, which can lead to duplicates under certain conditions. Amazon SQS FIFO (First-In, First-Out) queues, however, guarantee exactly-once processing and strict message ordering. By switching to an SQS FIFO queue and enabling message deduplication, the system ensures that messages are delivered exactly once and that any duplicate messages sent within the 5-minute deduplication interval are not processed again. Lowering Lambda concurrency might reduce the rate of duplicates but doesn't prevent them, as the SQS Standard queue itself can still deliver duplicates. Using /tmp for tracking is not reliable across Lambda invocations and doesn't scale. SQS Standard queues do not support message group IDs or message deduplication; these features are exclusive to FIFO queues.