A bank collects high-volume transactional data and streams it into Amazon Kinesis Data Streams using PutRecord. Network outages occur at certain times, and the data engineer needs exactly-once delivery semantics across the processing pipeline. Which approach will achieve that?

Kinesis Data Streams offers at-least-once delivery, meaning records might be delivered multiple times, especially during network issues or consumer failures. To achieve exactly-once processing, the most robust approach is to make your processing application idempotent. By embedding a unique ID (e.g., a UUID or a combination of source ID and timestamp) in each record at the source, your downstream application can detect and discard duplicate records based on this ID. Updating checkpoint configuration in Managed Service for Apache Flink helps with fault tolerance and recovery but doesn't inherently guarantee exactly-once delivery from the source to the sink without additional application-level deduplication. Designing the data source to avoid multiple ingestions is ideal but doesn't account for Kinesis's at-least-once delivery guarantee or potential retries from the producer side during network outages. Switching to Amazon EMR with Apache Flink and Spark Streaming doesn't inherently solve the exactly-once delivery problem; these frameworks also require idempotent processing or unique identifiers for deduplication to achieve exactly-once semantics.