A company runs a custom application on EC2 instances in an Auto Scaling group that processes records through multiple sequential, compute-heavy steps. Each step takes up to 5 minutes. Currently, if any step fails the entire record must be reprocessed from the start. The company wants to reprocess only the failed steps with the least operational overhead. What is the most operationally efficient architecture to meet this?

The correct answer is to use AWS Step Functions. Step Functions are designed for orchestrating multi-step workflows, making them ideal for this scenario. Each step can be implemented as a Lambda function, and Step Functions automatically manage state, retries, and error handling. If a step fails, Step Functions can be configured to retry only that specific step or move to a different error handling path, eliminating the need to reprocess the entire record. This provides high operational efficiency as Step Functions handle the complexity of workflow management. The other options are less efficient: S3 with SNS and an EC2 instance for polling introduces more operational overhead for managing the EC2 instance and custom polling logic. Storing intermediate results in S3 is good, but the orchestration is manual. Refactoring into a container on Fargate and having it invoke itself for state passing is complex and error-prone, as managing state within a self-invoking container is not a native pattern for Fargate. Kinesis and Lambda functions can process records, but orchestrating sequential, dependent steps and managing retries for individual steps across multiple Lambda invocations would require significant custom logic, increasing operational overhead compared to Step Functions.