A SageMaker training job reads millions of S3 files, each several megabytes, and training performance is slow. Which change will most quickly improve training throughput?

Reading millions of small files from S3 can be slow due to the overhead of many individual S3 requests. Amazon FSx for Lustre is a high-performance file system optimized for machine learning workloads that can transparently cache data from S3. By linking FSx for Lustre to the S3 bucket, the training job benefits from POSIX-compliant file access and the high throughput and low latency of FSx for Lustre, significantly improving data loading performance without modifying the training code. Moving data to S3 Express One Zone might offer slightly lower latency for individual S3 operations but doesn't fundamentally solve the overhead of millions of small file accesses. Amazon EFS is a general-purpose NFS file system, not specifically optimized for the high-throughput, low-latency needs of ML training with millions of small files, and copying data adds complexity. ElastiCache (Redis OSS) is an in-memory data store, not designed for streaming large volumes of file-based training data, and connecting it to S3 for this purpose is not a standard or efficient pattern.