A weather forecasting company needs to process hundreds of gigabytes of data with sub-millisecond latency. The company has an HPC environment on-premises and wants to expand to the cloud. A solutions architect must choose a highly available cloud storage solution that supports large amounts of sustained throughput and allows thousands of compute instances to simultaneously access and process the entire dataset. What should the solutions architect choose?

Amazon FSx for Lustre persistent file systems are designed for high-performance computing (HPC) workloads, offering sub-millisecond latency and high throughput, which aligns with the requirement for processing hundreds of gigabytes of data with low latency. Persistent file systems retain data even if the file system is deleted, making them suitable for long-term storage and high availability. Scratch file systems are temporary and lose data upon deletion, making them unsuitable for persistent data needs. Amazon EFS, even with Provisioned Throughput, generally offers higher latency and lower throughput compared to FSx for Lustre, making it less ideal for the extreme performance demands of HPC. Bursting Throughput mode on EFS is even less suitable for sustained high throughput.