A disk I/O–intensive Hadoop job runs much slower on Dataproc using Cloud Storage for intermediates than on-prem bare-metal HDFS. How do you fix this while separating storage and compute?

The problem describes a disk I/O-intensive job, indicating that the bottleneck is how quickly data can be read from or written to storage. Cloud Storage, while highly scalable, has higher latency than local HDFS, which can significantly impact jobs with frequent, small I/O operations or tight coupling between I/O and computation. Provisioning more persistent disk and using native HDFS for intermediates moves the data closer to the compute, leveraging the lower latency and higher throughput of local disk I/O, thus improving performance. Giving the cluster more memory might help if the job was memory-bound, but it won't resolve a disk I/O bottleneck. Increasing VM CPU cores or adding NICs to scale network bandwidth would be relevant if the bottleneck was network throughput to Cloud Storage, but the core issue is the latency and I/O characteristics of Cloud Storage for intermediate data in this specific scenario.