A Dataproc Spark job with many shuffles reads parquet files ~200–400 MB each and runs on preemptible workers with only two non‑preemptible nodes. To improve performance cost‑effectively, what change should you make?

The correct answer improves performance cost-effectively by addressing I/O bottlenecks and shuffle performance. Switching from HDDs to SSDs significantly boosts I/O operations, which is crucial for jobs with many shuffles and reading numerous files. Increasing the boot disk size provides more local storage for shuffle data, reducing reliance on network I/O and improving shuffle performance, especially on preemptible workers that might be re-provisioned. Increasing parquet file size to 1 GB might not be optimal; Spark generally performs well with files around 128-256 MB. Switching to TFRecords offers no inherent performance advantage over Parquet for this scenario and might introduce conversion overhead. Copying data to HDFS on SSDs and back to GCS adds unnecessary data transfer overhead and complexity without directly addressing the shuffle performance on preemptible workers.