Batch MapReduce analytics on Hadoop are falling behind after a data-volume increase. How can the team improve responsiveness without raising costs?

Rewriting the job in Apache Spark is the most effective way to improve responsiveness without significantly raising costs. Spark processes data in-memory, leading to significantly faster execution compared to Hadoop MapReduce, which writes intermediate results to disk. This can drastically reduce job completion times. Rewriting in Pig would not offer the same performance gains as Pig still relies on MapReduce for execution. Increasing the Hadoop cluster size would directly raise costs, which the question explicitly aims to avoid. Decreasing the cluster size would worsen performance, and while Hive offers a SQL-like interface, it still runs on MapReduce or Spark, so a smaller cluster would still be detrimental.