A streaming Dataflow job (Streaming Engine, horizontal autoscaling, max_workers=1000) reading Pub/Sub notifications from GCS emits one element per CSV line. The job is using only 10 workers and autoscaler isn't adding more. How do you improve parallelism and let autoscaler scale?

The correct answer is to change the pipeline code to add a Reshuffle step to break fusion. Fusion optimization in Dataflow can sometimes lead to bottlenecks if a fused stage has a hot key or uneven work distribution. Adding Reshuffle forces Dataflow to materialize data and redistribute it, allowing the autoscaler to better identify and address parallelism needs. Enabling Vertical Autoscaling or using Dataflow Prime's Right Fitting would increase worker capacity but wouldn't address the underlying parallelism issue if the work isn't being distributed effectively across existing workers. Increasing the job's maximum number of workers is ineffective if the autoscaler isn't adding more due to perceived lack of parallelizable work.