A Dataflow job ingesting gzipped text files uses SideInputs to join data and writes errors to a dead-letter queue, but job runtime is longer than expected. What change speeds up the pipeline?

Using CoGroupByKey instead of a SideInput can significantly speed up the pipeline, especially when the side input data is large or frequently updated. SideInputs are broadcast to all workers, which can be inefficient for large datasets due to network overhead and memory consumption on each worker. CoGroupByKey performs a shuffle operation to bring together elements with the same key, allowing for scalable joins of large datasets without the overhead of broadcasting. Switching to compressed Avro might offer some I/O improvements but won't address the fundamental inefficiency of large SideInputs. Reducing the batch size could increase overhead and potentially slow down processing. Retrying records that throw an error is a fault-tolerance mechanism and does not directly improve pipeline speed.