You must process batch and streaming events in Dataflow when events can arrive late or out of order. How should you design the pipeline to handle late or out-of-order data?

Dataflow, leveraging Apache Beam, uses event time and watermarks as its core mechanism for handling late and out-of-order data in both batch and streaming pipelines. Event time refers to when the event actually occurred, not when it was processed. Watermarks are a system-generated indicator of how complete the data is up to a certain point in event time. This allows the pipeline to process events based on their true occurrence time, even if they arrive out of order or with significant delays. Using a single global window would eventually capture all events but wouldn't provide timely results for streaming data or handle late events gracefully within bounded windows. Sliding windows can help include lagged events but don't inherently manage the concept of event time completeness as effectively as watermarks. While source timestamps are crucial, relying solely on them without the framework of watermarks doesn't provide a robust strategy for determining when to close windows or how to deal with late arrivals.