Streaming pipeline uses hopping windows but some late-arriving data is not being treated as late, causing incorrect aggregations. How do you capture late data into the correct window?

The correct answer is to use watermarks to define the expected data arrival window and allow late data. Watermarks are a crucial concept in stream processing for handling out-of-order or late-arriving data. They indicate the point in event time up to which all data is expected to have arrived. By setting an appropriate watermark and allowing late data, the system can correctly assign late events to their intended windows, even if they arrive after the initial window processing. Changing to tumbling windows would not solve the late data problem; tumbling windows are fixed, non-overlapping, and would still miss late data if it arrives after the window closes. Session windows are defined by user activity and gaps in that activity, which isn't directly addressing the issue of data arriving after a window has conceptually closed. Expanding the hopping window might temporarily mask the problem by making the window longer, but it doesn't fundamentally solve the issue of data arriving after the expanded window's end or provide a robust mechanism for handling varying lateness.