Watermarks and late events in stream processing: an interactive simulator

A stream processor that counts events per minute has to decide when a minute is over. Events carry the time they happened, their event time, but reach the processor later and out of order. A watermark is the processor’s running estimate that no event older than some time is still coming. When it passes the end of a window, the window’s result is emitted, and events that turn up after that are late.

The playground plots every event by when it happened against when it arrived. The staircase is the watermark, and each window’s result goes out where the staircase crosses the top of its band.

Completeness against latency

A watermark that trails the newest event by 2 seconds sends each window’s result 2.9 seconds after the window ends, with 95.5% of its events; the 13 events that missed their window are dropped, almost all from devices that were offline. With no delay, results come in under a second but 20 events are dropped. Waiting 15 seconds still drops 7, because some devices were offline for longer than that, and now every result waits 15 seconds for them. No fixed delay is right for every event: the delay sets how long the common case waits, not whether stragglers are caught.

Allowed lateness: send early, then correct

The better answer is to emit on time and fix it later. With allowed lateness, the processor keeps a window’s state for a while after the watermark passes it, and each late event updates the result: with a 2 second watermark and 30 seconds of lateness, 12 late events update their windows and only 1, arriving more than 30 seconds after its window closed, is dropped. Whatever reads the results has to handle updates, by upserting the window’s row rather than appending, and events later even than that go to a side output for a batch job to reconcile instead of vanishing.

• Event time vs processing time. Windowing by arrival time is simpler and never late, but a device that reconnects after an hour dumps an hour of events into the current minute.
• Where watermarks come from. From the source, such as per Kafka partition, with the operator’s watermark the minimum across its inputs. One idle partition holds every window open unless it is marked idle.
• Other windows. Sliding windows overlap, so each event lands in several; session windows close after a gap in activity, so they too wait for the watermark to pass the gap.

Watermarks and lateness in code

// Flink: count clicks per page in 10 second windows of event time
DataStream<PageCount> counts = clicks
    .assignTimestampsAndWatermarks(
        WatermarkStrategy.<Click>forBoundedOutOfOrderness(Duration.ofSeconds(2))  // watermark = newest - 2 s
            .withTimestampAssigner((click, ts) -> click.eventTime))
    .keyBy(click -> click.page)
    .window(TumblingEventTimeWindows.of(Duration.ofSeconds(10)))
    .allowedLateness(Duration.ofSeconds(30))   // late clicks update the result
    .sideOutputLateData(tooLate)               // later still: set aside, not silently lost
    .aggregate(new CountClicks());

Kafka Streams calls allowed lateness a grace period, and Spark Structured Streaming uses a single withWatermark delay both to decide when to emit, in append mode, and how long to accept late data.

In a system design interview

When you aggregate a stream, say which clock the windows use and what happens to late data: “Ad clicks are counted per minute by event time. The watermark trails by 10 seconds, so dashboards update within about a minute and ten seconds. Clicks up to an hour late update the minute’s row in the sink; anything later goes to a late-events topic, and the nightly batch job over raw clicks recomputes billing totals exactly.” Expect the follow-ups: duplicates when a late update is retried, how the sink handles updates, and how far the real-time numbers may drift from the batch ones.

Read the guide to windowing and watermarks or batch and stream processing, see Kafka partitions for where the events come from, or watch a strong candidate design an ad click aggregator.

Questions people ask