Watermarks & Event Time

Consumers reason about event time, not arrival time, and use watermarks to know when a time window is safe to close.

Event time vs processing time

Diagnostics from the breeder and burner are timestamped at the source (occurred_at). Because of buffering, retries, and multi-partition merges, they can arrive out of order and late. Any computation that windows over time (for example, energy or neutron-flux integrals during a shot) must use event time, or it will attribute samples to the wrong window.

The watermark

A watermark is the consumer's assertion that it has seen all events with occurred_at at or before time W. It advances as ordered streams progress and lets the consumer close windows deterministically. Late events arriving after their window closed are handled explicitly, not silently misplaced.

watermark = min(latest_occurred_at[p] for p in partitions) - allowed_lateness
for window in windows_ending_before(watermark):
    emit(aggregate(window))          # safe to close: no earlier event can still arrive
# late event (occurred_at < watermark): route to correction path, never drop
if e.occurred_at < watermark:
    corrections.publish(e)

Allowed lateness

• Set from measured worst-case delivery lag so genuinely-late samples still land in-window.
• Too small: correct samples get treated as late; too large: windows close slowly.
• Tuned per topic; high-rate magnetics tolerate less lateness than slow analytics.

Determinism preserved

Watermark-driven windowing is deterministic given the recorded stream: replay reproduces the same window closures and the same aggregates. This is why post-shot analysis of a breeder disruption matches what the live system computed, and why cross-partition merges during a transient are reproducible (see ordering).

Watermarks decouple correctness from timeliness: under backpressure windows close later but never incorrectly.