Data Systems

Stream Processing

Stream processing computes over unbounded data as it arrives, using windows, watermarks, and stateful operators to produce continuous results.

Unbounded computation

Batch processing runs over a finite, complete dataset. Stream processing runs over an unbounded flow of events that never ends, producing results continuously. The central challenge is that you can never see all the data, so any aggregation must be scoped to a bounded slice, and you must decide what to do about events that arrive out of order or late.

Windows

Kronos motion — data assimilation

A window bounds an aggregation in time. Tumbling windows are fixed, non-overlapping intervals (every 5 minutes). Sliding windows overlap (a 5-minute window advanced every minute). Session windows group events separated by less than a gap timeout, so activity defines the boundary. The window type must match the question: throughput per interval wants tumbling; a rolling average wants sliding.

Event time versus processing time

Two clocks matter. Event time is when the event actually happened; processing time is when the system saw it. Network delays make them differ, and results grouped by event time are far more meaningful. But event time forces the system to wait for stragglers, which brings us to watermarks.

Watermarks

A watermark is the system's assertion that it has probably seen all events up to a given event time. When the watermark passes a window's end, the window can be closed and emitted. Events arriving after the watermark are late and are either dropped, sent to a side output, or used to emit a correction, depending on the configured policy. The watermark is a tunable bet on how late data can be.

State and fault tolerance

Aggregations hold state (running counts, sums) between events
State is checkpointed periodically to durable storage
On failure the operator restores the last checkpoint and replays from the matching source offset
This combination yields effectively-once results despite crashes

See event streaming, real-time analytics, and lambda and kappa architectures.