Real-Time vs Archival Data
Live data is optimized for low latency and short horizons; archival data is optimized for durability, completeness, and long-term readability.
Two jobs, two designs
Data serves two different purposes with conflicting demands. Real-time data supports decisions happening now: monitoring, control, and alerting. Archival data supports understanding later: analysis, reproduction, and the historical record. Trying to serve both with one system usually compromises both, so mature programs run parallel paths.
Real-time path
- Low latency: data must be usable within milliseconds to seconds.
- Recent horizon: only the last minutes or hours are kept hot.
- Lossy tolerance: an occasional dropped sample may be acceptable for a dashboard.
- Optimized for reads of the newest data.
Archival path
- Completeness: every sample is preserved; nothing is dropped.
- Durability: multiple copies, checksums, and stable formats.
- Latency-insensitive: it is fine if writes settle seconds or minutes later.
- Optimized for occasional deep reads over long spans.
The lambda pattern
A common design, sometimes called the lambda architecture, splits ingestion into a fast streaming layer for live views and a batch layer that writes the authoritative archive. The two are reconciled so the archive is the source of truth and the stream is a low-latency approximation. See also storage tiers.
In a fusion program
During a pulse, operators need live diagnostic readouts to see machine state; after the pulse, the complete, calibrated record must be preserved for analysis and for the published archive. For the Hyperion breeder design today, this is expressed in simulation and format contracts, since the machines are design and simulation rather than built hardware. The archival path is where reproducibility lives, so it is engineered for permanence over speed.