Data Pipelines and Ingestion
The ingestion layer turns raw, messy sensor streams into clean, aligned, validated data the twin can trust.
Garbage in, garbage out
A twin is only as good as the data feeding it. The ingestion layer stands between the sensors and the models, and its job is to deliver clean, correctly timed, validated measurements. Skimping here corrupts everything downstream: an estimator fed bad data produces confident nonsense. Ingestion is unglamorous and decisive.
What ingestion does
- Acquire streams from many diagnostics at very different rates
- Timestamp against a common, precise clock
- Apply calibrations to convert raw counts to physical units
- Validate: range checks, rate-of-change checks, cross-channel consistency
- Flag or reject bad channels and record why
- Buffer and route data to the real-time path and to storage
Handling missing and bad data
Sensors drop out, saturate, and drift. The pipeline must detect these conditions and mark the data accordingly, so the estimator can widen its uncertainty or fall back on other sensors rather than treating a stuck reading as truth. Graceful handling of missing data is what gives a twin its robustness.
Batch and stream
Some data flows continuously and must be processed as it arrives, the streaming path feeding real-time estimation. Other data, such as post-campaign material samples in a fusion plant, arrives in batches and is processed on its own schedule. A mature pipeline handles both and reconciles them into one consistent record.
In the Kronos program
For the Hyperion breeder and the burner, the ingestion layer must span magnetic signals sampled thousands of times per second, optical and neutron diagnostics, and slow structural and material measurements. It is being designed now against the diagnostic set defined in the machine design, so that when hardware exists the pipeline is ready. Until then it is exercised with simulated and reference data. See diagnostic fusion and data standards.