Data Pipelines for Fusion ML
The unglamorous work of collecting, cleaning, aligning, and labeling data that determines whether any model succeeds.
Data before models
Model quality is bounded by data quality. In fusion, most of the effort in a successful ML project goes into building reliable data: pulling signals from diagnostic archives, aligning them in time, handling missing and faulty channels, and creating trustworthy labels. A brilliant model on bad data fails.
Alignment and resampling
Diagnostics sample at different rates, with different clocks and latencies. Bringing them onto a common time base without introducing artifacts, and without leaking future information for real-time use, is a careful step. Errors here silently corrupt every downstream result.
Cleaning and quality flags
- Detecting and marking saturated or dropped-out channels
- Handling calibration changes between campaigns
- Flagging shots with known diagnostic faults
- Recording provenance so results can be traced back
Labeling
Supervised tasks need labels, which for events like disruptions or instabilities come from experts and are limited, noisy, and sometimes inconsistent. Documenting how labels were made, measuring inter-expert agreement, and versioning label sets are part of honest practice. Unlabeled data can be used with semi-supervised or unsupervised methods.
Reproducibility
A result is only credible if the exact data, cleaning steps, and splits can be reproduced. Versioning datasets, recording every transformation, and fixing random seeds turn a one-off number into a repeatable finding. For design work such as at Kronos, where results feed decisions, this traceability is a requirement, not a nicety. The pipeline is where trustworthiness is won or lost.