Applications

Simulation Data Management

Organizing, versioning, and finding the large, heterogeneous datasets that simulation campaigns produce.

The scale problem

A simulation program generates many kinds of data at large volume: input decks, meshes, output fields, logs, derived summaries, and figures, across thousands of runs. Without deliberate management this becomes an unnavigable pile where nobody can find the run behind a given figure or tell which version of a result is current. Data management is the discipline that keeps it usable.

What must be managed

Kronos motion — data assimilation

Versioning: which version of each dataset is current, and its history
Metadata: enough context to find and interpret a dataset later
Storage tiering: hot data fast and accessible, cold data cheap and durable
Retention: what to keep, what to regenerate, what to discard
Discovery: search across runs by parameters and results

Metadata is the product

Raw output arrays are nearly useless without metadata: the parameters that produced them, the code version, the units, the meaning of each field. Rich, consistent metadata is what turns a directory of files into a queryable dataset. Capturing it automatically at run time (see HPC orchestration) is far more reliable than adding it later.

Regenerate versus store

Because runs are reproducible, some outputs can be regenerated on demand rather than stored forever. This is a real trade: storage is finite, and regeneration takes compute and time. Data management makes this choice explicitly per dataset rather than defaulting to hoarding everything or discarding recklessly.

python

def find_runs(catalog, **criteria):
    return [r for r in catalog
            if all(r['params'].get(k)==v for k,v in criteria.items())]

# e.g. find_runs(catalog, triangularity=-0.30, field_T=16.84)

Kronos use

Breeder and burner campaigns produce the data that feeds design decisions and licensing evidence. Managing it well is what makes results findable for review and what feeds the design-build-operate loop, where old data must be compared against new.