Quantization and Pruning for the Edge

Models are shrunk to meet the L1 latency budget through quantization and pruning, with accuracy and calibration re-verified so compression never silently degrades safety.

Meeting the microsecond budget

The L1 control plane runs on a bounded time budget; a model that is accurate but too slow cannot deploy. Quantization (reducing numeric precision) and pruning (removing redundant weights and structure) shrink models to fit that budget. Both are lossy, so the governing rule is that compression is only acceptable if the compressed model still passes the same accuracy, calibration, and safety gates as the original.

Quantization is done with awareness of the target. Post-training quantization is fast; quantization-aware training bakes the numeric format into training so the model learns to tolerate it, and is used where post-training quantization loses too much at the regime boundaries that matter. Pruning is structured where possible so the sparsity actually translates to speed on the edge hardware, not just to fewer nominal parameters.

Compression with guardrails

• Quantization-aware training for accuracy-critical models
• Structured pruning that maps to real hardware speedups
• Re-check accuracy, calibration, and safety-envelope after compression
• Watch tail behavior: compression can hurt rare regimes most
• Parity gate on the final compiled form

q = quantize_aware_train(model, fmt='int8', data=calib_set)
for gate in ['accuracy','calibration','safety']:
    assert passes(q, gate), f'compression broke {gate}'
# rare-regime check: error on tail subset must not regress
assert tail_error(q) <= tail_error(model) * (1 + TAIL_TOL)

The subtlety Kronos guards against is that compression often degrades exactly the rare, tail situations — disruptions on the breeder, off-normal transients on the burner — that safety models exist to handle, while leaving average accuracy untouched. Every compressed model is therefore re-evaluated on its tail subset before compilation proceeds to the parity gate.