Observability & SLOs

The orchestration layer is measured against explicit service-level objectives for latency, lag, and safety-check availability.

L4 · Orchestration — its place in the stack (left, click any layer) and its internal components (right). Telemetry rises; control descends.

Measure what protects the machine

Orchestration health is defined by explicit SLOs, not vibes. The metrics that matter are the ones that bound how quickly and correctly a proposal becomes a gated command, and how reliably safety checks are available. These are tracked live and gate deploys via replay regression.

Core SLOs

SLO	Target class	Why it matters
gate decision latency	tight, bounded	time from proposal to accept/reject
envelope-check availability	near-total	a missing check means fail-closed rejects
consumer lag	bounded	late state degrades control quality
dead-letter rate	near-zero	poison events signal upstream defects

Golden signals per domain

Latency: end-to-end and per-gate-stage, so a slow envelope check is caught.
Saturation: consumer lag and resource-pool utilization per failure domain.
Errors: rejects by stage, dead-letter rate, circuit-breaker trips.
Traffic: event and command rates, distinguishing burst (disruption) from steady load.

Safety metrics are first-class

Some metrics are not about performance but about the integrity of the guard: the fraction of commands that passed with a fresh (non-stale) envelope, the number of commands rejected for safety reasons, and the availability of the rules and envelope services. A degradation here pages immediately, because a machine whose safety checks are flaky must hold, not proceed.

Observability feeds control decisions

python

if slo.envelope_availability < THRESHOLD:
    orchestrator.enter_safe_hold()      # do not gate new energizing actions
    page("envelope checker degraded")
if slo.consumer_lag > LAG_CRIT:
    scheduler.pause_low_priority()      # shed load

All metrics share the correlation IDs used by tracing and the lineage bus, so a numeric SLO breach can be drilled straight to the causal events. SLOs are exercised in simulation for both machines ahead of FOAK.