Failure Domains & Bulkheads

Orchestration is partitioned into isolated failure domains so a fault in one subsystem cannot cascade into machine-wide loss of control.

Contain the blast radius

A single failing component, a stuck consumer, a crashed twin service, a flooded topic, must not take down the whole control plane. Layer 4 is partitioned into bulkheaded failure domains with bounded resource pools, so a fault is contained to its domain. The breeder magnetics pipeline failing does not stall the burner supervisor, and an analytics consumer crash never touches the gating path.

Bulkhead patterns

• Separate resource pools (threads, connections, buffers) per domain so one cannot exhaust another.
• Circuit breakers on cross-domain calls: a failing dependency is cut off and the caller degrades gracefully.
• The gating pipeline runs isolated from the ML stack so a twin/copilot outage cannot disable safety checks.

breaker = CircuitBreaker(fail_max=5, reset_after=30)
@breaker
def call_twin(state):
    return twin.predict(state)
# breaker OPEN -> skip twin, fall back to conservative control, never block the gate

Degradation modes

Each domain has a defined degraded mode. If the twin is unavailable, orchestration holds the current operating point and routes any needed change to a human gate rather than acting blind. If the copilot is unavailable, no new proposals arrive and the machine simply holds. In all cases the rules and envelope remain in force because they are in their own domain.

No shared fate with intelligence

The most important bulkhead is between the learned stack (L3/L5) and the guard (rules, envelope, interlocks). They share no process, no library that can crash both, and no resource pool. This is why a total ML-stack failure degrades the machine to a safe hold rather than to loss of control, the design principle behind the whole gating pipeline.

Validated by fault injection

Failure domains are proven by injecting faults, killing consumers, partitioning the network, corrupting a topic, in simulation against the twin, confirming each stays contained ahead of FOAK. Contained faults surface as SLO breaches, not silent machine-wide failures.