Mixture of Experts

Mixture-of-experts models route each input to a few specialized subnetworks, growing total capacity while keeping per-input computation low.

Conditional computation

A mixture-of-experts (MoE) layer contains many parallel subnetworks, the experts, and a lightweight gating network that decides which experts to use for each input. Rather than running every input through every parameter, MoE activates only a small subset of experts per token. This is conditional computation: the model can hold a very large number of parameters while spending only a fraction of them on any single input.

Sparse routing

Kronos motion — neural operator

The gate scores all experts for a given token and selects the top few, often just one or two, sending the token only to those. The chosen experts process the token and their outputs are combined, weighted by the gate scores. Because only the selected experts run, computation per token stays roughly constant even as the number of experts, and therefore total parameters, grows large. This decouples model capacity from inference cost.

python

# top-k gating (sketch)
# scores = gate(x)                 # one score per expert
# idx = top_k(scores, k=2)         # pick 2 experts
# out = sum(softmax(scores[idx]) * expert_i(x) for i in idx)

Load balancing

A recurring problem is that the gate may favor a few experts, leaving others untrained and underused. MoE training adds an auxiliary load-balancing loss that encourages tokens to spread evenly across experts. Capacity limits cap how many tokens each expert handles per batch, dropping or rerouting overflow. Getting this balance right is central to training MoE models well.

Advantages and costs

MoE lets a model reach enormous parameter counts, with capacity that specialized experts can exploit, at a compute cost far below a dense model of the same size. The trade-offs are engineering ones: all experts must be held in memory even though few run per token, routing complicates distributed training across devices, and load imbalance can waste capacity. MoE is attractive when memory is available but per-token compute is the binding constraint.

Use in large models

Sparse MoE layers appear in several large language models, typically replacing the feedforward sublayer of transformer blocks with an expert layer while keeping attention dense. This gives a large increase in parameters and quality for a modest increase in compute per token. The approach is one of the main techniques for scaling models beyond what dense architectures make practical.

Many experts, but only a few run per input.
A gate routes each token to top experts.
Load-balancing losses keep experts evenly used.
Grows capacity without growing per-token compute.