Decoder-Only Language Models
Decoder-only transformers predict the next token from all previous tokens, using causal masking so a single model both learns from and generates text.
One stack, left to right
The GPT family popularized the decoder-only transformer: a stack of transformer blocks trained to predict the next token given every token before it. Unlike an encoder that reads bidirectionally, a decoder-only model applies a causal mask so position i can attend only to positions 1 through i. This makes the model naturally generative, because at inference it can extend a sequence one token at a time, feeding each prediction back as input.
The causal mask
Causality is enforced inside self-attention by adding a mask to the attention logits before the softmax, setting the score of any future position to negative infinity so it receives zero weight. During training this lets the model compute the loss for every position in parallel from a single forward pass, because each position already only sees legal context. The same mask at inference guarantees generation depends only on the past.
Training objective
The objective is straightforward maximum likelihood: minimize the average negative log-probability the model assigns to the true next token across the corpus. This single self-supervised task, applied at scale to diverse text, is enough to induce grammar, factual associations, and reasoning-shaped patterns without any task-specific labels. See autoregressive modeling for the probabilistic view.
- Pre-norm residual blocks with attention and an MLP, repeated many times
- A shared token embedding and output projection in many designs
- Sampling strategies such as temperature, top-k, and nucleus sampling shape generation
- Context length is bounded by position handling and attention cost
Why decoder-only won at scale
A decoder-only model unifies understanding and generation in one architecture and one objective, which simplifies scaling. Because every position produces a training signal, the objective is dense and sample-efficient per token. Efficiency techniques such as flash attention, multi-query attention, and RMSNorm were widely adopted to make very large decoder stacks practical. The architecture stays conceptually simple while the capability comes from scale, data, and careful training.