Autoregressive Language Modeling

Autoregressive models factor the probability of a sequence into a product of conditional next-token probabilities, the foundation of generative text models.

The chain rule of probability

An autoregressive language model assigns a probability to a sequence of tokens by factoring it with the chain rule: P(x1..xn) equals the product over t of P(xt given x1..x(t-1)). Each factor is a conditional distribution over the vocabulary, produced by the network's softmax output. Training maximizes the log of this product, which decomposes into a sum of per-token cross-entropy terms, so a single forward pass over a sequence yields a loss at every position.

Teacher forcing

During training the model conditions on the true previous tokens rather than its own predictions, a technique called teacher forcing. This makes training parallel and stable. At inference the model must instead condition on its own generated tokens, creating a distribution mismatch known as exposure bias: small errors can compound over a long generation. Sampling temperature and careful decoding mitigate but do not eliminate this.

import torch.nn.functional as F
# logits: (T, V) for one sequence; targets: (T,) next tokens
loss = F.cross_entropy(logits, targets)      # mean NLL per token
perplexity = loss.exp()                        # geometric-mean branching

Perplexity

Perplexity, the exponential of the mean per-token negative log-likelihood, is the standard intrinsic measure of a language model. It can be read as the effective number of equally likely choices the model faces at each step; lower is better. Perplexity is only comparable across models that share a tokenizer, since it is defined per token.

• Greedy decoding picks the highest-probability token each step
• Temperature scales logits to make sampling sharper or flatter
• Top-k and nucleus (top-p) sampling truncate the tail before sampling
• Beam search keeps several partial sequences for likelihood-oriented tasks

Why it generalizes

Next-token prediction seems narrow, yet to predict the next token well across broad text a model must represent syntax, facts, and long-range dependencies. This is why the objective, applied at scale in decoder-only models, produces general capabilities. The same factorization underlies autoregressive models of audio such as WaveNet, showing the principle is not specific to language.