BERT and Masked Language Modeling

BERT pretrains a bidirectional transformer encoder by predicting masked words, producing representations that read context from both directions.

Bidirectional context

BERT is an encoder-only transformer trained to understand text. Unlike autoregressive models that read strictly left to right, BERT attends to the entire sentence at once, so each word's representation reflects both what came before and after. This bidirectional view is well suited to tasks that require understanding a whole input, such as classification, entailment, and span extraction.

The masked language model objective

Kronos motion — cta read papers

You cannot train a bidirectional model with next-word prediction, because a word could trivially see itself. BERT instead uses masked language modeling: a fraction of tokens (about 15%) are replaced with a special mask token, and the model must predict the originals from surrounding context. Because prediction depends on both sides, the model is forced to build deep contextual representations rather than shallow left-to-right ones.

python

# masked language modeling target
# input:  the [MASK] sat on the mat
# label:  position 1 -> 'cat'
# loss:   cross-entropy over vocabulary at masked positions only

Pretraining then fine-tuning

BERT popularized the two-stage recipe now standard across the field. First, pretrain on a large unlabeled corpus with the masked objective, learning general language structure. Then fine-tune on a specific labeled task by adding a small output head and updating the weights. Pretraining is expensive and done once; fine-tuning is cheap and repeated per task, transferring broad knowledge to narrow problems.

Contextual embeddings

Before BERT, word embeddings gave each word a single fixed vector regardless of context, so 'bank' had one representation in every sentence. BERT produces contextual embeddings: the vector for a word depends on the sentence it appears in, distinguishing river bank from financial bank. These context-sensitive representations are a large part of why BERT improved so many understanding tasks.

Legacy and variants

BERT's release reset benchmarks across natural language understanding and inspired many variants that refine the training recipe, shrink the model, or extend it to other languages and domains. Encoder-only models remain the tool of choice for retrieval, classification, and embedding generation, while decoder-only models took over generation. The masked-prediction idea also carried over to vision and other modalities.

Encoder-only transformer with bidirectional attention.
Trained by predicting masked tokens from both sides.
Pretrain once, fine-tune cheaply per task.
Produces context-dependent word representations.