Tokenization and Subwords

Tokenization splits text into the units a model processes; subword schemes balance vocabulary size against the ability to represent any word.

Why tokenization matters

A language model does not see characters or words directly; it sees a sequence of integer token ids drawn from a fixed vocabulary. Tokenization is the step that converts raw text into those ids. The choice of tokenization affects vocabulary size, sequence length, how well rare and misspelled words are handled, and even how the model performs across languages. It is a small component with outsized influence.

Words versus characters

Kronos motion — power balance

Splitting on words gives short sequences but a huge vocabulary and no way to represent words not seen in training. Splitting into characters handles any input with a tiny vocabulary but produces very long sequences and forces the model to reassemble meaning from scratch. Neither extreme is ideal, which motivates a middle ground.

Subword tokenization

Subword methods break text into common fragments: whole words when frequent, smaller pieces when rare. The word 'tokenization' might become 'token' + 'ization'. This keeps the vocabulary at a manageable size (typically tens of thousands of tokens) while guaranteeing any string can be represented by falling back to short pieces or single characters. It gives the coverage of characters with sequence lengths closer to words.

Common algorithms

Byte-Pair Encoding (BPE): start from characters and iteratively merge the most frequent adjacent pair into a new token.
WordPiece: similar merging, but chooses merges that most improve the language-model likelihood; used by BERT.
Unigram / SentencePiece: start from a large candidate set and prune tokens to maximize likelihood, working directly on raw text including spaces.

python

# byte-pair encoding: merge the most frequent adjacent pair
# 'l o w e r' , 'l o w' -> learn merge ('l','o')='lo'
# repeat until vocabulary reaches target size

Practical consequences

Because models operate on tokens, costs and limits are measured in tokens, not words; a rough rule is that a token averages a few characters of English. Numbers, code, rare technical terms, and non-Latin scripts often fragment into many tokens, which lengthens sequences and can weaken performance. Byte-level tokenizers avoid unknown tokens entirely by falling back to raw bytes. Understanding how text tokenizes helps explain model behavior on edge cases such as arithmetic and unusual formatting.

Text becomes integer token ids from a fixed vocabulary.
Subwords balance vocabulary size and coverage.
BPE, WordPiece, and Unigram are the common schemes.
Tokenization shapes cost, length, and edge-case behavior.