Tokenization
Splitting text into the discrete units a language model reads and predicts.
Definition
Tokenization converts raw text into a sequence of tokens, the atomic units a model processes. Modern systems use subword tokens: common words become single tokens while rare words split into pieces, balancing vocabulary size against sequence length.
Because a model's cost and context limit are measured in tokens, tokenization has direct practical consequences: a passage of code or a non-English language may consume far more tokens than an equivalent amount of English prose, affecting both speed and how much fits in context.
The vocabulary is fixed when the model is built, so text that tokenizes poorly, unusual symbols, some languages, long numbers, costs more and can be handled less reliably than common English. This has quiet fairness and efficiency implications, and it explains why tasks involving arithmetic or code sometimes behave oddly. Designers weigh vocabulary size against sequence length, since a larger vocabulary shortens sequences but enlarges the model's input and output layers.
Approaches
- Word-level: simple but a huge vocabulary and poor on rare words.
- Character-level: tiny vocabulary but long sequences.
- Subword (BPE, WordPiece): the practical middle ground.
Why it matters
Tokenization determines how text maps to model inputs and directly affects sequence length, context limits, and cost. Numbers, code, and non-English text can tokenize inefficiently, which shapes model behavior in subtle ways.
Fusion connection
Careful tokenization of technical notation matters when language tools process Kronos documents dense with symbols, units, and equations, so that quantities are not fragmented into meaningless pieces.