Embedding
A learned mapping of discrete items into a continuous vector space where geometry encodes meaning.
Definition
An embedding represents discrete items, words, categories, or nodes, as dense vectors in a continuous space, learned so that similar items sit near one another. Distances and directions in this space capture relationships the raw symbols do not express.
Embeddings are typically learned as a byproduct of a larger task and then reused, and their quality is judged by whether distances and neighborhoods in the space match human notions of similarity. Pretrained embeddings let small projects benefit from patterns learned on far larger corpora.
The geometry of an embedding space can encode biases present in the training data, since it faithfully captures the statistical regularities it was shown, including undesirable ones. Auditing and, where needed, mitigating such bias is part of responsible use. On the positive side, the same geometry powers similarity search and retrieval, letting systems find related items by proximity in the space.
Properties
- Similarity: nearby vectors mean similar items.
- Compactness: fewer dimensions than one-hot encoding.
- Transferable: embeddings trained on one task often help another.
Why it matters
Embeddings turn symbolic data into a form neural networks can operate on and generalize over. Word embeddings famously encode analogies as vector arithmetic, and modern retrieval systems compare embedding vectors to find relevant content.
Fusion connection
Embedding categorical design choices, such as material or coil layout, gives Kronos surrogates a continuous space in which optimizers can interpolate between discrete Hyperion options.