Retrieval-Augmented Generation
Retrieval-augmented generation fetches relevant documents from an external store and conditions a generative model on them, grounding outputs in retrievable knowledge.
Parametric plus non-parametric knowledge
A language model stores what it knows in its weights, which is fixed after training, hard to update, and prone to confabulation when asked about facts it never learned well. Retrieval-augmented generation (RAG) adds a second, non-parametric source of knowledge: an external collection of documents that the system searches at inference time. The model first retrieves passages relevant to the query, then generates its answer conditioned on both the query and the retrieved text, grounding its output in explicit, inspectable evidence.
The two stages
Retrieval usually works by embedding the query and every document into a shared vector space and finding the nearest documents by similarity, a dense retrieval that captures meaning beyond keyword overlap. The top passages are then concatenated with the query and passed to the generator, which attends over them while producing its response. Some systems retrain the retriever and generator jointly; many combine an off-the-shelf retriever with an off-the-shelf model, which is simpler and still effective.
- Knowledge can be updated by changing the document store, without retraining
- Answers can cite the retrieved sources, improving verifiability
- Reduces confabulation by grounding generation in real passages
- Retrieval quality bounds answer quality: missing evidence cannot be used
Design choices that matter
The document chunk size, the number of passages retrieved, how they are ranked and ordered, and how the generator is prompted all affect results. Too little retrieved context starves the model; too much dilutes the relevant evidence and can exceed the context window. Re-ranking retrieved passages with a stronger model and filtering low-relevance ones are common refinements.
Relation to memory and attention
RAG is the practical, large-scale cousin of memory-augmented networks: the document store is an external memory, and dense retrieval is a content-based read. It differs in that the memory holds raw text and the read is a discrete search over a large index rather than a soft attention over slots. By separating storage from the generator, RAG lets a model draw on far more knowledge than its parameters could hold, and keeps that knowledge current.