Retrieval, context and grounding a model in your data

A model that answers from its weights is recalling. A model given the relevant document and asked to answer from it is summarising. The second is far more reliable, and building the pipeline that supplies the document is the actual work.

Why it helps

Fabrication happens when a model produces a plausible continuation without having the information. Supplying the information removes the condition that causes it.

It also solves several adjacent problems: information newer than the training cutoff, proprietary data the model never saw, and the ability to cite a source that a reader can check.

That last point matters more than it is usually given credit for. A grounded answer with a link is verifiable; an ungrounded one is not.

How the pipeline works

Chunking. Documents are split into passages, because whole documents are too large to supply and too coarse to retrieve precisely.

Chunk size is a real design decision. Too small and passages lose the context that makes them meaningful; too large and irrelevant text crowds the window.

Splitting on semantic boundaries — sections, paragraphs — generally beats splitting on fixed token counts, and overlapping chunks reduce the chance of cutting a relevant passage in half.

Embedding. Each chunk is converted to a vector representing its meaning, so that semantically similar text has nearby vectors.

Indexing. Vectors are stored in a structure supporting fast nearest-neighbour search.

Retrieval. The query is embedded and the closest chunks are returned.

Generation. The retrieved chunks are supplied with the question and the model answers from them.

Where it goes wrong

Almost always in retrieval rather than generation.

Semantic search misses exact matches. Embeddings capture meaning and are poor at precise identifiers — part numbers, names, error codes.

The fix is hybrid retrieval: combine vector search with traditional keyword search, which handles exact terms well, and merge the results.

This single change fixes a large share of retrieval failures and is frequently skipped.

The query is not the document. A user asks a short question; the answer lives in a passage phrased entirely differently.

Query rewriting — having a model expand or rephrase the question before retrieval — helps substantially.

Ranking is not relevance. Nearest-neighbour search returns the closest vectors, which is not the same as the most useful passages.

A reranking model that scores query-passage pairs directly, applied to a larger candidate set, improves precision considerably at modest cost.

Retrieval returns nothing useful and the system answers anyway. The most damaging failure.

Systems should detect low retrieval confidence and say they do not know, which requires an explicit threshold and a prompt that permits refusal.

Long context is not a replacement

A common assumption as context windows have grown.

Supplying an entire corpus rather than retrieving from it fails for several reasons: cost scales with tokens, latency rises, and attention over very long contexts is uneven — retrieval accuracy within a long context degrades in measurable and position-dependent ways.

Long context and retrieval are complementary. Retrieve well, then supply generously within the window.

Evaluation

The part most implementations skip and the part that determines quality.

Evaluate retrieval separately from generation. Retrieval is measured by whether the correct passage was returned, which is a conventional information retrieval problem with conventional metrics.

Generation is measured by whether the answer is supported by the retrieved text — faithfulness — which is separate from whether it is correct.

An answer can be faithful to a retrieved passage that was itself wrong, which is why source quality matters as much as pipeline quality.

Build a set of representative questions with known correct sources, and measure both stages against it. A few hundred examples is enough to find most problems.