Retrieval-Augmented Generation: Confident, Well Cited, and Wrong

Contents10 sections

The build that passed on one file

Evaluating graphify for a code index, we measured the full build and the incremental merge on the same repository. A code graph maps a codebase as nodes and edges: files, functions, calls. The full build over a mid sized Python tree finished in tens of seconds. Nobody would have minded running that on every change. The incremental path was faster. The first test of it was a one file change. Both graphs matched.

On a three file change the incremental merge came back missing cross file edges that the full build recovered.

The case for the incremental path was built on that one file sample, and it was built too early. That was an error in reading the data, and it was the ordinary kind: a one file change exercises no cross file merge at all, so the sample that looked like a pass could not have produced any other result no matter how broken the merge underneath it was.

An index has two paths.

One is fast and one is correct. Benchmark the fast one and you ship a graph quietly missing the edges your questions depend on; every answer it gives will look exactly like the answers it gave while it was still complete.

What retrieval changes about an answer

Retrieval augmented generation, RAG from here on, is one sentence long. Before the model answers, fetch the material the answer should rest on. Put it in front of the model. Embeddings, rerankers and chunk sizes all sit underneath that sentence as implementation. It matters at product level because it changes what an answer is. A model answering from its weights is fluent whether or not it is true, and a reader cannot tell those two apart. A model answering from retrieved material produces prose attached to passages that somebody can open, so a reader gets something to check.

Four things follow, and each one is a product decision before it is a tuning decision. Freshness arrives without retraining: a policy that changed this morning is in this afternoon's answer, because the answer reads the document instead of remembering it.

Provenance becomes possible, and it is often the whole reason a regulated or clinical project is allowed to exist at all. The design consequence is a field on the record rather than a formatting step at the end. In our own resident the confidence pipeline attaches source, score and classification to every retrieved fact, and the chat surface renders its sources disclosure straight off those fields, which means a reader who wants to check a claim is following a stored identifier rather than a sentence the model wrote about where the claim came from.

Access control holds at retrieval time. That one is structural: a model trained on everything cannot be untrained for one reader. Being specific also gets cheaper, and the saving is the kind a budget notices. Fine tuning teaches style; retrieval supplies facts. Reaching for the first when you needed the second is the most expensive common mistake in this work, and it costs twice, once in the training run and again in the months before anybody notices that the facts still are not there.

What retrieval costs is that the failure mode moves rather than disappearing. In place of a model that invents an answer, you have a system that can hand the model the wrong three paragraphs. What comes back is confident, well cited and wrong. You have traded a language problem for a search problem. That is a good trade.

It is also a conditional one. Search is measurable and hallucination is not, and the trade pays only if somebody actually measures.

One product question decides most of what follows. Can you write down thirty real questions and, for each of them, the passage that ought to come back? Where that list exists, every later decision is a finding. Where it does not, every later decision is a preference.

Six substrates, six questions

“Which vector database should we use” presumes the answer is a vector database. A retrieval store is a bet about the questions your users ask, and six shapes recur often enough to have stores built for them.

Table 1. What each substrate answers, and the signal that points to it.

Our own system runs three of them side by side: a relational spine, a vector index beside it, and a code graph. How many facts we hold about a subject, what a passage is about, and what calls a function are three different questions. One index answers one of those three well, and the substrate follows from which question is being asked.

Where each one fails

Each substrate's weakness is what the next one is for.

Vector search fails on precision, and it fails quietly. A rare identifier or a negation gets averaged into its neighbourhood and disappears. That is the exact case where the rare term weighting in BM25 is strongest. Lexical search fails the other way round, on vocabulary mismatch, where the user and the document name the same thing in different words; embeddings exist for that case. Neither one computes anything. If the answer is a count or a date range it has to come out of a query. Text to SQL over a small curated schema is the retrieval you actually needed.

None of them traverse. Chunk retrieval can surface two facts that bear on each other and still leave the model guessing at the relation. A graph is the store that states it.

Code graphs, once the numbers are in

A commit pinned graph turns questions that vector search answers badly into questions answered exactly. Blast radius, covering tests and the dependency slice for a change are all path questions, and a path is what a graph returns. Building one over a mid sized Python tree costs tens of seconds, which is less than the test run it protects. The caution is the one from the opening. Build time decides nothing here. The incremental merge was faster and wrong in a way a one file sample could not show, so a fast path earns its place only when it has been measured against the correct path on a change big enough to exercise the merge that separates them.

Composition, and over-fetching on purpose

Single store retrieval plateaus for a structural reason: each store's failure mode is another store's strength. The most reliably reproduced result in the field is also the least exciting. Combining dense and lexical retrieval beats either one alone, because the identifiers that embeddings blur are the terms BM25 weights highest. Over fetching is the cheap move. Take fifty candidates cheaply and let a cross encoder choose the five that reach the prompt, which is a ten to one cut made by a model that reads query and passage together instead of comparing two vectors that were computed without ever having seen each other. That step usually beats changing embedding model.

Fuse by rank instead of by score. A cosine of 0.83 and a BM25 score of 14.2 are not comparable quantities; no hand tuned constant keeps them comparable past a release or two. Reciprocal rank fusion sidesteps it by using positions in each list. Chunking is a retrieval decision rather than preprocessing. Split on structure instead of token count, and never discard the source ID: provenance lost at packing time cannot be recovered at answering time.

Staleness deserves to be a first class answer. An index caches a corpus that has moved on. Every index does. Ours is commit pinned. Its loader reports that it was built at one commit while the tree sits at another, rather than answering off it. Against a stale index, a confident answer is indistinguishable from a correct one.

Which half is wrong

Diagnose the layer before tuning it. A wrong answer has two causes. They take opposite fixes.

Either the right passage never came back, which is a retrieval fault. Measure recall@k against the labelled set built from those thirty questions, and work on the retriever. Or the passage came back and the answer contradicts it, which is a generation fault. Measure faithfulness, meaning whether every claim in the answer is supported by the passages that reached the prompt, and work on the prompt. A retrieval fault answered with prompt engineering is a week spent on the wrong layer, and it is the commonest way a RAG project stalls, because the two faults read identically from the answer alone unless somebody has written down which passages actually reached the prompt.

One line of logging separates them for good.

Two signals that point past the store

Under about ten thousand documents, start with BM25 and a reranker, and add dense retrieval when you can show a query the pair fails on. That is one index and one model. A dense stack asks for three moving parts on day one, and the third of them earns its place when a query forces it.

If answers require joining facts that live in different documents, you needed a graph. The relation was never in the text, so no amount of embedding tuning reaches it.

The question to ask first

Ask what a correct retrieval looks like here, before any store gets chosen. Where the answer comes back crisp, this row or this path or these callers, a specialised store wins. Where it stays vague, dense retrieval is the honest default. The three file run cost a few seconds more than the one file run. Every edge the merge had dropped was inside that difference, and an index missing them answers in the same voice as an index holding them all.

Appendix, glossary