Retrieval-augmented generation demos beautifully and fails quietly. You embed a folder of PDFs, wire up a vector store, and the first ten questions come back plausible. Then someone asks about the cancellation clause in this year's contract and the system quotes last year's with total confidence. We have rebuilt enough of these to state the pattern plainly: the failure is almost never the model. It is retrieval, an engineering problem with measurable answers.
Why naive vector search underperforms
Embedding similarity is not relevance. Three failure modes cover the gap.
Identifiers do not embed. Part numbers, error codes, statute references, surnames. Ask "what does error E-4412 mean" and the query vector lands near every other error paragraph in the corpus: the encoder captures the shape of the sentence and discards the token carrying all the meaning. Dense vectors compress, and identifiers are what compression throws away.
Chunk boundaries destroy answers. A fixed 512-token window splits a table from its header row, or a clause from the definition of the term it uses. The chunk still scores well and contains nothing usable, so the model invents the missing half. That reads as a hallucination but is a retrieval bug.
Similarity has no idea what is current. A superseded policy, a draft, a support ticket quoting the old rule and the live handbook all sit near the query in vector space. Cosine distance will never tell you which is true today.
On one client corpus of roughly 40,000 pages of compliance material, plain top-5 dense retrieval put the correct passage into context for 61% of our test questions. Structure-aware chunking, hybrid retrieval and a cross-encoder reranker took that to 89% with no change to the generation model.
Chunk on structure, not character count
Chunking is where most of the win lives. The rules we apply:
- Split on the document's own boundaries first: headings, articles, clauses, list items, slides, ticket threads. Character-count splitting is the fallback for a section that is genuinely too long.
- Keep tables whole, serialized as Markdown with the header row intact. A table split across two chunks is worse than no table at all.
- Prepend the heading path to every chunk's embedded text. "This must be requested within 30 days." is useless alone; "Refunds > Consumer returns > Timing: This must be requested within 30 days." retrieves and reads correctly.
- Target 300 to 800 tokens. Below that you lose the context the sentence depends on. Above it, one chunk covers several topics and its embedding averages all of them.
- Overlap is a patch, not a strategy. 10 to 15% helps at awkward boundaries. Needing 50% means your splitter is ignoring the document structure.
- Store metadata beside the text and filter on it: source id, heading path, effective date, version, access scope. Half the "wrong answer" tickets we investigate are a scoping problem a filter would have solved.
Hybrid retrieval, then rerank
Run lexical and dense search in parallel and fuse the results. BM25 finds the error code; the dense index finds the paraphrase. Reciprocal rank fusion is our default because it fuses ranks rather than scores, so nothing needs normalizing. Postgres does both sides well enough for most corpora, which saves running a second datastore:
-- pgvector + tsvector, fused with reciprocal rank fusion (k = 60)
with dense as (
select id, row_number() over (order by embedding <=> $1) as rank
from chunks
where tenant_id = $2 and effective_to is null
order by embedding <=> $1
limit 50
),
lexical as (
select id, row_number() over (
order by ts_rank_cd(tsv, websearch_to_tsquery('english', $3)) desc
) as rank
from chunks
where tenant_id = $2
and effective_to is null
and tsv @@ websearch_to_tsquery('english', $3)
limit 50
)
select c.id,
c.heading_path,
c.content,
coalesce(1.0 / (60 + d.rank), 0) + coalesce(1.0 / (60 + l.rank), 0) as score
from chunks c
left join dense d on d.id = c.id
left join lexical l on l.id = c.id
where d.id is not null or l.id is not null
order by score desc
limit 50;
Note the effective_to is null predicate: recency and supersession belong in the query, not in the system prompt.
Then rerank. A cross-encoder scores the query and a candidate together instead of comparing two independent vectors, which is why it is far more accurate and far slower. Take 50 candidates down to 8. On a small GPU a base-size reranker scores 50 pairs in roughly 40ms; on CPU budget 250 to 400ms and cache. That step alone was worth 14 points of recall@5 on the compliance corpus, more than any embedding model swap we tried.
Build the evaluation set before you tune anything
Without a labeled set, every change is judged by typing three questions into a chat window and forming an impression, which cannot detect a regression. Write 150 to 200 questions with a domain expert, each labeled with the chunk ids that genuinely answer it. Include the categories that break systems: exact-identifier lookups, answers spanning two sections, superseded content, near-duplicate topics needing disambiguation, and 20 to 30 questions the corpus cannot answer.
Measure retrieval separately from generation. If recall@10 is 0.6, no amount of prompt engineering will save you.
# eval/harness.py
import json
import statistics
from dataclasses import dataclass
@dataclass
class Case:
qid: str
question: str
gold_chunks: set[str] # empty set means "the system should abstain"
category: str
def recall_at_k(retrieved: list[str], gold: set[str], k: int) -> float:
if not gold:
return 1.0
return len(gold & set(retrieved[:k])) / len(gold)
def reciprocal_rank(retrieved: list[str], gold: set[str]) -> float:
for position, chunk_id in enumerate(retrieved, start=1):
if chunk_id in gold:
return 1.0 / position
return 0.0
def run(cases: list[Case], retrieve, answer) -> dict:
rows = []
for case in cases:
ranked = [chunk.id for chunk in retrieve(case.question)]
result = answer(case.question, ranked[:8])
cited = set(result.citations)
rows.append(
{
"qid": case.qid,
"category": case.category,
"recall@5": recall_at_k(ranked, case.gold_chunks, 5),
"mrr": reciprocal_rank(ranked, case.gold_chunks),
# every cited passage must be one we actually put in context
"citation_valid": float(bool(cited) and cited <= set(ranked[:8])),
"abstained": result.abstained,
"should_abstain": not case.gold_chunks,
}
)
def mean(key: str) -> float:
return round(statistics.fmean(float(row[key]) for row in rows), 3)
unanswerable = [row for row in rows if row["should_abstain"]]
summary = {
"n": len(rows),
"recall@5": mean("recall@5"),
"mrr": mean("mrr"),
"citation_valid": mean("citation_valid"),
"false_answer_rate": round(
sum(1 for row in unanswerable if not row["abstained"])
/ max(1, len(unanswerable)),
3,
),
}
print(json.dumps(summary, indent=2))
return summary
Run it in CI on every change to the chunker, the prompt, the retriever weights or the model version, and fail the build on a regression greater than two points. An afternoon of work, and the difference between tuning and guessing.
Citations have to be verifiable
Ask a model to cite its sources and it will produce citation-shaped text whether or not it used them. Make citations structural: give every retrieved passage an id in the prompt, require the answer as JSON with a citations array of those ids, then validate server-side that each id was in the context you supplied. In the UI, a citation links to the exact passage with the matched text highlighted, never to a 90-page PDF. Once reviewers can click through, they report which passage was wrong, and you get labeled evaluation cases out of real usage.
Saying "I don't know"
Abstention is a feature, and it needs a threshold rather than a polite instruction. We gate on the reranker score of the top passage: below a calibrated cut-off the pipeline never calls the generation model and returns the nearest passages as suggested reading. Calibrate that cut-off against the unanswerable questions in your evaluation set.
Then instruct explicitly: answer only from the supplied passages, and when they do not contain the answer, say so and name what is missing. "The provided policy documents cover EU returns but not US returns" is a useful answer. A confident invention is worse than silence, and in a regulated domain it ends the project.
How we work
We build the evaluation set in week one, with the client's domain expert in the room, before any tuning happens. Retrieval quality is reported as a number on every release, ingestion is a reproducible pipeline rather than a one-off notebook, and abstention thresholds are tuned against real questions. Clients get the harness with the code, so when they add 5,000 documents next year they can tell whether the system got better or worse without asking us.