What actually breaks in a RAG system

Retrieval-augmented generation is the most-explained architecture in software right now, and the explanations all stop at the same place. Chunk your documents, embed them, store the vectors, retrieve the top k, put them in the prompt. Every tutorial ends there, because that part works on the first afternoon.

Then you put it in front of people and discover that the retrieval was never the hard part.

Not sure what RAG is yet? Read how do you stop an AI from making things up? first — it explains the whole idea with no jargon, and everything below assumes it.

What follows is the set of failures I think are structural — the ones that show up in any serious RAG system regardless of stack — with the fixes. I'll use one system as a running example throughout, an assistant I built that answers as me on my own site, because abstract advice about RAG is worth very little and a concrete number is worth a lot. Where you see specifics, read them as one instance of a general shape.

---

0. The naive pipeline, and why it survives so long

question → embed → top-k by cosine → stuff into prompt → answer

This is a good design. It is also a design with no opinion about anything: what happens when nothing is relevant, what happens when the user says "yes", what happens when one document is a hundred times larger than the others, what happens when a retrieved passage contains something the user is not allowed to see.

It survives because those cases are invisible in a demo. You ask it questions that have answers, and it answers them. The failures need real users, a real corpus, and a few weeks.

---

1. Grounding is a property of the system, not of the prompt

The first instinct is to write a very firm prompt. Only answer from the provided context. Never invent facts. If the context does not contain the answer, say so.

This works most of the time, which is the problem — it makes the failures rare enough to be surprising and frequent enough to matter.

Here is the shape of it. A document gets filed as readable-by-everyone when it should not have been — an internal list, a set of notes, an old export nobody reclassified. Someone asks a question that document happens to answer. The model does exactly what it was told and answers from the retrieved passage, and out comes something that was never meant to leave the building.

The prompt said never disclose that. The same prompt also said, at length and with feeling, answer only from the source material in front of you. When the source material is right there and says it plainly, the instruction not to repeat it is a request competing with another request. It loses.

So the rule moves out of the prompt and into the pipeline:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

The general principle: anything you ask the model not to do while handing it the material and telling it to be helpful is a request. If it matters, it happens before the model sees the context. In a RAG system the enforcement point is the retrieval boundary, and there are usually four things that belong there:

• visibility — who may see this document
• use — may this be quoted as evidence, or is it only style/tone material
• trust — which sources this particular caller may draw on
• redaction — passages that are retrievable but must be dropped for this asker

The prompt should still state the rule. Two weak guarantees are better than one. But the prompt is the second layer, never the first.

A corollary that took me longer to see: assert the gate twice. My retrieval checks visibility on the chunk and on its parent document, because the chunk's copy is denormalised for speed and a denormalised copy can go stale. It costs nothing and it means a bad migration cannot open a door.

---

2. Pure vector search misses things that are obviously there

Embeddings capture topical similarity. A great deal of what people actually ask about is identity: a product name, a company, a repository, a date, a code.

A worked example. I had a small C library in the corpus, correctly chunked and embedded. The question "have you written anything in C?" returned my essays.

Why: "written" sits much closer to prose than to source code in embedding space, and the line that should have matched was Language: C, which resembles nothing conversational. The retrieval was working exactly as designed and the design was wrong.

The fix is hybrid retrieval — run a lexical arm alongside the vector arm and fuse the rankings:

WITH semantic AS (
  SELECT c."id",
         c."embedding" <=> $vector AS distance,
         row_number() OVER (ORDER BY c."embedding" <=> $vector) AS rank
  FROM chunks c JOIN items i ON i.id = c."itemId"
  WHERE <visibility gate> AND c."embedding" <=> $vector < 0.85
  ORDER BY 2 LIMIT 40
),
keyword AS (
  SELECT c."id",
         row_number() OVER (
           ORDER BY ts_rank_cd(to_tsvector('english', c."content"), q.query) DESC
         ) AS rank
  FROM chunks c JOIN items i ON i.id = c."itemId"
  CROSS JOIN plainto_tsquery('english', $text) AS q(query)
  WHERE <visibility gate> AND to_tsvector('english', c."content") @@ q.query
  ORDER BY 2 LIMIT 40
)
SELECT ... FROM semantic FULL OUTER JOIN keyword USING (id)

Reciprocal rank fusion is the standard way to combine them and it is three lines:

function fuse(row) {
  let score = 0;
  if (row.semanticRank !== null) score += 1 / (K + row.semanticRank);
  if (row.keywordRank  !== null) score += 1 / (K + row.keywordRank);
  return score * (row.confidence || 1);
}

The property worth understanding is why the lexical arm is valuable, and it is not "keywords are also good". Postgres's plainto_tsquery requires every term to be present. That makes it nearly silent on questions your corpus has no words for, and loud on exact identifiers. Which means a lexical hit is strong evidence in its own right — strong enough to accept even when the vector distance is poor:

const hasEvidence =
  (closest !== null && closest <= EVIDENCE_MAX_DISTANCE) ||
  hits.some((hit) => hit.keywordRank !== null);

The general principle: your two retrieval arms are most valuable exactly when they disagree. If you fuse them and then only look at the top of the fused list, you have thrown away the disagreement, which was the information.

---

3. Corpus imbalance quietly destroys relevance

This is the failure I have seen catch the most people and it gets discussed the least.

Top-k retrieval implicitly assumes your documents are comparable in size. They are not. Somebody uploads a 200-page handbook and it becomes 340 chunks in a corpus of 450. Now every broad question returns eight paragraphs of that handbook, because it has eight paragraphs near everything.

The symptom is confusing, because retrieval looks healthy — good scores, relevant enough passages — and the answers get vaguer. In my case a relevance judge correctly reported that the passages did not answer the question, and the assistant started declining questions it could easily have answered from three other documents that never made it into the top eight.

The fix is a per-document cap:

const MAX_PER_ITEM = 3;

function diversify(sorted, limit) {
  const perItem = new Map();
  const chosen = [];
  const overflow = [];

  for (const hit of sorted) {
    const used = perItem.get(hit.itemId) ?? 0;
    if (used < MAX_PER_ITEM) {
      perItem.set(hit.itemId, used + 1);
      chosen.push(hit);
      if (chosen.length === limit) return chosen;
    } else {
      overflow.push(hit);
    }
  }

  return [...chosen, ...overflow].slice(0, limit);
}

The last line is the whole design. Overflow is appended, not discarded — a question that genuinely is answered by one document should still get a full result set rather than three passages and silence. Which means this is not a hard cap at all. It is a no-crowding-out rule.

I got that wrong in my own test first: I asserted "no document supplies more than three of eight results", it failed, and I went hunting for a bug in diversify. The bug was in the assertion. The test now states what the code actually guarantees — everything else relevant gets in before a fourth passage of the big document — which is both true and the thing I actually care about.

The general principle: any ranking over a heterogeneous corpus needs a diversity constraint, and the constraint should degrade to "return what you have" rather than to "return less".

---

4. If you add a relevance judge, it will fail silently

A distance threshold separates answerable from unanswerable questions cleanly when your corpus is small. Measure it again once the corpus grows and the distributions overlap: I have an answerable question at 0.7755 and an unanswerable one at 0.7553. No threshold fixes that, however carefully tuned.

So you add a second model call that reads the passages and says whether they address the question. Good idea. Two things will then go wrong, and both are about the judge, not the retrieval.

It will read truncated passages

I passed the judge the first 400 characters of each passage to save tokens.

My date of birth sits at character 722 of its chunk. My favourite food at 403.

Retrieval had found exactly the right chunk, first, on both counts. The judge was handed a passage with the answer sliced off and truthfully reported that the answer was not there. Chunks are already capped by the chunker, so passing them whole is bounded — a few thousand tokens against a wrong refusal is not a difficult trade.

There is a second-order version of this. A dense document packs many facts into one chunk, which dilutes its embedding. The passage holding my date of birth sat at cosine distance 0.8232 — past my evidence ceiling — while ranking first on the lexical arm, because "date of birth" appears in it verbatim. Fusion put four vector-close but useless chunks above it, and the judge only saw those. So the judge now always sees the best lexical hits as well as the top of the fused list:

function judgeSet(hits) {
  const chosen = hits.slice(0, 4);
  const seen = new Set(chosen.map((h) => h.chunkId));

  const byKeyword = hits
    .filter((h) => h.keywordRank !== null && !seen.has(h.chunkId))
    .sort((a, b) => (a.keywordRank ?? 99) - (b.keywordRank ?? 99))
    .slice(0, 2);

  return [...chosen, ...byKeyword];
}

It will be disabled by an exception you never see

A judge should fail toward answering — if the check itself breaks, defer to the model, whose grounding rules already say to decline when the material does not cover the question. One weak safeguard failing should not take the other with it.

That reasoning is correct and it is also how the judge died for weeks without a single error in my logs.

row_number() is bigint in Postgres. The driver returns a JavaScript BigInt. My type declared number, so TypeScript never questioned it. The fusion function got away with it because it coerces. The judge sorts by rank, and BigInt arithmetic against a number throws:

TypeError: Cannot convert a BigInt value to a number

Swallowed by the try/catch. Judge skipped. Every ambiguous question went to the model unjudged, silently.

// Converted once, where the values enter the type, rather than at each use —
// the next reader of `keywordRank` should not have to know this.
const hits = rows.map((row) => ({
  ...row,
  distance:     row.distance     === null ? null : Number(row.distance),
  semanticRank: row.semanticRank === null ? null : Number(row.semanticRank),
  keywordRank:  row.keywordRank  === null ? null : Number(row.keywordRank),
  score: fuse(row),
}));

The general principle, and it is the most important one here: a catch on the permissive path hides its own bug forever. If you fail soft, log loud — and put a test on the safeguard, not just on the thing it safeguards.

---

5. A RAG system is not a question-answering system

This is the architectural mistake, and it is invisible until you watch a real conversation.

The naive pipeline has one path: retrieve, and if nothing comes back, refuse. So what happens when the user says "yes"?

"Yes" retrieves nothing. My assistant would offer to pass a question on, the visitor would say "Yes", and the same canned refusal came back. Forever. And the reverse: "hey what's up" retrieves plenty — lots of text is vaguely near a greeting — so a hello came back with four citations stapled to it.

The failure mode generalises badly. Every unrecognised phrasing did not degrade to a worse answer, it degraded to no answer, because retrieval was a precondition for speaking at all.

The fix is to classify the turn before deciding whether to retrieve:

export type TurnKind =
  | "greeting" | "smalltalk" | "handoff-accept"
  | "advisory"    // a general question — judgement, not biography
  | "artefact"    // they want the file, not a fact about it
  | "crude"       // deflected, not answered
  | "role" | "pitch" | "money"  // a hiring conversation
  | "question";   // grounded — answered only from source material

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

0

Your categories will differ. The principle will not:

Grounding protects claims about the domain. It was never meant to be a precondition for speaking.

Only turns that actually make a claim about your corpus should need evidence. Everything else — chat, clarification, acknowledgement, a general question your model can answer perfectly well on its own — should be answered by talking.

There is a related decision worth making explicitly: what happens when a grounded turn finds nothing. Refusing is right when the question is genuinely about your domain and there is nothing behind it. Refusing is wrong when the question merely looked domain-shaped. I ended up with a separate predicate for that, so a question that needs evidence and lacks it declines, and everything else falls through to conversation.

---

6. The context bug will not be in the model

I want to spend a section on this because it is the least glamorous and it cost me the most.

Follow-up questions never worked. Names given a moment earlier were forgotten. Every conversation in my log said turns=1. I spent days on the retrieval, the follow-up rewriter, and the identity flow.

The browser was sending history: [] on every request. I had assembled the history inside a React state updater:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

1

A state updater is not a synchronous read of state. The fix is a ref mirroring the messages:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

2

Every feature that depended on conversation context had been broken since the day it was written, and I had been debugging the symptoms.

The general principle: when something is inexplicable, check the wire before you check the model. One log of the actual request body would have found this in a minute. RAG systems have an unusual amount of plumbing between the user and the model, and plumbing fails in ways that look like intelligence failures.

The related design question — what to carry forward — is genuinely hard, and mostly a cost problem. A visitor pasting a five-thousand-character document should not re-send it on all twenty subsequent turns. I trim history entries from both ends, because the beginning of a pasted document is its title and the end is often the actual question:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

3

And a follow-up has to be resolved into something searchable before it hits retrieval, because "where was that, which company?" has no searchable words at all. Worth knowing: the referent usually lives in the previous answer, not the previous question. My first version expanded against the user's own last message, where the company name never appeared.

One more, since it is the same class: my request validator capped questions at 2,000 characters. Someone pasted a job description and got "Invalid request". Raising the cap is obvious; the part that is not obvious is that you must raise the history cap too, or you have simply moved the failure to turn two.

---

7. Retrieval is not the only thing a grounded assistant needs to do

Two capabilities do not fit the retrieve-and-answer shape at all, and trying to force them produces bad answers rather than missing ones.

Producing an artefact. Asked for my CV, the system retrieved the CV's text, found the text did not answer the question — because the question was not about its contents — and declined. Later, with a friendlier prompt, it did something worse: it pasted several hundred words of CV headings into the chat. A document's contents are not the document. This is a missing capability, not a retrieval gap, so it gets a tool:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

4

Two details generalise. First, narrow the menu before offering it — a model that has decided to attach something will pick the nearest available thing, and asked for a CV with no CV on file it attached a portrait. Second, never name a tool that is not on the table. Told to call show_artefact when nothing was available, it wrote the literal text show_artefact('CV-2026.pdf') into the reply, with an invented filename. An instruction to use an absent capability is not a safeguard, it is a prompt to fake it.

Taking an action. Mine can put a message in my inbox. The security design is one line:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

5

The destination resolves from configuration. Nothing a visitor types can influence it. If you take one thing from this section: for any outbound action, the model supplies the content and never the target.

---

8. Evaluating a RAG system

Most of the consequential logic in a RAG pipeline is small pure functions and regexes that decide something important: what kind of turn this is, what may be retrieved, what may be said, when to decline. They break quietly when you change something adjacent.

I run two tiers, on deliberately separate runners.

Tier 1: pure. No database, no model, ~230 tests, about a second. Every case is a bug that has already happened. It is fast enough to run on every save, which is the entire point — an eval suite that takes four minutes and costs money gets run once a week and catches nothing.

It earns its keep constantly. Adding self-introduction handling, it immediately caught that my new pattern classified "what do you do when a migration fails?" as a request for a career summary. It also caught this, which had been shipping:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

6

The assistant had been addressing people as Working On.

Tier 2: retrieval. Real Postgres, real embedded fixture corpus. The critical design rule is that assertions are about which documents come back, never about wording — so they survive a re-embed, a chunker change, or a new model. Nine cases, each one a regression:

• a question about employers reaches the employment documents
• a language question reaches the repository, not the essays
• no single document supplies more than three of eight results
• a public passage naming a private person is filtered
• a question the corpus cannot answer returns nothing
• a private document is unreachable by every audience
• an internal document is hidden from a visitor and visible to the owner
• a style-only artefact is never returned as evidence
• a contentless follow-up is rewritten into something searchable

And a guard I wrote in blood:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

7

Earlier in the project a cleanup step could not tell a fixture from a real record and deleted two rows that mattered. I recovered most of it from archives; the hand-edited fields were gone for good. Every fixture is now named so implausibly that the two can never be confused in either direction.

---

9. Guardrails collide with your own features

A late lesson, and a funny one.

I have a refusals list — subjects the assistant declines, in my own words. "Day rates and salary" is on it, matched by a cheap keyword pass that runs before any model call.

I then built a feature where visitors paste a job description and get a reasoned pitch back.

Every job description names a salary band as a matter of course. So the document's own boilerplate tripped my guardrail, and the assistant refused the exact turn the feature existed for.

Two things came out of fixing that. The narrow exemption is easy:

// Enforced here rather than asked of the model, because the model is being told
// in the same breath to answer from whatever is in front of it.
const allowed = await redact(found, caller);

8

The interesting one is what happened a turn later. The recruiter asks "what are your salary expectations?" and gets the canned refusal — word for word, immediately after a warm and detailed pitch. Technically correct; reads as a shutter coming down at the exact moment the conversation is most likely to be lost.

So the subject is still declined, but it is declined like a person, with a handoff. And my first attempt at detecting it was wrong in a way worth stating: I keyed it off the guardrail firing, which matches the words on my refusal list. It catches "salary expectations" and sails straight past "the budget is 180k, does that work for you?" — same conversation, and the more expensive one to get wrong.

The general principle: a guardrail keyed to your vocabulary will miss the user's. If a subject matters, detect the subject, not your phrasing of it.

---

10. What this class of system cannot do

Worth being honest about, because most RAG write-ups end on a demo.

A document filed wrongly is a leak you can only partly compensate for. Redaction at the retrieval boundary catches what it can match on — a name, an identifier, a tag. It cannot catch a paragraph that gives the same thing away without ever naming it. Classifying a document correctly is a human judgement, and the whole system rests on it.

Retrieval quality is bounded by how the corpus is written. A fact stated once, in passing, in a document about something else, is hard to retrieve and easy to miss. RAG rewards corpora that were written to be found.

Every safeguard is a cost. The relevance judge is a second model call on the ambiguous band. Diversity constraints mean sometimes returning a less relevant passage. Redaction means occasionally dropping something the asker was entitled to. All of these are the right trade and none of them is free.

Conversation-level correctness is still mostly untested. My two tiers cover rules and retrieval. Nothing yet asserts on the shape of a whole exchange — the tier that would is planned and unwritten, and I suspect that is true of almost everyone shipping these.

---

The short version

If I could send five lines back to myself at the start:

1. Put the safeguard in code, not in the prompt. Anything you ask the model not to do while handing it the material is a request, and requests lose.
2. A catch on the permissive path hides its own bug forever. Fail soft, log loud, and test the safeguard itself.
3. Classify the turn before you retrieve. Grounding protects claims; it was never meant to be a precondition for speaking.
4. Cap how much any one document may contribute, and let the overflow back in rather than returning less.
5. When it is inexplicable, check the wire. The bug is more often in the plumbing than in the model.

And one more that is not about RAG at all: write down why. Six months from now MAX_PER_ITEM = 3 is an arbitrary number somebody will helpfully tune. "One document held 336 of 450 passages" is a reason.

---

If you want the basics rather than the scars, start here. The system used as the example throughout is Ask Kobby, the assistant on this site — Next.js, Prisma, Postgres with pgvector, and the OpenAI Responses API. If you want the specifics rather than the principles, the architecture notes and the code are linked from the repository.