RAG Retrieval: Chunking, Embeddings, and Why It Misses

Retrieval-augmented generation has a simple promise: put the relevant paragraphs in the prompt and the model answers from them instead of from memory. When it fails, the answer is usually confident and wrong, and the instinct is to blame the model.

It's almost never the model. The relevant paragraph was never retrieved.

The pipeline, and where it breaks

Four stages, each with its own failure mode:

1. Chunk the documents into passages.
2. Embed each chunk into a vector, once, offline.
3. Retrieve the top-k chunks most similar to the embedded query.
4. Generate an answer from those chunks.

Stage 4 gets the attention. Stages 1 and 3 cause the failures.

Similarity is not relevance

An embedding measures how alike two pieces of text are in meaning. That is correlated with relevance, and it is not the same thing — which is the whole problem.

Ask "how long does a refund take?" of a document whose refund-policy section repeats the words "refund" and "policy" constantly, and the policy-heavy chunks win on similarity even when the actual answer — a sentence about payment processing times — lives elsewhere and shares fewer words with the question.

The retriever has no idea what an answer is. It ranks text that sounds like the question, and a question rarely sounds like its own answer.

This is also why pure vector search struggles with exact terms: an error code, a product SKU, a person's name. Embeddings smooth over exactly the detail you needed to match. Hybrid search — combining vector similarity with keyword scoring (BM25) — fixes a surprising share of "why didn't it find that?" complaints, because the two methods fail on different things.

Chunking decides what can be found

A chunk is the unit of retrieval. If the answer spans two chunks, no retrieval strategy will return it intact.

Two techniques earn their keep almost everywhere:

Overlap. Have each chunk repeat the last ~10–20% of the previous one, so a fact sitting on a boundary appears whole in at least one chunk. Cheap insurance against the worst chunking failure.

Context headers. Prepend the document title and section path to each chunk before embedding it. A chunk that reads "…must be approved by a manager" is nearly unretrievable on its own; the same chunk prefixed with "Expenses › International travel › Approvals" is both retrievable and interpretable once it lands in the prompt.

Top-k is a trade, not a setting

A small k keeps the prompt tight and misses things. A large k rarely misses and fills the context with near-relevant text that dilutes the prompt, costs tokens on every call, and — past a point — makes answers worse, because the model has more plausible-looking material to be distracted by.

The better shape is two stages:

Retrieve wide, then rerank. Pull 20–50 candidates by vector or hybrid search, then score each candidate against the query with a cross-encoder or a small model that reads both together. Reranking is more accurate than embedding similarity precisely because it looks at the pair rather than at two independently-computed vectors — and because it only runs on a shortlist, it stays affordable. With a reranker in place, k can drop back to 3–5 and quality goes up.

An end-to-end build in Postgres

You don't need a dedicated vector database to start. Postgres with the pgvector extension holds the chunks, the vectors and a full-text index side by side, which makes hybrid search one query:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id          bigserial PRIMARY KEY,
  doc_id      text NOT NULL,
  heading     text NOT NULL,                 -- "Expenses › Travel › Approvals"
  body        text NOT NULL,
  product     text,                          -- metadata for filtering
  updated_at  timestamptz NOT NULL,
  embedding   vector(1024) NOT NULL,         -- dimension = your embedding model's
  tsv         tsvector GENERATED ALWAYS AS (to_tsvector('english', heading || ' ' || body)) STORED
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);

Ingestion splits each document, prefixes the heading path, embeds, and upserts:

for (const doc of docs) {
  const chunks = splitByHeadings(doc, { maxTokens: 400, overlapTokens: 60 })
  const vectors = await embed(chunks.map((c) => `${c.heading}\n\n${c.body}`))
  await db.tx(async (t) => {
    await t.none('DELETE FROM chunks WHERE doc_id = $1', [doc.id])  // re-index whole doc
    for (const [i, c] of chunks.entries()) {
      await t.none(
        `INSERT INTO chunks (doc_id, heading, body, product, updated_at, embedding)
         VALUES ($1, $2, $3, $4, $5, $6)`,
        [doc.id, c.heading, c.body, doc.product, doc.updatedAt, toSql(vectors[i])])
    }
  })
}

Retrieval runs both searches and merges them with reciprocal rank fusion — each result scores 1 / (60 + rank) in each list, and the scores are summed. It needs no tuning of how to weigh a cosine distance against a BM25 score, because it only uses ranks:

WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS r
  FROM chunks WHERE product = $3
  ORDER BY embedding <=> $1 LIMIT 50
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank(tsv, q) DESC) AS r
  FROM chunks, websearch_to_tsquery('english', $2) q
  WHERE tsv @@ q AND product = $3
  ORDER BY ts_rank(tsv, q) DESC LIMIT 50
)
SELECT c.id, c.heading, c.body,
       coalesce(1.0 / (60 + s.r), 0) + coalesce(1.0 / (60 + k.r), 0) AS score
FROM chunks c
LEFT JOIN semantic s ON s.id = c.id
LEFT JOIN keyword  k ON k.id = c.id
WHERE s.id IS NOT NULL OR k.id IS NOT NULL
ORDER BY score DESC
LIMIT 30;

Those 30 go to the reranker; the top 5 go into the prompt. The details of fusion are in hybrid search and rank fusion, and reranking in reranking with cross-encoders.

How big should chunks be?

There's no universal number, but there's a sensible starting range: roughly 200–500 tokens for prose, with 10–20% overlap. Smaller chunks retrieve more precisely and lose context; larger ones carry context and dilute the embedding with unrelated sentences. Tune with the eval below, not by intuition — and remember that the structural boundaries of your documents (headings, list items, functions) usually matter more than the exact token count.

The generation prompt

Retrieval decides what the model sees; the prompt decides how it uses it. Three instructions do most of the work:

Answer the question using only the sources below.
Cite the source id in square brackets after each claim, like [3].
If the sources don't contain the answer, say you don't know — don't guess.

<sources>
<source id="1" title="Refunds › Processing times">…</source>
<source id="2" title="Refunds › Eligibility">…</source>
</sources>

Question: How long does a refund take?

"Only the sources" reduces answers from memory; citations make every claim checkable — and make retrieval failures visible, because an answer that can't cite anything is a retrieval miss you can count. "Say you don't know" gives the model a sanctioned alternative to inventing an answer, one of the main defences against hallucination. Validate the citations in code: an id that wasn't in the sources is a red flag worth logging.

Keeping the index fresh

A RAG system is a cache of your documents, and it has the usual cache problem: staleness.

• Re-index on change, not on a schedule, when the source can tell you what changed — a CMS webhook, a git push, a database change event.
• Delete what was deleted. A retired policy that's still in the index is worse than a missing one, because the model will cite it with confidence. Re-index per document, replacing all its chunks, as above.
• Store updated_at and filter or boost by it, so newer versions of a document beat older ones when both exist.
• Re-embed everything when you change embedding models. Vectors from two models live in different spaces and can't be compared. Build the new index alongside the old one and switch over.

Evaluate retrieval separately

This is the discipline that separates RAG systems that improve from ones that get endlessly re-prompted. Build a small set of question → correct-chunk pairs, a hundred is plenty, and measure retrieval on its own:

• Recall@k — is the right chunk in the top k at all? If this is low, nothing downstream can save you.
• MRR / precision@k — how high does it rank?

Computing both takes a few lines once the labelled pairs exist:

let hits = 0, reciprocal = 0
for (const { question, relevantIds } of evalSet) {
  const ids = (await retrieve(question, { k: 10 })).map((c) => c.id)
  const rank = ids.findIndex((id) => relevantIds.includes(id))
  if (rank !== -1) { hits++; reciprocal += 1 / (rank + 1) }
}
console.log({ recallAt10: hits / evalSet.length, mrr: reciprocal / evalSet.length })

Run it on every change to chunking, embeddings, filters or fusion. A chunking change that looks harmless and drops recall from 0.92 to 0.81 is exactly the regression you want to see before users do.

Then, separately, measure whether the model answers correctly given the right chunks. Two numbers, two different fixes. Without the split, every failure looks like a prompting problem and you tune the wrong stage for weeks.

Things that fix more than prompt tweaking

• Hybrid search. Vectors plus BM25. Covers semantic and exact matching.
• Reranking. The single biggest quality jump per unit of effort.
• Query rewriting. Expand the user's question, or generate two or three variants, and retrieve for each. Handles the case where the question is short and the document is verbose.
• Metadata filters. Filter by product, version, or date before ranking. Most "it cited the old docs" problems are a missing filter, not a ranking problem.
• Citations in the output. Ask for the chunk id alongside each claim. It makes retrieval failures visible instead of silent, and it lets users check.

Debugging one bad answer

When a user reports a wrong answer, work backwards through the pipeline instead of starting with the prompt:

1. Find the right chunk by hand. Search the corpus for the sentence that answers the question. If it isn't there — the document was never ingested, or is outdated — no retrieval tuning will help.
2. Was it retrieved? Log the candidate list for the query. If the right chunk is missing from the top 50, it's a recall problem: try the keyword side (an exact term the embedding missed?), the chunk boundaries (is the answer split across two chunks?), or a metadata filter that excluded it.
3. Was it ranked high enough? If it's in the top 50 but not the top 5 after reranking, look at what outranked it — usually chunks that repeat the question's words without answering it.
4. Did the model use it? If the right chunk was in the prompt and the answer is still wrong, now it's a generation problem — conflicting sources, a confusing chunk, or a prompt that doesn't insist on grounding.

Add the case to the eval set either way. Each step points at a different fix, and skipping straight to step 4 is how teams spend weeks rewording a prompt to work around a chunking bug.

When RAG is the wrong tool

RAG answers "what does the corpus say about X". It is a poor fit for:

• Aggregation. "How many tickets mention crashes?" needs a query, not a similarity search over ten retrieved tickets.
• Whole-document reasoning. "Summarise this contract" wants the whole contract in context — modern windows are large enough that chunking it actively hurts.
• Freshness-critical lookups. If the answer is a row in a database, give the model a tool that queries the database. A tool call returns the current value; a vector index returns whatever was true when you last reindexed.

RAG is a retrieval system with a language model attached. Most of the engineering — and nearly all of the failures — are on the retrieval side.