Every RAG demo works. You hand it a clean PDF, it answers, the room nods. Then you point the same system at the client's real corpus, a decade of scanned contracts, multi-column reports, spreadsheets someone exported to PDF in 2014, and it starts returning confident nonsense. The retrieval did not change. What you fed it did. The hybrid search this series opened with is only ever as good as the chunks underneath it, and turning real documents into good chunks is the half of RAG that nobody puts in the demo.
Here is the rule the whole article hangs on: the chunk is the unit of retrieval. Your reranker, your dense-plus-sparse fusion, your agent, none of them ever see the document. They see chunks. A chunk that was mangled on the way in, a table row that lost its header, a paragraph that lost its heading, is retrievable by no one and useless to the model if it somehow surfaces. The work that decides retrieval quality happens before retrieval ever runs. Three moves carry it.
Move one: parse by the document, not by the hype
The first instinct in 2026 is to send every page to a vision model and let it read the layout like a person. It works, and on a large corpus it will quietly cost you a fortune. The same ten thousand pages can be basically free or several hundred dollars depending only on which path you send them down, so the move is to route by what the document actually is.
A clean digital PDF already contains its text. A library like PyMuPDF pulls it out in milliseconds for nothing, and a vision model would add latency and cost to recover text that was already sitting there. Scans and photographs have no text layer, so those need OCR, and modern OCR is cheap enough that it is rarely the expensive part. The genuinely hard documents, dense tables, multi-column reports, forms, are where the layout-aware and vision parsers earn their price. Reserve them for the pages that need them instead of paying that rate for every page.
The discipline is boring and it is the whole game. Detect the document type once, cheaply, then route: native text for the digital ones, OCR for the scanned ones, a layout-aware parser only for the few that defeat both. Most corpora are mostly easy pages with a hard minority, and paying the hard-page rate across every page is the most common way to set money on fire in an ingestion pipeline.
Move two: chunk along the structure, or orphan the meaning
Once you have the text, the temptation is to split it every five hundred tokens and move on. Fixed-size chunking is fast to write and it is where retrieval quality goes to die, because a document has structure and a token counter cannot see it. Cut every five hundred tokens and you will eventually cut a table in half, and the second half lands in the index looking like this: 4,200, 2026-03, approved. No header, no label, no idea what those numbers are. That chunk is dead. No query matches it on meaning, and if it ever surfaces the model cannot use it.
Structure-preserving chunking splits along the document's own boundaries instead, its sections, headings, tables, and lists. When a table is too big for one chunk you split it by rows and repeat the header in every piece, so no fragment is ever headerless. When you lift a paragraph out of a ninety-page report you prepend the heading it came from, so the chunk still knows where it lives. The point is the same both times: a chunk has to carry enough of its own context to be found on meaning and to be usable once it is found. That is the same instinct behind two techniques worth knowing once the basics are solid. Contextual retrieval, which Anthropic put a name to, prepends a short model-written line to each chunk that situates it in the whole document before embedding, and in their numbers it cut failed retrievals by up to about two-thirds. Late chunking flips the order: embed the whole document first with a long-context model, then split, so each chunk's vector already carries the surrounding text instead of losing it at the cut. Neither one replaces keeping the table intact; they sit on top of it. Structure first so nothing is orphaned, then context so nothing is stranded, and retrieval accuracy moves more than almost anything you can bolt on downstream.
Move three: re-index the delta, not the corpus
The third move is about time, not quality. A real knowledge base changes. Contracts get amended, reports get reissued, someone fixes a typo in a policy and expects search to know by lunch. The naive answer is to re-embed the whole corpus on every change, which is fine at a thousand documents and unbearable at a million. The production answer is to touch only what moved.
Two cheap mechanics make that work. Give every chunk a deterministic ID, so re-ingesting the same source overwrites the old version in place instead of piling a duplicate next to it. And store a content hash on each chunk, so you can tell, without embedding anything, whether it actually changed. If the hash matches, skip it. If it does not, re-embed just that chunk and upsert it. Plenty of vector stores have no true in-place update, so design for delete-then-add from the start.
# One deterministic ID per source chunk.
# Re-ingesting the same document upserts in place; it never duplicates.
chunk_id = uuid5(NAMESPACE, f"{doc_id}:{section}:{index}")
# Hash the source text. If nothing changed, skip the embed entirely.
digest = sha256(text.encode()).hexdigest()
stored = store.get(chunk_id)
if stored and stored.payload["digest"] == digest:
continue # unchanged, no work
# New or changed: embed once, then upsert by the stable ID.
# Some stores have no in-place update, so delete-then-add.
vector = embed(text)
store.upsert(
id=chunk_id,
vector=vector,
payload={"digest": digest, "doc_id": doc_id, "section": section},
)Our own matcher already runs the first half of this. Every point's ID is a uuid5 derived from stable source fields, so re-ingesting a profile updates it in place and never leaves a ghost copy behind. It is a small detail you only appreciate the first time a corpus would otherwise have doubled itself overnight.
There is one change that still makes you touch every vector: swapping the embedding model. A new model is a new vector space, and old vectors are not comparable to new ones, so the whole corpus has to be re-embedded. What changed in 2026 is that this no longer means downtime. You can train a small linear map between the old and new spaces from a sample of the corpus, serve live queries through it at close to full recall, and re-embed everything in the background, retiring the map once the new vectors land. So the rule is softer than it used to be: incremental for content, a background migration for a model change, and a full stop-the-world rebuild almost never. Budget for the migration and it does not surprise you.
The half the whole series stands on
Step back and the four pieces of this series sit on top of this one. The flagship was retrieval, choosing the database was where the chunks live, evaluation was how you prove retrieval improved, and the agent was what calls retrieval as a tool. All four assume the chunks are good. Ingestion is the floor they stand on, and a cracked floor stays invisible until everything above it tilts.
What 2muchcoffee covers
We build production RAG, and in practice the ingestion pipeline is where most of the real engineering hides. Retrieval is a solved shape by now; the document grinder in front of it is where a project is won or lost, because every client's documents are messy in their own specific way. If you are looking at a corpus of scans and exports and wondering why your clean-PDF prototype fell apart on it, that is the conversation we have early. The plain way in is the AI work we do.
One concrete action
Take your worst document, the scanned one, or the one with the giant nested table, and run it through your current pipeline. Do not look at the answer. Look at the chunks it produced. If a table row lost its header, or a number lost its label, or a heading got stranded on its own, you just found your retrieval ceiling, and you found it before spending a dollar on a better model. Fix the chunk, and the rest of the stack finally has something to work with.
