PDF Chunking for RAG: Route OCR by Page, Keep Table Headers Attached, Re-index the Delta

Every RAG demo works. You hand it a clean PDF, it answers, the room nods. Then you point the same system at the client's real corpus, a decade of scanned contracts, multi-column reports, spreadsheets someone exported to PDF years ago, and it starts returning confident nonsense. The retrieval did not change. What you fed it did. The hybrid search this series opened with is only ever as good as the chunks underneath it, and PDF chunking, the work of turning real documents into good chunks, is the half of RAG that nobody puts in the demo.

Here is the rule the whole article hangs on: the chunk is the unit of retrieval. Your reranker, your dense-plus-sparse fusion, your agent, none of them ever see the document. They see chunks. A chunk that was mangled on the way in, a table row that lost its header, a paragraph that lost its heading, is retrievable by no one and useless to the model if it somehow surfaces. The work that decides retrieval quality happens before retrieval ever runs. Three moves carry it.

The chunk is the unit of retrieval. Nothing downstream ever sees the document.

Chunking in RAG means cutting each document into pieces small enough to embed and retrieve, each carrying enough context to be found by a query and used by the model. PDF chunking is the hard case: a PDF stores drawing instructions for a page, so the parser has to recover paragraphs, headings and tables before any splitter sees them.

Move one: parse by the document, not by the hype

The first instinct now is to send every page to a vision model and let it read the layout like a person. It works, and on a large corpus it will quietly cost you a fortune. The same ten thousand pages can be basically free or several hundred dollars depending only on which path you send them down, so the move is to route by what the document actually is.

Jerry Liu of LlamaIndex has the one-line version of why any of this is hard: PDFs are designed for display, so the file's internal representation is a stream of instructions like "render this string at coordinate (84, 720) with this font", and everything a chunker needs, the paragraph, the heading, the table, has to be inferred back from where the ink landed.

A clean digital PDF at least carries its text in that stream. A library like PyMuPDF pulls it out in milliseconds, and a vision model would add latency and cost to recover text that was already sitting there. Scans and photographs have no text layer, so those need OCR, and modern OCR is cheap enough that it is rarely the expensive part. The genuinely hard documents, dense tables, multi-column reports, forms, are where the layout-aware and vision parsers earn their price. Reserve them for the pages that need them instead of paying that rate for every page.

Measured evidence backs the native-text branch of that routing. El Bachyr and colleagues (ICSE-SEIP 2026, arXiv 2604.12047) ran six parsers over 337 pages of financial filings and scored page-level retrieval for every parser and chunker, so read the figures quoted here as whether the right page came back, never as answer accuracy. Rule-based parsers without OCR held their own: pdfminer led on narrative text with an MRR of 0.646 and pdfplumber led on table pages at 0.807, while the OCR-plus-layout-model Unstructured pipeline came third on both at 0.630 and 0.788.

Their Table 8 is the part that matters for your bill. Unstructured averaged 1.888 seconds a page against 0.080 for pdfminer and 0.004 for PyMuPDF, roughly two orders of magnitude slower than the fastest stream parsers, and it still landed behind the two leaders on retrieval. Send a whole corpus down that path and you pay the slow rate on the easy pages too.

Tool choice beyond that routing is a separate comparison, with licences, benchmarks and GPU bills of its own. The short version for this article is that the fast path costs milliseconds and the slow path is where the money goes, so the discipline is boring and it is the whole game.

Detect the document type once, cheaply, then route: native text for the digital ones, OCR for the scanned ones, a layout-aware parser only for the few that defeat both. Most corpora are mostly easy pages with a hard minority, and paying the hard-page rate across every page is the most common way to set money on fire in an ingestion pipeline.

How to tell a page needs OCR

The detection step is cheap enough to run on every page of every document. Ask PyMuPDF for the page's text layer. A digital page returns its text, usually hundreds of characters, and extraction costs you nothing more. A scanned page returns almost nothing while one image covers most of the page, and only those pages go to the OCR engine. A page that is empty on both counts, a separator sheet or a blank back page, stays out of the index altogether.

import pymupdf  # PyMuPDF


def page_needs_ocr(page, min_chars=50, image_cover=0.6):
    if len(page.get_text("text").strip()) >= min_chars:
        return False  # a usable text layer exists, extract it for free
    covered = sum(abs(pymupdf.Rect(i["bbox"])) for i in page.get_image_info())
    return covered / abs(page.rect) >= image_cover  # one big picture, no words: a scan


doc = pymupdf.open("report.pdf")
scan_pages = [p.number for p in doc if page_needs_ocr(p)]

Two more PDF habits bite once the text is out. Multi-column pages come back in the wrong reading order when a parser walks the page top to bottom, so ask for block-level output and sort the blocks by column before you join the text. Running headers, footers and page numbers repeat on every page, so drop them by their position on the page before chunking, or every chunk inherits the same three lines of noise and the embedding of a pricing table starts to look like the embedding of the cover letter.

Move two: chunk along the structure, or orphan the meaning

Once you have the text, the temptation is to split it every five hundred tokens and move on. Fixed-size chunking is fast to write, and on page-level retrieval it scores better than its reputation.

In the El Bachyr study, plain token windows reached an MRR of 0.647 on narrative pages and 0.806 on table pages, close to the best chunker's 0.658 and 0.833, while the two semantic chunkers, which the paper calls structure-aware, came last (0.606 and 0.795 for semantic, its double-pass variant a shade lower).

Their six chunkers are all text splitters, so the heading- and table-aware chunking below sits outside what the study measured. All of those scores are about finding the right page, and they say nothing about what the model can do with the chunk once it arrives.

That second half is where a token counter hurts you, because a document has structure and the counter cannot see it. Cut every five hundred tokens and you will eventually cut a table in half, and the second half lands in the index looking like this: 4,200, 2026-03, approved. No header, no label, no idea what those numbers are. That chunk is dead for the model even when its page is retrieved: the query lands, the row arrives, and the answer still comes out wrong.

Structure-preserving chunking splits along the document's own boundaries instead, its sections, headings, tables, and lists. When a table is too big for one chunk you split it by rows and repeat the header in every piece, so no fragment is ever headerless. When you lift a paragraph out of a ninety-page report you prepend the heading it came from, so the chunk still knows where it lives. The point is the same both times: a chunk has to carry enough of its own context to be found on meaning and to be usable once it is found.

Chunk size still matters inside structural splits; treat it as a cap, so a long section gets cut at the cap instead of growing without limit. The same study capped every chunker at 512 tokens and, with its best chunker held fixed, found 25 percent overlap (128 tokens) beat both none and 50 percent on MRR, lifting narrative MRR from 0.529 with no overlap to 0.658, while doubling to 50 percent grew the index and scored slightly lower.

Start at 512 tokens with 25 percent overlap, measure on your own evaluation set, and move either number only when the measurement says so.

Chunking tables for RAG

Tables are the chunk type most parsers mangle and most splitters finish off. Have the parser emit each table as Markdown, one row a line with the header row on top, instead of flattened text, and then split long tables by row count and repeat the header line at the top of every piece.

A chunk that starts with a line reading Vendor, Amount, Month, Status and holds twenty rows is searchable on meaning and readable by the model; the same twenty rows without that first line are the dead chunk from above. Give the table chunk a short caption too, the table's title or the heading above it, so a query about approved vendor payments finds it even when the cells hold only names and numbers.

Six chunking strategies compared on PDFs

The last two rows are the same instinct carried further, two techniques worth knowing once the basics are solid. Contextual retrieval, which Anthropic described in September 2024, prepends a short model-written line to each chunk that situates it in the whole document before embedding.

Their metric was the top-20-chunk retrieval failure rate, defined as 1 minus recall@20: contextual embeddings alone took it from 5.7% to 3.7% (a 35% cut), adding a contextual BM25 index took it to 2.9% (49%), and a reranker on top took it to 1.9% (67%). The figures are Anthropic's own, on their corpus, and the method bills a model call per chunk.

Late chunking, from Günther and colleagues at Jina AI (2024), flips the order: embed the whole document first with a long-context model, then split just before pooling, so each chunk's vector already carries the surrounding text instead of losing it at the cut. Neither one replaces keeping the table intact; they sit on top of it.

Move three: re-index the delta, not the corpus

The third move is about time, not quality. A real knowledge base changes. Contracts get amended, reports get reissued, someone fixes a typo in a policy and expects search to know by lunch. The naive answer is to re-embed the whole corpus on every change, which is fine at a thousand documents and unbearable at a million. The production answer is to touch only what moved.

How much re-indexing work does each structuring approach create once documents start changing? Work it through for one edited page in a two-hundred-page report. With fixed windows, an insert shifts every chunk boundary after it, so every chunk downstream of the edit gets a new hash and re-embeds. Recursive splitting snaps boundaries to paragraph breaks, so churn tends to stay nearer the edit than with fixed windows, though packing pieces up to the cap can still move chunks downstream until the packing realigns.

Structure-preserving chunking contains the change inside its section, so one section re-embeds and the rest of the report is untouched. Semantic chunking moves its cut points with the content, so boundaries can shift well past the edit.

Contextual retrieval adds a second cost, because each chunk's context line is written with the whole document in the prompt, and a material change can make lines across the document stale.

Late chunking is the expensive end: every chunk's vector carries its neighbours, so the whole document re-embeds on any edit, and once the document is longer than the model's window, an edit that changes its length re-embeds everything from the macro chunk holding it to the end of the document, because the macro chunks are fixed token spans. That walk-through is reasoning from how each method draws its boundaries; the content hash in the card below is what turns it into a number for your corpus.

Three cheap mechanics make the delta work. Give every chunk a deterministic ID, so re-ingesting the same source overwrites the old version in place instead of piling a duplicate next to it. Store a content hash on each chunk, so you can tell, without embedding anything, whether it actually changed; if the hash matches, skip it, and if it does not, re-embed just that chunk and upsert it.

Then sweep for ghosts: when a document shrinks, the chunk ids it used to produce stay in the index unless you compare the ids you just wrote against the ids the store holds for that document and delete the difference. Plenty of vector stores have no true in-place update, so design for delete-then-add from the start.

Log, per document, everything ingestion filtered, split off or dropped: the pages routed to OCR, the tables split by rows, the running headers removed, the chunks the hash let you skip. Then write some of your test questions straight from the source files, because a golden set sampled from your own index can only test content that already made it into the index.

Missing content is the first of the seven RAG failure points Barnett et al. catalogued, the answer sitting in none of the documents the system can see, and a passage that ingestion dropped lands there with the source file still on disk. A question written from that file catches the failure, after which the log tells you which ingestion step lost the passage.

import hashlib
import uuid

NAMESPACE = uuid.NAMESPACE_URL  # any fixed UUID; change it and every id changes


def chunk_id(doc_id, section, index):
    # Same source position, same id: re-ingesting a document overwrites in place.
    return str(uuid.uuid5(NAMESPACE, f"{doc_id}:{section}:{index}"))


def reindex_document(store, embed, doc_id, chunks):
    """chunks: (section, text) pairs in document order. Embeds only what changed."""
    counts, live = {}, set()
    for section, text in chunks:
        index = counts.get(section, 0)
        counts[section] = index + 1
        cid = chunk_id(doc_id, section, index)
        live.add(cid)
        digest = hashlib.sha256(text.encode()).hexdigest()
        stored = store.get(cid)
        if stored and stored["payload"]["digest"] == digest:
            continue  # unchanged: skip the embed
        payload = {"digest": digest, "doc_id": doc_id, "section": section}
        store.upsert(cid, embed(text), payload)  # delete-then-add if your store lacks upsert
    stale = store.ids_for_doc(doc_id) - live  # the document shrank: these ids vanished
    if stale:
        store.delete(stale)
    return live, stale

Given a store with get, upsert, ids_for_doc and delete, re-ingesting an unchanged document costs zero embeds, editing one chunk costs one embed, and dropping the last chunk of a section costs one delete; drop an earlier one and the chunks after it in that section re-embed, which is the same containment the structure-preserving row promises. Our own matcher takes a simpler route in public code: each chunk's uuid5 is a hash of its full payload, so re-ingesting an unchanged profile rewrites the same points, and a full reindex rebuilds the collection from its source file.

There is one change that still makes you touch every vector: swapping the embedding model. A new model is a new vector space, and old vectors are not comparable to new ones, so the whole corpus has to be re-embedded. What changed in September 2025 is the price of doing it without downtime. Dual-index serving already kept search up through a swap, at up to double the serving cost while the full re-embed ran.

Drift-Adapter, published at EMNLP 2025, trains a small transformation between the old and new spaces from a sample of paired embeddings, maps new queries into the old space so the existing index keeps serving, and recovered 95 to 99 percent of full re-embedding recall in its tests at over 100 times less recompute, which buys the time to re-embed in the background.

So the rule is softer than it used to be: incremental for content, a background migration for a model change, and a full stop-the-world rebuild is rarely needed. Budget for the migration and it does not surprise you.

The half the whole series stands on

Step back and the four pieces of this series sit on top of this one. The flagship was retrieval, choosing the database was where the chunks live, evaluation was how you prove retrieval improved, and the agent was what calls retrieval as a tool. All four assume the chunks are good. Ingestion is the floor they stand on, and a cracked floor stays invisible until everything above it tilts.

FAQ

Why does RAG fail on real documents when the demo worked? Because the demo used a clean digital PDF and the client's corpus is scans, multi-column reports, and spreadsheets exported to PDF years ago. The retrieval stayed the same while the chunks underneath it changed. A mangled chunk is retrievable by no one and useless to the model if it surfaces.

What is the unit of retrieval in RAG? The chunk. Your reranker, your dense-plus-sparse fusion, and your agent never see the document, only the chunks it became. That is why the work that decides retrieval quality happens during ingestion, before retrieval ever runs, and why a cracked ingestion floor stays invisible until everything above it tilts.

Should every PDF page go to a vision model? Only the hard minority: dense tables, multi-column layouts and forms that defeat both text extraction and OCR. A clean digital PDF already contains its text, and a library like PyMuPDF or pdfplumber pulls it out in milliseconds. Route by document type: native text for digital pages, OCR for scans, layout-aware or vision parsing for the rest. In El Bachyr et al. (ICSE-SEIP 2026) the two best rule-based parsers without OCR beat the OCR-plus-layout pipeline on page-level retrieval, and the faster of them, pdfminer, parsed a page about 24 times faster (0.080 against 1.888 seconds in their Table 8).

What is wrong with fixed-size chunking? A token counter cannot see structure. Cut every five hundred tokens and you eventually cut a table in half, leaving a fragment of numbers with no header and no labels. That chunk is dead for the model even when its page is retrieved: El Bachyr et al. found token windows close to the best chunker on page-level MRR, and page-level MRR never measures whether the model can use what arrived. Split along the document's own boundaries and keep a token cap inside them.

What chunk size and overlap should I use for PDF chunking? Treat size as a cap inside structural splits rather than a fixed window. El Bachyr et al. capped every chunker at 512 tokens and, with their best chunker fixed, found 25 percent overlap (128 tokens) beat both no overlap and 50 percent on page-level MRR, while doubling the overlap grew the index and scored slightly lower. Start at 512 tokens with 25 percent overlap, keep tables and headings whole inside that cap, and measure on your own evaluation set before moving either number.

How do you chunk tables in a PDF for RAG? Have the parser emit each table as Markdown with its header row, split long tables by rows, and repeat the header line at the top of every piece so no fragment is ever headerless. Add a one-line caption from the table title or the heading above it, so a query phrased in words can find a chunk whose cells hold only names and numbers.

How do I tell whether a PDF page needs OCR? Ask the parser for the page's text layer. A digital page returns its text, usually hundreds of characters, and extraction is free. A scanned page returns almost nothing while one image covers most of the page, and only those pages go to the OCR engine. In PyMuPDF that is page.get_text() plus page.get_image_info(), a check cheap enough to run on every page of every document.

What are contextual retrieval and late chunking? Contextual retrieval prepends a short model-written line to each chunk situating it in the whole document before embedding. In Anthropic's September 2024 study, the top-20-chunk retrieval failure rate (1 minus recall@20) fell from 5.7% to 3.7% with contextual embeddings, to 2.9% with a contextual BM25 index added, and to 1.9% with a reranker on top. Late chunking (Günther et al., Jina AI, 2024) embeds the whole document first, then splits, so each chunk's vector keeps its surroundings. Both sit on top of structure-preserving chunking, not instead of it.

Do you have to re-embed everything when documents change? Only when you change the embedding model, and a query adapter such as Drift-Adapter (EMNLP 2025) keeps the old index serving while that re-embedding runs in the background, so a full stop-the-world rebuild is rarely needed. For content changes, give every chunk a deterministic ID so re-ingestion overwrites in place, store a content hash so unchanged chunks are skipped, and sweep the IDs a shrunken document no longer produces. How much re-embeds per edit depends on the chunking approach: structure-preserving splits contain it to one section, fixed windows shift everything after the edit, and late chunking re-embeds the whole document.

What 2muchcoffee covers

We build production RAG systems, grounded and evaluated. Our public matcher shows how we cut structured records into typed chunks with deterministic ids. Grounded answers from your documents start with the ingestion layer, because every client's documents are messy in their own specific way. If you are looking at a corpus of scans and exports and wondering why your clean-PDF prototype fell apart on it, that is the conversation our engineers have early. The plain way in is the AI work we do.

One concrete action

Take your worst document, the scanned one, or the one with the giant nested table, and run it through your current pipeline. Do not look at the answer. Look at the chunks it produced. If a table row lost its header, or a number lost its label, or a heading got stranded on its own, you just found your retrieval ceiling, and you found it before spending a dollar on a better model. Fix the chunk, and the rest of the stack finally has something to work with.