Benchmarking Document Extraction Accuracy on Production Document Sets

Document extraction benchmarks report accuracy scores, but if you deploy against those scores, you can watch accuracy fall once the system meets real production traffic. The gap is not a vendor problem or a one-off bad model: it is a structural mismatch between what benchmarks test and what production documents actually look like.

Standard benchmarks and production performance

An engineer picks a model off a leaderboard, runs a pilot, and finds the accuracy numbers from the benchmark do not hold once real documents start flowing through the pipeline. The reason sits in the test set, not the model. The Fraunhofer IAIS and Lamarr Institute study from 2025 ran the same model, with the same prompt, across three document types, and found a nine-point spread between clean digitally generated invoices and scanned receipts. That spread shows accuracy is a property of the document set a model is run against rather than a fixed trait of the model itself. Most public benchmarks draw their test documents from curated, well-aligned scans, but when those same models meet realistic industrial documents, research into OCR robustness for retrieval-augmented systems has found clear accuracy degradation, even when the benchmark scores looked strong going in. High character-level accuracy does not carry over into strong downstream performance, because structural and semantic errors cause retrieval failures, and character-level metrics never catch them. Public leaderboards rarely say anything about the operating conditions of production deployment: how a system handles scan defects, what latency budget it needs, what governance a deployment requires, and what happens at real processing volume.

Production document sets versus benchmark test sets

Production documents carry layout complexity, length, physical degradation, and domain-specific structure that curated benchmark corpora leave out by design, which makes benchmark scores optimistic before a single document is deployed. Document quality turns out to be the variable that decides whether a score means anything. Scan noise, handwriting, unusual reading orders, extreme page layouts, watermarked backgrounds, and high-resolution pages all appear constantly in production queues, and almost none of them appear in the clean test sets most benchmarks use. OCR robustness research for industrial retrieval systems lays out the specific document types that expose this gap: extreme layouts, high-resolution pages, complex or watermarked backgrounds, historical documents with non-standard reading order, visually decorated text, tables, and mathematical formulas. A line-item table on a logistics manifest, a handwritten insurance claim form, and a scanned permit with a watermark running behind the text each break a different assumption that clean benchmark data quietly relies on.

Length and repetition create a failure mode of their own. A system can read every individual value on a page correctly and still drop rows once a record list runs long: ExtractBench, published in July 2026, formally separates this into task-structure and table-structure challenges, naming cross-page continuation and large or irregular tables as document-understanding problems that standard benchmarks simply do not penalize. Commercial vision-language models can perform well on short documents, but when those documents get long, they quietly truncate record lists, and because the failure doesn't throw an error, field-level accuracy scores never catch it.

Schema complexity adds a second axis of difficulty, one that benchmarks built around a single, fixed set of fields cannot represent. ExtractBench covers schemas that range from tens to several hundred fields, and one of its examples, a W-14 disposal permit, spans 175 fields, arrives scanned and handwritten, and requires a system to tell an unchecked box apart from a blank one. Single-domain benchmarks built around a fixed, narrow ontology don't carry any of that complexity. Finance, healthcare, logistics, and real estate each bring their own document types, invoices, medical claim forms, bills of lading, permits, and each of those types carries its own layout conventions, field densities, and reading order that differ from the invoice-heavy corpora most benchmarks still lean on.

How existing benchmarks are built

No single benchmark available today covers every dimension that matters for a production shipping decision. Each one covers a slice, and the slices line up in a fairly predictable pattern once you look across them. OmniDocBench, presented at CVPR 2025, evaluates four parsing modules, text, tables, formulas, and reading order, using document component segmentation and matching. It measures parsing quality rather than extraction quality, and it does not score schema-guided output or cost at all. RealDoc-Bench takes a different angle: it checks whether a parser produces accurate layouts, preserves reading order, and lets a downstream agent answer objective questions correctly against real-world documents. So this design centers the benchmark on the agent use case, not on character-level OCR metrics.

Between these two sits a separate concern: who runs the benchmark and whether the methodology can be trusted on its own terms. A benchmark that a vendor builds, publishes, and runs against its own product carries an inherent credibility problem, since the party reporting the score also controls every variable that produces it. Open, reproducible benchmarks, built around published datasets and public evaluation code, exist as the field's answer to that problem, because they let any practitioner rerun the evaluation and check the number independently. Taken together, the current landscape asks a practitioner to triangulate across several benchmarks, each testing a different layer of the problem, since no one of them, on its own, gives a complete picture of whether a system is ready to ship.

The specific failure modes that benchmark scores hide

Picture a healthcare operations team that extracts line items from a batch of claim forms. A system scores 95% field accuracy on the batch, which sounds like a system ready for production. Aggregate accuracy scores hide four distinct failure modes that each independently corrupt downstream agent behavior: silent row truncation, hallucinated values, grounding failure, and reading-order errors.

Row truncation is the most dangerous of the four because it produces no error signal at all. ExtractBench's findings show commercial vision-language models performing well on short documents while quietly truncating record lists on long ones. A system that correctly extracts 95 of 100 line items scores 95% field accuracy, and that score hides the fact that five payments or five shipments never made it into the output. Nothing in the pipeline flags the gap, because the fields that were extracted were extracted correctly.

Hallucination is a separate and more familiar risk. Vision-language-enhanced extraction systems sometimes produce plausible values that never appeared anywhere in the source document, and because these failures are stochastic, no amount of inspecting the input document in advance will predict them. An invoice total that the model invents, or a policy number fabricated out of whole cloth, can pass a field-level accuracy check and still turn into a wrong payment or a compliance violation once it reaches a downstream system.

Grounding failure gets less attention than the first two but matters just as much in any regulated workflow. ExtractBench scores word-level and page-level grounding separately from value accuracy, because a system can return the correct value and still attach it to the wrong page or the wrong bounding box. That mismatch breaks the audit trail and makes human review unreliable, since a reviewer checking the system's work has no way to confirm where a value actually came from. In a regulated industry, a value that cannot be traced back to its source is not auditable, no matter how accurate the value itself turns out to be.

Reading-order errors round out the list. OCR robustness research shows that structural and semantic errors drive substantial retrieval failures even when the word-error rate on a document stays low. The model reads the correct characters, places them in the wrong order, and hands a downstream agent a structurally corrupted version of the document to act on. Field-level F1 averaged across an entire document rewards a system that nails the easy fields and fails the hard ones, so a system that extracts vendor name and date perfectly while truncating the line-item array can still score well on most benchmarks, and it still fails once it reaches production.

What a rigorous production benchmark must measure

A benchmark built for production decisions has to score value accuracy, record completeness, source grounding, and cost as four separate measurements, not blend them into one number, because optimizing for one of these can quietly degrade another.

Value accuracy needs to be scored in a way that does not penalize correct extraction for arriving in an unexpected order. ExtractBench uses order-insensitive value F1 specifically so a correctly extracted line-item array isn't marked down for the sequence in which its rows come back. Position-dependent scoring rewards a system for guessing the right order, not for extracting the right values.

Record completeness needs its own score, separate from field accuracy, so that truncated output cannot hide behind a high accuracy number on the fields that did make it through. If a benchmark folds completeness into one blended F1 score, truncation failures can disappear behind the accuracy of whatever remained. A dropped row has to count as a distinct, explicit penalty, not get treated the same as one misread character.

Grounding has to be measured at two separate levels. Word-level grounding confirms a value traces back to the correct bounding box on the page. Page-level grounding confirms it traces back to the correct page entirely, which matters most in multi-page documents where the same field label can appear more than once. ExtractBench scores both levels independently, and most benchmarks currently score neither.

Challenge tags give practitioners a way to look past the aggregate number entirely and evaluate a system against the specific document types it will actually face. ExtractBench uses 22 tags covering conditions like scan noise, handwriting, merged headers, headers that aren't positioned at the top of the page, cross-page tables, and unusually large tables. With that tagging, you can slice results by difficulty instead of relying on one blended score that mixes easy documents with hard ones.

Cost belongs inside the accuracy evaluation itself, not reported as a separate line item. A system that reaches higher accuracy by running coding agents at 16.2 to 27.8 cents per page is making a different tradeoff than one that reaches comparable accuracy at 8.1 cents per page, and a benchmark that reports accuracy without cost hides that tradeoff from the person making a deployment decision. ExtractBench reports a cost range spanning 0.21 cents to 34 cents per page across the systems it evaluates, so you can compare accuracy and cost side by side rather than argue about them separately.

Finally, the test corpus itself has to include real scans, handwriting, and multi-page documents alongside clean digitally generated PDFs, and it has to span multiple business domains. RealDoc-Bench builds its evaluation around real-world documents and checks whether a parser lets a downstream agent answer objective questions correctly, grounding the benchmark in agent correctness rather than isolated OCR metrics alone.

In-domain private evaluation sets alongside public benchmarks

Even if a benchmark meets every criterion above, it cannot substitute for testing against a team's own documents, because the mix of challenges in any public corpus will never exactly match what a specific deployment receives. A public benchmark answers a narrower question well: which systems can be ruled out immediately because they can't handle scans, can't handle long arrays, or can't handle a complex schema. That capability triage is valuable, and it is not sufficient on its own to justify a shipping decision.

The methodology that holds up combines both layers: public benchmarks for capability triage, paired with a private, in-domain evaluation set built from a team's actual business traffic and scored against a defined rubric. Building that private set means sampling from documents the system will actually see in production, including the edge cases and failure cases a team has already run into, rather than curating a tidy set of representative examples. It means scoring that private set along the same dimensions a rigorous public benchmark uses: value accuracy, completeness, grounding, and cost. Ground truth has to come from human verification, because a system graded against its own output will always look like it's passing.