Ground Truth Dataset Construction for Document Extraction Tasks

Ground truth construction for document extraction decides whether an extraction system's accuracy can be measured and improved once it is running in production. It is not a labeling step tacked onto the end of a project. LLM-based extractors behave like black boxes: they can hallucinate values, skip fields, misformat a date, or drop a row from a table without any visible error. The only way to catch any of this is to compare what the system produced against a human-confirmed correct answer. Without that comparison, broken extraction code ships and nobody finds out until a customer does.

A 2026 ICML position paper by Högberg, Johnson, and Wagstaff argues that ground truths are not neutral measurements that exist in the world waiting to be recorded, but get built by particular arrangements of people, tools, and decisions, and any given ground truth for a machine learning project is only one of many that could have been built from the same documents. The choices behind that construction, and what they cost the benchmark, usually go undiscussed and unreported in the resulting benchmark papers.

That has a direct practical cost. When the decisions behind a ground truth set stay invisible, a low benchmark score cannot be traced back to its cause. It might mean the extraction system failed. It might mean the schema was ambiguous, the annotators disagreed, or the corpus never included the kind of document that actually breaks in production. A team that cannot tell these apart cannot direct its engineering effort toward a fix. Every section that follows is really about one question: which construction decision produced this number, and what does the number hide if that decision was made carelessly.

Schema design as the first construction decision

Schema design comes before any annotation work begins, because a schema poorly matched to its document family makes every downstream error unattributable. A low extraction score might reflect a flaw in the schema rather than the system being tested, so the schema has to be right before the first label gets written.

A workable schema needs a clear expected value for every field it defines, for every document that field applies to. Fields left ambiguous produce ambiguous ground truth no matter how careful the annotators are. The schema also has to hold across layout variation within a document type. ExtractBench's invoice schema has to cover invoices from every vendor it includes, regardless of how different those invoices look from one another. A schema that only works for one vendor's layout isn't really a schema for invoices; it's a schema for one invoice template. The decisions made here, about which fields to include, how to type them, how to handle fields that are allowed to be empty, set the outer boundary of what the benchmark can ever penalize. A schema that leaves out a failure mode leaves the benchmark blind to it, permanently.

ExtractBench's W-14 injection and disposal permit schema shows what disciplined schema design looks like in practice. The schema defines 159 typed fields, mixing strings, booleans, and nullable fields, and each one gets scored on its own: whether the value is right, whether a checkbox reads as checked or blank, whether a field that should be empty was correctly left empty. The schema itself becomes a specification of what counts as correct extraction, field by field, rather than a loose description of what the document contains.

The failure to avoid is a schema with fields that are underspecified or that overlap with each other, something like "address" sitting next to "location" with no clear boundary between them. Annotators facing that kind of schema will label the same document two different ways, and inter-annotator agreement collapses before any annotation work has even started. ExtractBench extends this same schema discipline to the benchmark's own structure through 22 challenge tags: scan and handwriting noise, hierarchical headers, cross-page continuation, large or irregular tables, sparse facts, dense forms. These tags let scores get sliced by document difficulty. The benchmark's own difficulty taxonomy is itself a schema, applied one level up from the extraction task.

Three ground-truth construction methods

No single method for building ground truth works across every document type. The method has to match the structure of the documents in front of it and the specific failure mode the benchmark is trying to expose. Three methods cover most of the territory, and each one has a blind spot that matters as much as its strength.

The first is adjudicated multi-system agreement, built for real enterprise documents. Multiple independent extraction systems run on the same document, and wherever they disagree, a human adjudicator decides the correct value. ExtractBench uses this method for its real document corpus, and the adjudication step is what turns disagreement among systems into a settled ground truth value. The blind spot: if every system in the comparison shares the same failure mode, say, all of them truncate long record lists past a certain page count, adjudication will never catch it, because there's no disagreement to flag. The benchmark will understate that failure across the board.

The second is known-value synthetic generation, suited to documents with repeated structure, especially long lists and large tables. Here the values are fixed before the document is even rendered, so the ground truth exists before the document does, and no conversion artifact can appear between the true value and the recorded one. ExtractBench uses this for its synthetic list documents. A related approach pulls real tables out of arXiv papers and embeds them into synthetically generated PDFs, which gets exact ground truth without manual annotation while keeping the layout diversity of real documents. This method matters most for benchmarking large-array extraction, because silently dropped rows are the central failure mode on long, table-dense documents in production, and known-value generation is the only one of the three methods that can reliably catch it, since the correct row count is locked in before any extraction system touches the file.

The third is human verification, needed for forms and any document where field identity depends on bounding-box position. Annotators mark boxes and values directly against the source document. ExtractBench uses this for its form documents, because handwriting is often unclear and a mark can sit ambiguously between two fields, so a person has to decide. This method costs the most per document, and it's the only one that produces the bounding-box grounding metadata needed to score source traceability as well as value accuracy.

Choosing the wrong method for a document type doesn't just waste annotation budget. It produces a benchmark that is structurally unable to see the failure mode that matters most for that document type, no matter how carefully the rest of the pipeline is built.

Inter-annotator agreement as the governing quality check on constructed ground truth

Inter-annotator agreement is the only honest stand-in for objective ground truth in enterprise annotation work. A benchmark's accuracy numbers are only as trustworthy as the IAA behind the labels that produced them.

Accuracy as a concept assumes a known ground truth already exists to measure against. In most enterprise document work, that ground truth is what the annotation team is trying to build in the first place, so accuracy can't be the check. IAA measures something more modest: whether two or more independent reviewers, given the same example and the same written guideline, land on the same label. General enterprise classification work targets a Cohen's κ above 0.80, and regulated, safety-critical domains like medical imaging or financial fraud detection set the bar higher still.

Real annotation work often falls well short of that target. A 2024 multi-task annotation study on financial annual reports, covering five tasks (relevance, sentiment, objectivity, forward-looking statements, and ESG classification) across multiple annotators and sentences, found Cohen's κ scores ranging from 0.10 to 0.43. These are low numbers for a domain that enterprises routinely treat as well-understood and straightforward to label. If trained annotators working on financial text, a domain with established conventions and plenty of prior annotation work to draw on, produce scores in that range, a document extraction benchmark built without per-field IAA measurement and adjudication will report accuracy scores that reflect annotator disagreement as much as they reflect how well the extraction system actually performed.

Regulated healthcare settings offer a model for what it takes to push IAA higher. Annotation workflows there run multiple rounds of review, build consensus among reviewers, and check inter-rater agreement at each stage. Many projects use redundant labeling, where every item gets labeled by more than one annotator and any discrepancy gets resolved by a senior expert. That redundancy costs more, but it produces ground truth that holds up, and PHI handling rules add further controls on top, including on-premises annotation and signed confidentiality agreements. The lesson for document extraction is that IAA has to be measured and acted on as part of building the dataset, not checked once at the end as an audit of work already considered finished.

Coverage of hard cases: why a benchmark that avoids difficult documents measures the wrong thing

Which documents go into a benchmark's corpus is a design decision with the same weight as schema design and annotation method, not a neutral sampling exercise. A benchmark built from easy, clean, short documents produces accuracy figures that don't predict how a system performs in production. Scanned pages, handwritten entries, multi-page records, dense tables, facts that continue across a page break: these are the documents that stress extraction systems, and they are the documents most enterprises actually process day to day. A benchmark that underrepresents them is a benchmark that reports numbers too good to be true.

ExtractBench's challenge tags identify the specific failure modes that appear once hard documents are included. Commercial vision-language models often truncate record lists on long documents even when every value they do read is correct. Missing rows in long lists is a distinct and common failure. A value that appears more than once in a document, in different contexts, can trip up a system into selecting the wrong occurrence of a sparse fact. Dense forms invite overfilling, where a system inserts a value from an adjacent field into the wrong slot. Similar dates, identifiers, and amounts get confused with each other. And perception itself can fail outright, on scan or handwriting noise, hierarchical headers, cross-page continuation, and large or irregular tables.

Document length adds a separate axis of difficulty that short-document benchmarks cannot capture. A corpus built around documents averaging hundreds of pages, with tens of thousands of ground-truth fields, shows recall, the fraction of ground-truth rows a system actually returns, ranging from below half to near-perfect depending on the system, while precision and leaf-level accuracy stay clustered high across the board. On documents that long and that dense, completeness separates systems from each other, not per-value accuracy.

That has a direct structural consequence for how any benchmark gets read. A benchmark reporting only per-value accuracy on short documents cannot tell apart a system that extracts every row in a table from one that silently drops most of them. Both can post identical scores on short, clean documents and then diverge sharply once they meet the documents enterprises actually have sitting in their systems.

Scoring design: what the choice of metric decides about what the benchmark can see

The metric chosen to score a benchmark decides which extraction failures are visible, which ones get averaged away, and which system behaviors get rewarded even when those behaviors would be unacceptable in production. A benchmark with the wrong metric can rank systems in the opposite order from how they'd actually perform once deployed.

Large-array extraction exposes the core tension between precision-oriented and recall-oriented scoring most clearly. A system that returns a small, highly accurate subset of rows from a long table will score well on precision and on per-value F1, because every row it did return was correct. But in production, silently dropping most of the rows in a table is a catastrophic failure on its own terms, not a tradeoff an enterprise can live with. LongExtractionBench addresses this directly by reporting precision, recall, and a combined leaf score as three separate numbers rather than collapsing them into one figure that would hide a completeness failure behind a strong precision score. It also records explicitly when a system fails or refuses to process a document, reporting a separate completion rate alongside any accuracy number.

For scoring rows within a table, order shouldn't matter, since rows in most enterprise tables carry no meaningful sequence. ExtractBench scores array rows with order-insensitive value F1, matching rows in any order rather than by position, using a globally optimal one-to-one assignment (the Hungarian algorithm) to pair extracted rows against ground-truth rows. That avoids penalizing a system for a position difference that carries no real meaning.

Value accuracy is only one axis. ExtractBench also reports word-level and page-level grounding F1, which measures whether a system can point to where in the source document each extracted value came from. A system that returns the right values with no traceable evidence behind them cannot support the human review and audit workflows that regulated industries require as standard practice, so grounding has to be scored as its own dimension.

A related risk is conflating OCR benchmarks with extraction benchmarks, since the two measure genuinely different things. OCR benchmarks score character-level or word-level text accuracy, reported as word error rate or character error rate. Document extraction benchmarks score structured field accuracy, table completeness, confidence scoring, and multi-page handling. A high score on an OCR benchmark says nothing reliable about extraction accuracy on a multi-column table, where getting the field right depends on position on the page as much as on reading the characters correctly.

Cost belongs in the same scoring picture as accuracy and grounding. ExtractBench measures per-page cost for any evaluated system that exposes a meaningful price, because a system's accuracy and grounding scores only mean something to a buyer once set against what each page costs to process. A benchmark that reports accuracy without cost leaves out the number that actually decides which system gets deployed.