Field-Level Accuracy Metrics for Structured Document Extraction

An engineer looks at a vendor's extraction accuracy claim, signs off on a pipeline, and watches it fail in production anyway. Nobody on the team can explain why, because the number on the spec sheet and the number the pipeline actually delivers are measuring two different things. This article exists to close that gap: vendor accuracy claims blend together at least four distinct metrics, and mistaking one for another is what turns a clean benchmark into a broken production system.

Why "accuracy" in document extraction is not a single number

The technical literature on document extraction defines four separate measures, and each one answers a question the others can't. Character accuracy, often expressed as Character Error Rate, counts the percentage of individual characters wrongly recognized through substitutions, deletions, and insertions. It's the oldest metric in OCR and the least useful one once the job is pulling structured values out of a document, because it says nothing about whether the right value landed in the right field. Field accuracy checks each named field in a schema and asks a binary question: correct or not, with no partial credit. Document accuracy, sometimes called the zero-error rate, asks the stricter question of whether every field in a document came out correct, so one wrong field fails the whole document. Pipeline-level accuracy measures end-to-end correctness after human review and correction, and it is the number that decides whether a business process works, but it's also the number vendors rarely publish.

When a vendor states an accuracy number, three questions separate an engineering claim from a marketing one: which of the four metrics is being reported, what document mix was it measured on, and how was it validated against ground truth. A vendor that cannot answer all three isn't giving you data you can build on.

Character accuracy as the wrong unit for extraction pipelines

Character accuracy measures how well a system reads text. It does not measure whether that text ends up correctly assigned to a field, which is the job an extraction pipeline actually has to do.

The failure is structural. A document can be wrong on every field that matters and still score well on CER, because CER has no concept of "fields that matter."

The problem compounds once layout enters the picture. Standard OCR tools flatten a page's spatial structure into a single stream of characters. Take a bank statement with multi-column transaction tables: the date from one column ends up textually adjacent to the description from the next column over, and the spatial relationships that gave those numbers their meaning are gone before an extraction model ever sees the text. A large language model trying to pull structured fields out of that flattened stream is no longer reading a document. It's reading noise that happens to be shaped like text.

Character accuracy survives in vendor benchmarking for a practical reason: it's cheap to compute and easy to make look good, but the value a system read must end up correctly labeled in the schema, which is what field accuracy was built to answer.

Field-level accuracy and downstream reliability

Field accuracy counts the percentage of named fields, invoice number, date, total, vendor name, tax amount, that come out both correctly extracted and correctly matched to their label. It's a binary call on each field, and that binary nature matches the binary reality of the systems downstream. Reconciliation gives no partial credit, so you shouldn't grade extraction as though it did.

The consequences of a wrong field are concrete and specific to the field itself. A wrong invoice number breaks reconciliation against the purchase order. Field accuracy is the metric that predicts these outcomes because it's measured at the same grain the downstream process operates on.

Not all fields are equally hard to extract, and this is where an honest evaluation has to do more work than a vendor benchmark usually does. Header fields, the invoice number at the top, the date, the vendor name, appear in predictable locations and follow common formats, and most modern extraction systems handle them reliably. The real test is line-item extraction, pulling individual product descriptions, quantities, unit prices, and line totals out of tables that vary in structure from one supplier's invoice template to the next. This is where extraction systems diverge from one another in capability, and most benchmarks underweight it.

Standard metrics like precision, recall, and F1 score get used here, but they weren't built to capture the business-critical precision you need for a field like a tax ID or a payment term. A reliable field-level framework instead requires a clear ground truth for each targeted field, measurement taken at the field level rather than aggregated across the document, and results stratified by document condition: digital PDF against scanned PDF, skewed or noisy scans, stamps and handwriting, different languages and currencies, and templates from suppliers the system has never seen before. That last condition matters most for long-tail suppliers, where layouts vary so widely that a system tuned on a handful of common templates can quietly fail on everything else.

Different field types also need different definitions of "correct," and applying one rule to all of them produces a misleading score. An identifier like an invoice number or tax ID needs an exact match, with no tolerance and no normalization, because a one-character difference points to a different invoice. A name or description needs semantic equivalence rather than string equality: "International Business Machines" and "IBM" may both be the correct answer to the same field. Collapsing all three of these into one recursive string comparison, the way a lot of evaluation tooling still does, treats a cosmetic formatting difference in a vendor name the same as a wrong tax ID, which tells an engineer nothing useful about where the real risk sits.

How document-level accuracy exposes failures that field accuracy hides

Document accuracy applies a harsher standard than field accuracy, and the gap between the two numbers is where a lot of production risk hides. A pipeline can post strong field accuracy and still send a large share of documents to a human reviewer, because document accuracy asks a stricter question: did every field on this document come out correct, with zero exceptions.

Think of each required field as its own point of failure. A document with five required fields only needs one of them wrong to fail the document-level check, even if the other four are perfect. Adding more fields to the schema adds more chances for at least one of them to come out wrong, so the probability that every single field lines up correctly shrinks as the schema grows, even while the accuracy of any individual field stays fixed. If a schema has many fields, each individually accurate at a respectable rate, the document accuracy can still come out low enough to make fully automated, touchless processing impractical.

That's the number that determines the straight-through processing rate: the share of documents that can move through a pipeline without a human ever looking at them. ExtractBench found sharp degradation in this measure as schema breadth grew, and on a 369-field financial reporting schema, every tested frontier model returned zero percent valid output. A failure that total cannot be surfaced by field-level accuracy taken alone, since it only appears when document accuracy is measured across the whole schema.

The business metrics that matter track document accuracy directly rather than any single field's score: the straight-through processing rate, the manual review rate, and the false auto-approval rate, the share of documents that get waved through as correct when they actually contain an error. Those three numbers reflect the operational risk a finance or operations team actually carries, risk that a character accuracy score or an averaged field accuracy score cannot capture.

Schema breadth as the dimension vendor benchmarks systematically hide

Published benchmarks tend to test narrow, flat schemas, and a system that performs well on five or ten fields can fail outright once the schema reflects what enterprise documents actually require. That gap exists because of two structural holes in how extraction gets evaluated.

The first gap is in the benchmarks themselves. Document understanding benchmarks focus on span- or entity-level extraction against small schemas. Structured output benchmarks test whether a model can produce valid output against a schema using clean text prompts, but they don't test whether the values it extracted are correct. No existing benchmark, prior to ExtractBench, jointly covered extraction straight from PDFs, enterprise-scale JSON schemas, and evaluation at the level of individual keys.

The second gap is in methodology. Most existing evaluation frameworks run a single recursive comparison across every field in a schema: exact equality for primitive values, position-dependent matching for arrays of objects, set overlap for arrays of scalar values. That one-size-fits-all comparison penalizes an array that got extracted correctly but in a different order than the ground truth, and it draws no distinction between a field that's missing entirely and a field that was correctly extracted as an explicit null.

Arrays expose the sharpest version of this problem. Aligning extracted rows against ground-truth rows requires matching each row to its counterpart individually. That result scores near-perfect on character accuracy, high on field accuracy computed across the nine rows that are present, and zero on document accuracy. No metric in a flat benchmark flags the dropped row at all, unless the evaluation was explicitly built to penalize omission as its own failure mode, distinct from a wrong value.

ExtractBench addresses this by contributing a dataset of PDF, JSON Schema, and gold JSON triplets, with human-annotated labels spanning multiple domains and a wide range of schema complexities, covering 12,867 evaluatable fields in its 2026 analysis. Its contribution is treating schema breadth as a variable to test against instead of holding it fixed and small, and that variable reveals the degradation flat accuracy metrics miss.

Vendor benchmarking compounds the problem on top of the methodology gap. Vendors choose which fields to report on, usually the easier header fields, and they run their evaluations against clean test documents drawn from a narrow set of templates. None of that makes the reported numbers false. It makes them conditional on favorable inputs that don't resemble the document mix a production pipeline actually has to handle.

Designing a field-level evaluation that reflects production conditions

An honest evaluation has to treat the schema itself as an executable specification: a structure of typed, individually verifiable fields. Each field should declare the comparison rule appropriate to its type, and the evaluation should enforce that rule rather than running one comparison across the whole document.

That means different scoring logic for different field types. Arrays need alignment-aware matching that compares each extracted row to its true counterpart individually, and the evaluation has to treat a missing row and a wrong row as separate failure modes, since they carry different downstream risk: a wrong row produces bad data, while a missing row produces an incomplete record that may never get caught.

Results need to be broken out by document condition as well as by field. A system's real reliability only becomes visible once results are split across digital versus scanned PDFs, skewed or noisy scans, stamps and handwriting, different languages and currencies, and templates from suppliers the system has never encountered before.

The test set itself has to resemble the documents a pipeline will actually face in production. Header fields alone won't do it. Ground truth also needs to come from independent annotation.

Confidence calibration belongs inside this evaluation, not off to the side as a separate concern. The failure mode that causes the most downstream damage is a confident wrong answer: the system returns an incorrect value without flagging it as low-confidence, the value passes through review routing untouched, and it lands in a downstream system as though it were verified. An evaluation that skips testing calibration has no way to catch a system that's fluent and wrong.

Probabilistic extraction also needs deterministic rules sitting alongside it. A duplicate invoice number, a failed three-way match between invoice, purchase order, and receipt, or a changed bank account on file should route to human review automatically, regardless of how confident the model sounds about the extraction. Those rules exist to catch the cases where a model is confidently wrong, cases that calibration alone can miss.

Open-source benchmarks and frontier model performance on real schemas

When evaluation methodology accounts for real schema complexity instead of the small, clean schemas you see in academic datasets, frontier models show failure patterns that occur only under narrow benchmarks' blind spots.

ExtractBench's results make the pattern concrete. Performance degrades sharply as schema breadth increases, and on the 369-field financial reporting schema, every frontier model tested returned zero percent valid output, a collapse that you'd never see in a benchmark limited to a handful of fields. In one domain tested, models produced syntactically valid output at a high rate while the actual extraction correctness told a very different story, which confirms that whether a model's output parses and whether its output is right are independent questions, and a benchmark that checks only the first one is answering the easier half of the problem.

Part of the cause is structural. Long documents paired with large schemas force models to generate long, deeply structured JSON output, and that kind of output is brittle: it's prone to truncation partway through and to formatting errors that break parsing, regardless of whether the underlying extraction was accurate. The pattern holds across the frontier models ExtractBench tested, and it points to the same conclusion the rest of this piece has been building toward: a number that looks strong on a small schema is not evidence that the same system will hold up once the schema looks like the ones enterprises actually run.