Error Analysis for Low-Accuracy Document Extraction Fields

An extraction pipeline can report high overall accuracy and still fail, consistently and predictably, on the three or four fields a downstream agent actually depends on. That is the central problem this piece addresses: aggregate accuracy is a summary statistic, and summary statistics hide exactly the information a team needs to fix anything. A single blended number can rise even as checkbox states, table totals, and blank fields degrade, because those fields are often a small share of the total field count even when they carry most of the business risk. The fields most likely to break are not spread evenly across a document. Values tied to a specific row and column, checkbox or selection states, figures pulled from dense tables, and fields that must be correctly reported as empty are structurally harder to extract than a paragraph of plain text, and they tend to be the fields an agent takes action on. In mortgage underwriting, financial reporting, supply-chain logistics, and clinical records, a misread field does not just lower a score. It triggers a wrong approval, a miscoded claim, or a shipment against the wrong quantity, so the cost of failure concentrates on a narrow set of fields rather than spreading across the document evenly. Diagnosing those failures, field by field and mechanism by mechanism, is the only way to fix them without guessing.

The four mechanisms that reliably produce low-accuracy fields

Low-accuracy fields do not appear at random. They cluster around four identifiable causes: source-quality degradation, layout and table complexity, a mismatch between the benchmarks used to select a model and the documents that model will actually process, and field-label variance combined with brittle parsing rules. Each of these four produces a different failure signature, and each has a different fix. Treating them as one undifferentiated "accuracy problem" is how teams end up retraining a model to solve a schema bug, or rewriting a schema to solve a scanner problem. The four mechanisms form a map rather than a ranked list: none is inherently worse than the others, and a single document can suffer from more than one at once. A faded handwritten form with a nonstandard table layout processed by a model chosen for its leaderboard score, mapped against an ambiguous schema, can fail for all four reasons simultaneously, and a team that only checks the overall accuracy score will never find out which one to fix first. The next four sections take each mechanism in turn, starting with the one that happens before the extraction model ever sees the page.

How source quality degrades specific fields before extraction begins

Source-quality problems rarely degrade a whole document evenly. They corrupt particular fields and leave others intact, because the noise, handwritten annotations, stamps, faded ink, scan skew, is concentrated in specific regions of a page rather than spread across all of it. A loan form scanned at low resolution with a handwritten signature field and a stamped date can produce a clean extraction on every typed field while failing completely on those two. RealDocBench, a benchmark built specifically from real regulated documents, includes handwriting, stamps, redactions, and scanner artifacts as core characteristics of its corpus, reflecting that these are standard features of production documents in mortgage, financial reporting, and logistics workflows, not rare edge cases a system can safely ignore. ExtractBench takes this further by tagging its 370-document corpus, spanning 8 business domains and 67 document types, with 22 distinct challenge tags, including scan noise and handwriting, and scoring systems per-challenge. That tagging makes it possible to isolate exactly which challenge type is responsible for a given field's failure, rather than guessing from an aggregate number. Medical and clinical records are a persistently hard case in RealDocBench's evaluation, in part because they combine dense forms, physician handwriting, and multi-page structure, and these problems compound rather than average out. The diagnostic step that follows from this is simple to state and easy to skip: when a field fails consistently, inspect the source image before touching the model. A field that fails on every document from one scanner, one fax line, or one originating office is a source-quality problem, and no amount of model substitution will fix it.

Table structure, layout complexity, and parsing-layer failures

A large share of field failures happen before extraction even starts, at the parsing layer, where the raw document gets converted into structured, machine-readable content. If a table's rows and columns are misread at that stage, the extraction model never had a chance to get the field right, because it is reasoning over corrupted input rather than the actual document. ParseBench evaluates parsing systems across five capability dimensions, tables, charts, content faithfulness, semantic formatting, and visual grounding, specifically because agent workflows depend on more than text similarity. An agent needs correct table structure, precise chart data points, formatting that preserves meaning, and reliable visual grounding, and ParseBench finds that no current parsing method is consistently strong across all five dimensions. The consequences of small parsing errors are concrete and immediate. An agent approving an insurance claim can act on the wrong value if a table header is misaligned by one column. An agent summarizing a financial report can produce a materially wrong analysis if a chart gets flattened into raw, disordered text instead of preserved as data points tied to axis labels. Formatting that carries meaning, strikethrough text marking a voided line, superscripts, indentation that signals a subtotal, gets dropped by parsers routinely, and once it is dropped, the context an agent reasons over is silently wrong without any visible error. ExtractBench's list of common failure cases in schema-guided extraction traces back to this same layer: missing rows in long lists, picking the wrong instance of a value that appears more than once, overfilling dense forms with values that do not belong, and confusing similar dates, identifiers, or amounts that sit close together on the page. Cross-page continuation is one of the more measurable failures in this category. ParseBench scores it directly as part of its Tables dimension, alongside merged cells and hierarchical headers, and ExtractBench finds that commercial vision-language models, which do well on short documents, often truncate record lists once a table runs across multiple pages. The diagnostic pattern to watch for: a field that fails inconsistently across documents of the same type, tracking the issuer's layout rather than the document's content, points to the parsing layer. The fix starts with inspecting the intermediate parsed representation of the document, not the final extracted value, because the final value is only as good as what the parser handed the model.

Model selection, benchmark results, and field failures on production documents

A model selected because it topped a standard benchmark leaderboard can still fail reliably on production documents, because most leaderboard benchmarks test a version of the problem that production documents do not resemble. Standard benchmarks tend to use clean, typed text in consistent layouts. Production workflows hand the same system scanned PDFs with page skew, a mix of handwriting and type on the same form, rotated headers, and compression artifacts from a fax machine or a phone camera, none of which most benchmarks score. RealDocBench was built to target this exact gap, pointing out that popular OCR and document-understanding benchmarks draw heavily on academic papers, books, and content that was born digital, all of which carry clean, predictable layouts that bear little resemblance to a mortgage file or a clinical chart. A model can post a strong number on a string-similarity metric and still misread the one field a downstream agent needs to act on correctly, because string similarity and field-value correctness are not the same measurement. RealDocBench names this gap explicitly: existing benchmarks test unrealistic documents, score the wrong metric (string similarity instead of field-value retrieval), and report a single blended number that hides exactly which domain or structure is driving the failure. ExtractBench adds dimensions that fixed-ontology benchmarks such as SROIE and DocILE mostly leave out, scans and handwriting, long multi-page records, and grounding, and finds that scoring on these dimensions produces meaningfully different system rankings than the standard leaderboard order. A related failure mode occurs when teams skip specialized parsing entirely and route raw document images straight to a frontier language model, as if it were a substitute for OCR. That substitution is an active source of field-level errors in naive pipelines: general-purpose vision models are not built to handle complex, irregular layouts the way dedicated parsing systems are. RealDocBench's own evaluation, which spans commercial parsing APIs, general-purpose vision-language models, and open-source OCR systems, finds a wide spread in per-field accuracy across these system types, with no single type dominating across the board. Cost compounds the problem, because it is invisible on most accuracy leaderboards. ExtractBench measures cost per page across systems and finds a very large spread, which makes "best benchmark accuracy" an incomplete basis for choosing a system when two systems can reach similar accuracy on simple documents at wildly different costs. The practical diagnostic question, when a field fails in production but passed on the test set used to select the model: does that test set contain documents with the same structural properties as the ones failing in production? If it does not, the benchmark was never predictive for that field type, and no amount of re-running the same benchmark will reveal the gap.

Schema design, field-label variance, and extraction failures that look like model errors

Some of the fields marked as model failures are schema failures, and the distinction changes everything about how a team should respond. Schema-guided extraction, as ExtractBench defines it, requires a system to follow a user-defined schema faithfully across every document of a given type, no matter how different each document looks on the page. A schema that is ambiguous, or that was written around the layout of one specific issuer's document, will produce inconsistent field values across other issuers even when the underlying extraction model is working exactly as intended. Consider an invoice schema meant to cover every vendor a company works with. If a field in that schema is defined by a specific label string rather than by its semantic meaning (the final payable amount), it will fail the moment a different vendor labels the same value with different wording. The model is following a schema that was never built to recognize that label in the first place, not confused about what number to extract. The diagnostic signal here is distinct from the other three mechanisms: a field that fails on some issuers but succeeds on others, with no meaningful difference in scan quality or layout complexity between them, points to a schema definition problem rather than a model or parsing problem. Teams that find this pattern should treat it as fixable with schema work: refining the field description to capture semantic intent rather than a literal label, adding known label variants, or restructuring how the field is defined. None of that requires touching the model, retraining anything, or switching vendors. It requires sitting down with the schema and rewriting the field definition so it matches intent rather than one issuer's wording.

Per-field confidence scoring as the mechanism for catching which failure mode is active

A confidence score attached to a whole document tells a team almost nothing useful. A confidence score attached to each individual field tells a team which of the four mechanisms is likely responsible for that field's failure, and that distinction is what makes per-field scoring the right production instrument for diagnosis. ExtractBench builds this logic into its evaluation by introducing grounding metadata, word-level and page-level traceability back to the source document, as a scored dimension alongside value accuracy. A confidence number alone cannot tell a team whether a correct extraction happened to carry low confidence or an incorrect extraction carried high confidence; grounding metadata is what closes that gap, by pointing to exactly where on the page a given value came from. The pattern of low-confidence fields across a document set becomes a diagnostic map once grounding is available. If fields cluster at low confidence on documents coming from one specific source or scanner, that points toward source-quality degradation as the active mechanism. Fields that fail inconsistently across different issuers while carrying high confidence point toward schema ambiguity, because the model is confidently applying a schema that does not generalize. If fields fail specifically on long, multi-page documents but succeed on short ones, that points toward the truncation failure mode that ExtractBench documents in commercial vision-language models. Grounding metadata makes the underlying mechanism visible rather than inferred: if a low-confidence field's cited source region shows a visibly degraded scan, that confirms a source-quality problem. If the citation points to the wrong row in a table, that confirms a parsing-layer problem rather than a model or schema issue. The score and the citation together turn a vague "this field is often wrong" observation into a specific, testable claim about which of the four mechanisms is at fault.

Routing low-confidence fields and using human corrections to close the accuracy gap

Per-field confidence scores become useful the moment they are used to route fields rather than just to flag them. The dominant production architecture sorts extracted fields into three tiers by confidence. High-confidence fields are auto-accepted and written directly to the system of record without a human in the loop. Mid-confidence fields go to human review with the model's extracted value pre-filled, so a reviewer is confirming or correcting rather than starting from a blank form. Fields that fall below a set low-confidence threshold get routed to manual entry from scratch, on the assumption that a pre-filled wrong answer is often more dangerous than no answer. Setting that low-confidence threshold strictly, so that any field below it is always flagged rather than silently passed through, recovers meaningful accuracy gains without requiring any change to the underlying extraction model. The gain does not come from the model getting smarter. It comes from stopping incorrect values before they reach a system of record where a downstream agent would act on them as fact. The corrections human reviewers make at the mid-confidence and low-confidence tiers are not just a safety net. They are the most direct signal available for identifying which of the four root causes needs fixing next. A reviewer who keeps correcting the same field on documents from the same scanner is pointing at a source-quality problem. A reviewer who keeps correcting the same semantic field across different issuers using different labels is pointing at a schema that needs to be rewritten. Multi-agent parallel extraction, where multiple extraction backends process the same document independently and a consensus mechanism checks their outputs against each other, extends this same logic one step further: agreement across backends raises confidence on a given value, and disagreement flags it for review automatically, without waiting for a human to notice the pattern first. The loop closes when those corrections feed back into the right layer: a source-quality fix for a scanner, a parsing fix for a layout, a schema fix for a label, and a model or vendor change reserved for the failures that are actually the model's to own.