2024 Document Extraction API Accuracy Benchmark Scorecard

The Document Extraction API Accuracy Benchmark Scorecard sets out how to choose a high-accuracy API for complex real-world layouts. Compare schema adherence, row retention, field-level accuracy, source grounding, parsing quality, and cost separately. A headline accuracy score cannot tell an engineer whether an API will handle scanned invoices, multi-column documents, or long tables without losing data silently.

Why aggregate accuracy scores on vendor scorecards hide the extraction failures that matter most in production

A vendor reports high accuracy, an engineer ships the API, and rows go missing in production. The tension comes from what the score counts, even when the calculation is correct. A headline average can hide how broad the schema was, whether every document returned valid output, and whether rows that were left out counted as errors.

ExtractBench identifies several failures that an aggregate can absorb without making them visible, including missing rows in long lists, selecting the wrong occurrence of a sparse fact, overfilling dense forms, and confusing similar dates, identifiers, or amounts. These errors matter because downstream agent logic depends on particular fields and records. A better framework uses a separate measure for each of these failure modes so engineers can see them.

What schema-guided extraction benchmarks actually measure, and why fixed-schema KIE benchmarks do not capture it

Traditional key information extraction benchmarks such as SROIE and DocILE use a fixed ontology. Their fields are predefined for a domain, so they test whether a system can locate known information within that schema. That distinction limits what a strong result can establish. Success on a fixed set of fields is not evidence that a system will faithfully populate an arbitrary enterprise schema.

ExtractBench defines schema-guided extraction as following a user-defined schema to produce correct output with source evidence as grounding metadata. UNIKIE-BENCH distinguishes constrained-category KIE, which uses scenario-predefined schemas, from open-category KIE, which extracts any explicitly present key information. Its constrained track spans 3 domains and 11 scenarios, with separate open-category documents. The schema itself is therefore part of the evaluation challenge.

ExtractBench is the only benchmark in its comparison table that jointly evaluates long-record completeness, real scans and handwriting, word- and page-level grounding, and measured cost. Those tests matter when the input is a new 159-field oil-and-gas disposal permit schema. A scorecard must establish whether the test asks the same kind of question as the production workload.

How schema breadth causes extraction quality to collapse, and what that means for enterprise document types

UNIKIE-BENCH reports substantial performance degradation across 15 state-of-the-art large multimodal models under diverse schema definitions, long-tail key fields, and complex layouts. Performance also differs markedly across document types and scenarios. Schema breadth is therefore not a minor detail attached to an accuracy result. It changes the difficulty of the task and the meaning of the score.

A model must resolve the visual layout, select the correct occurrence of each fact, and populate the required slots together. As the schema widens, there are more opportunities for a grounding error. With schemas as broad as those used by enterprises, a model can get fields wrong or return no valid output at all. Financial reporting, regulatory filings, and clinical documents expose this weakness in ways that narrow-schema tests do not.

For infrastructure such as Extend, the relevant production requirement is to handle the wide schemas used in financial services, healthcare, and logistics, or to make failure explicit through confidence signals. Partial JSON can be worse than an error when an agent silently treats it as complete. Tests must show whether field accuracy falls or the model fails to return valid, schema-conforming output. Otherwise, a score earned on a small schema offers false confidence about a much wider one.

Large-array row retention: the failure mode that value-F1 alone will not catch

ExtractBench finds that commercial vision-language models perform well on short documents but often truncate record lists on long ones. The missing rows are silently absent. This compounds the problem of schema breadth because a wide schema can also contain large arrays. A system must preserve both the fields within a record and the full set of records.

Value F1 computed only on returned rows cannot expose that omission. A model may get each row it returns right while leaving the downstream agent without all the evidence. ExtractBench addresses this by scoring order-insensitive value F1 across all expected rows, which directly penalizes omissions. The important distinction is not whether a benchmark uses F1, but whether its scoring includes the records the system failed to return.

LongArray-Extract, Extend's open-source benchmark, tests large-array row retention and gives separate scores for mean per-document extraction accuracy and run completion rate. On that benchmark, Extend MAX has 99.2% mean accuracy and completes every run. That result describes performance on a test designed to penalize both dropped rows and incorrect returned values. In a loan package, truncated income rows leave a decision based on incomplete evidence, just as dropped gene-level biomarker rows do in a sequencing report.

Grounding and source traceability as extraction quality dimensions, not audit add-ons

Completeness establishes whether the required records are present, but it does not establish where their values came from. Grounding supplies that missing connection through word-level bounding boxes and page citations. ExtractBench scores word-level box F1 and page-level F1 separately from value accuracy. Source traceability is therefore evaluated as part of the output accompanying the JSON.

A correct value can be attached to the wrong page, while an accurately located source can accompany a wrongly extracted value. Those are distinct failures, and a single value score cannot distinguish them. Without a traceable source, a correctly extracted value and a hallucinated value can look identical to the consuming agent.

Extend's extraction API returns per-field confidence scores and source citations with schema-defined JSON. Those outputs give an agent machine-readable evidence, but the citations still need to be checked for errors. A rigorous comparison scores the fidelity of that evidence independently. For financial audit, clinical data extraction, and KYC, that same evidence chain also meets a requirement for procurement.

How to pick a parsing API that will not fail on scanned invoices with irregular layouts

Extraction accuracy depends on the representation of the document available to the extraction step. If parsing misorders columns, detaches table cells from headers, or drops pages, extraction operates on damaged content. Recognizing every word is not enough when the relationships between words are wrong. A parser can preserve the text while assigning a cell to the wrong row, allowing a structural error to propagate into the extracted fields.

Parsing benchmarks and extraction benchmarks consequently answer different questions. Parsing evaluation tests layout, reading order, table structure, and OCR fidelity, while extraction evaluation tests whether schema fields are correctly populated. Merged cells, spanning headers, and rows split across pages expose differences that clean single-page invoices do not. For scanned invoices and multi-column documents, these are accuracy dependencies.

RealDoc-Bench evaluates parsing through field-level document Q&A accuracy and layout-adjusted F1 as separate measures. On that benchmark, Extend Performance Parse has scores of 95.7% for field-level Q&A accuracy and 0.847 for layout-adjusted F1. Keeping these parsing measures distinct from extraction results gives engineers a way to locate the source of a regression. A combined end-to-end score can show that quality fell without showing which layer failed.

Cost per page as a required benchmark dimension, not a separate procurement question

ExtractBench places LlamaExtract Agentic Plus first on its value accuracy and two grounding metrics, with higher accuracy than the coding agents at lower measured cost per page. That result applies within the benchmark's tested comparison. The useful comparison is what a system delivers at the measured cost, rather than accuracy without its operating conditions.

ExtractBench also finds that coding agents retain higher accuracy on long-record documents than commercial vision-language models, but at materially higher cost per page. An accuracy-only table hides that tradeoff. Engineers need the scorecard to show the cost of better record retention so they can judge whether it is worth paying for. For loan origination at scale or clinical processing across millions of records, small differences in cost per page add up.

A configurable response is available through Extend's light and performance extraction modes. They let teams process large volumes of straightforward pages at a lower cost and save heavier processing for complex documents. This design choice lets teams manage the tradeoff according to their workload. The benchmark must report cost at the accuracy level shown so teams can evaluate the result at their actual volume.

Confidence calibration and field-level scoring as the operational layer that turns benchmark accuracy into production reliability

Flatiron Health uses Extend's Split and Extract APIs to turn dense, variable next-generation sequencing reports into structured biomarker data across hundreds of genes. In that workload, field-level confidence and human review routing are prerequisites because document-level correctness is too coarse to govern individual outputs. Determining which extracted fields can be trusted and which need review requires more than a benchmark average.

Field-level scoring shows which field types have errors across a set of documents. Confidence calibration then addresses whether the system's confidence is a reliable guide to those errors. High confidence on wrong extractions makes the threshold unreliable, so engineers cannot tell which results to trust and which need review. A confidence score has operational value only when its relationship to correctness is dependable.

Review queues should focus on fields that fall below reliable accuracy thresholds, sparing teams from having to review every document by hand. Extend integrates built-in evals, per-field confidence scores, and human review to support that control and catch regressions before deployment. The evaluation requirement remains separate from the availability of the feature. A production scorecard must show per-field accuracy and, where possible, calibration metrics so engineers can use benchmark results to decide what needs review.

What a rigorous extraction API scorecard must include: the evaluation dimensions that separate production-ready benchmarks from marketing tables

A rigorous scorecard begins with schema breadth, including the field count and schema complexity actually tested. It gives completion rate its own score, the fraction of documents that return valid, schema-conforming output, so scores based only on completed runs cannot hide failures. It also penalizes omitted array rows explicitly. These measures distinguish a system that returns correct fragments from one that completes the requested extraction.

Field-level breakdowns must expose the fields that need remediation or review, with confidence calibration reported where possible. Grounding fidelity must separate word- and page-level citation accuracy from value accuracy because correct values do not prove correct attribution. Parsing metrics must remain distinct from extraction metrics so layout damage can be diagnosed upstream. Cost per page must correspond to the reported accuracy, making the operating tradeoff visible rather than leaving it to a separate procurement exercise.

Extend's open-source benchmark suite includes RealDoc-Bench, LongArray-Extract, and PoliTax Split to show the different ways document processing can fail, which a single score can hide. RealDoc-Bench and LongArray-Extract provide independently reproducible evaluations for parsing and extraction, and engineers can run evaluations on their own document corpus. ExtractBench's HuggingFace and GitHub releases likewise support scrutiny beyond a vendor table. Reproducibility makes every other dimension trustworthy, so the final requirement is open datasets, published evaluation code, and an independently runnable harness.