Agentic Document Workflows With Tool-Calling LLMs

Document AI is moving from a back-office clerical task to something closer to a data layer, and that shift changes what "accuracy" even means. Agentic document workflows only become production-ready when a tool-calling LLM gets paired with a structured system harness, one where parsing, classification, extraction, and validation each run as a discrete, callable tool rather than getting buried inside a single giant prompt. That distinction is the entire argument. Get it backwards, and you end up with a system that scores well on a benchmark and falls apart on the first invoice batch that doesn't match the layout it was trained on.

The old model of document automation ran on template-based OCR and rigid intelligent document processing (IDP) pipelines. Those systems work fine until a vendor changes their invoice layout, at which point someone has to go recalibrate the template by hand. Worse, they output flat text: a wall of characters with no structure, no confidence signal, nothing an autonomous agent downstream can act on without a human squinting at it first.

Enterprises now need something else entirely: JSON, clean Markdown, semantic annotations that an LLM or an agent can consume directly. Stanford HAI's 2026 AI Index put organizational AI adoption at 88%. AI has moved out of pilot mode and into production-scale deployment across the enterprise. Once agents are acting on document data at that scale, document ingestion stops being a clerical chore. It decides whether the agent acts correctly.

A document AI agent reads, decides, acts inside a business system, and can show what it did. A classifier that matches a template and spits out fields does none of that. Production readiness doesn't come from making the underlying LLM smarter in isolation, and teams that chase a bigger model as the fix are solving the wrong problem. It comes from giving that LLM a structured set of callable tools, one per pipeline stage, each with a narrow job and a defined contract.

What tool-calling means in the context of a document pipeline

The agent gets a set of external tool descriptions, and it has to decide whether to invoke one, when to invoke it, and with what parameters. Each invocation is a structured action (Xu et al., arXiv:2603.22862).

Anthropic's guide on building effective agents draws a useful line between workflows, where an LLM gets orchestrated through predefined code paths, and agents, where the LLM directs its own process on the fly. Most production document systems sit somewhere in between: enough structure to be predictable, enough flexibility to handle the document that doesn't fit the template.

Documents fit that middle ground well, because a document pipeline has natural stage boundaries. Parse, classify, split, extract, validate: each one maps cleanly onto a tool call with defined inputs and outputs, no ambiguity about where one stage ends and the next begins.

Precision at the tool-call boundary matters more than most teams treat it. The DiffuAgent study (arXiv:2601.12979) found that diffusion-based LLMs fail in tool-calling settings because they can't hold onto symbolic precision: they violate strict JSON schemas and hallucinate API parameters that don't exist. Autoregressive LLMs don't share that failure mode, and that's a big part of why they remain the backbone of production tool-calling systems instead of diffusion architectures. If you're picking a model family for a document pipeline on anything other than this distinction, you're picking on the wrong basis.

There's a design detail that gets overlooked constantly: how you shape the syntax of the tool call itself. Tweaking that agent-computer interface, the exact structure and wording of the tool description, to match the quirks of a given model can produce large swings in performance. The interface does as much work as the tool behind it.

For the builder, the takeaway is concrete. Each pipeline stage needs a well-specified API contract: schema-validated inputs, typed outputs, defined error codes. Not a freeform prompt instruction tucked into a system message somewhere, hoping the model interprets it the same way twice.

The four orchestration patterns

Kulkarni & Kulkarni (arXiv:2603.22651, March 2026) ran a systematic benchmark of four multi-agent orchestration architectures across 10,000 SEC filings, 25 extraction field types, and five frontier and open-weight models. The results give architects an actual basis for choosing a pattern instead of guessing, and the choice most teams default to isn't the one the numbers support.

The sequential pipeline is the simplest: a fixed chain, context passed cumulatively from stage to stage, deterministic order, no parallelism. Latency grows in a straight line with the number of agents in the chain, and upstream errors compound downstream unchecked. Fine for simple, predictable document types. Poor for anything else.

Parallel fan-out with merge distributes subtasks at the same time, which suits latency-sensitive workloads where sections of a document don't depend on each other, a multi-section financial filing being the obvious case.

Hierarchical supervisor-worker puts a coordinator agent in charge of delegating to specialist agents underneath it. In the Kulkarni benchmark, this pattern hit 98.5% of the best observed F1 score while running at 60.7% of the cost of the top performer. That's the best cost-accuracy tradeoff on offer for production financial document processing, by a wide margin. It's the practical default, full stop.

The reflexive self-correcting loop, where agents check and revise their own output over multiple passes, posts the highest field-level F1 in the study at 0.943. It also costs 2.3x what the sequential baseline costs to get there. Paying that premium makes sense only where accuracy is the overriding constraint and the budget can absorb it, which rules out most enterprise document programs before the conversation even starts.

Set the numbers side by side: an F1 of 0.929 against the reflexive loop's 0.943, at $0.261 per document against $0.430, and the case for hierarchical architecture is hard to argue against. A three-cent-per-point accuracy gain doesn't justify nearly doubling the cost for most enterprise use cases, and any team that picks the reflexive loop by default is paying for accuracy it doesn't need.

Choosing a pattern also means inheriting its failure modes. The Kulkarni study catalogs twelve of them specific to multi-agent financial extraction, each with a prevalence rate tied to the architecture that produces it. The pattern is only the outer shell, though. What happens inside each individual stage, how parsing, classification, splitting, extraction, and validation actually get built as tools, is what decides whether the accuracy numbers survive contact with real production volume.

Mapping each document pipeline stage to a discrete callable tool

Each stage below should be read as a tool with a defined responsibility, an input schema, an output schema, and a known failure mode. Not a paragraph of instructions folded into one long system prompt.

Parsing turns raw document bytes into a structured intermediate form: clean Markdown or JSON carrying layout annotations, reading order, bounding boxes. Reading order and layout fidelity are the critical accuracy dimension here, because errors at this stage don't stay contained. A parser that scrambles the column order of a financial table corrupts every field extracted from that table later on. The mature 2026 architecture routes documents down differentiated paths inside this tool depending on document complexity, a design pattern addressed in recent agentic pipeline work (Fehlis et al., arXiv:2605.18818, 2026). The output contract has to be unambiguous, something the extraction tool downstream can consume without guessing, and schema validation has to happen right at the handoff, not later.

Classification identifies document type, sub-type, and structural variant: a 10-K versus a 10-Q versus an 8-K, a single-page invoice versus a multi-supplier invoice packet. Its output steers everything after it, which extraction schema applies, which splitting strategy runs, which validation rules fire, so getting classification wrong means the entire downstream chain inherits that error. A practical hybrid setup pairs a lightweight classifier for the easy majority with an LLM fallback for ambiguous cases, which keeps GPU-bound inference cost off the documents that don't need it.

Splitting breaks multi-document packets into individual logical units before extraction can run, a loan file containing an application, pay stubs, and tax returns being the standard example. Get a split wrong and fields from one document get attributed to another. Get a split wrong and fields from one document get attributed to another, a major compliance and accuracy risk. It's a compliance risk and an accuracy risk at the same time, and splitting performance tends to depend more on the surrounding system harness than on which frontier model powers it, because that harness produces the accuracy or the errors, regardless of which model sits underneath.

Extraction pulls a schema-defined set of fields with typed values attached, once a document has already been parsed, classified, and split. The Kulkarni study found GPT-4-class models landing in the 85 to 92% range on field-level accuracy for clean filings, a number that drops sharply once tables get complex. That gap is the clearest evidence that extraction isn't a problem model capability alone solves. Silently dropping rows in a large array extraction is one failure mode that deserves naming directly. A system that drops rows without flagging them hands back output that looks complete and is materially wrong, and any benchmark worth trusting has to penalize that specific failure rather than averaging it away. Extraction output should carry field-level confidence scores alongside the values.

Validation applies business rules, cross-field consistency checks, and confidence thresholds, flagging anything that needs a human look before it triggers a downstream action. The MADP production deployment (Gosmar et al., arXiv:2605.17159, May 2026) shows this validation layer working at scale in real-world document processing, with documents that fall below threshold routed to human review. Automation at that level is only safe to act on automatically because the validator is standing behind it. Confidence scoring and built-in evals are a core part of the product from the start. They're a product responsibility, and catching an error before it reaches a downstream agent is the entire difference between a pipeline that's reliable and one that quietly corrupts business data.

State management and error propagation across tool-call boundaries

In a sequential or hierarchical pipeline, each tool call inherits whatever state and whatever errors the previous call produced. Error propagation runs one direction only, downstream: an upstream mistake compounds silently in later stages instead of getting caught at the point where it happened.

What gets carried across each tool-call boundary is a design decision in its own right. The intermediate representation, the parsed document plus its metadata plus its confidence scores, needs to be explicit, versioned, and schema-validated at every handoff. Passing it as freeform text defeats the point of building discrete tools.

Fehlis et al. point to asynchronous processing as a core design decision for IO-bound steps in production. Separating GPU-bound inference from CPU-bound orchestration lets a microservice deployment process thousands of multi-page documents an hour without the most expensive compute stage becoming the bottleneck for everything else.

The twelve failure modes the Kulkarni study catalogs, each tied to a specific architecture and a specific prevalence rate, make a related point: teams need to design against named failure modes, not against a generic notion of "error handling" that treats every failure as interchangeable.

Reflexive self-correction runs at 2.3x the sequential baseline cost, so running every stage through a self-correction loop is a budget mistake. Trigger the loop only when a confidence score on a specific field drops below threshold, applying it selectively rather than as a blanket retry across everything that passes through.

Human review deserves the same reframing. Treated that way, the human-in-the-loop step is itself a tool call inside the pipeline, not an escape hatch bolted on for the cases the system couldn't handle, and the corrections it produces can feed back into the system to improve future automated runs.

Capacity planning has to account for how each architecture's accuracy and cost profile behaves under increasing production volume, and that behavior needs to be characterized before hitting the limits of a given pattern in production, not after.

How benchmarks expose pipeline-level accuracy versus model-level accuracy

OmniDocBench, presented at CVPR 2025, is the primary open benchmark for document parsing, covering text, tables, formulas, and reading order with fine-grained annotations, scored on a blend of Normalized Edit Distance, Tree Edit Distance-based Similarity, and Character Detection Matching. The leaderboard (arXiv:2603.27365) shows FalconOCR at 88.64 overall, DeepSeek OCR v2 at 87.66, and GPT 5.2 at 86.56, a tight spread that says the frontier is genuinely close on clean or semi-clean documents. OmniDocBench's coverage represents a narrower slice of document variance than what enterprise deployments actually encounter, and scores on it do not fully reflect real-world degradation conditions.

Real5-OmniDocBench tests something OmniDocBench doesn't: parsing robustness under five real-world conditions, including 3D distortion and uneven lighting, across 1,355 high-fidelity reconstructed images. Running current parsing systems through that gauntlet exposes vulnerabilities that the clean-document benchmarks never surface, and the gap between the two leaderboards is the whole point.

ParseBench (arXiv:2604.08538, April 2026) is the first benchmark built explicitly around enterprise document types, 2,078 documents with full coverage of tables, charts, formatting, and grounding. It closes a gap OmniDocBench leaves wide open.

Even the benchmarks themselves need scrutiny. Stanford HAI's 2026 AI Index found invalid question rates on evaluation sets ranging from 2% on some benchmarks to 42% on others, and that concern applies just as much to document AI accuracy claims resting on flawed evaluation data as it does anywhere else. A benchmark can be wrong before a model even touches it.

Real pipeline-level evaluation means testing on actual customer files, surfacing failure patterns at the level of individual fields rather than whole documents, and feeding corrections back into an optimization loop. An accuracy tool that skips those steps is measuring something, just not the thing that matters. A benchmark that doesn't penalize dropped rows in a large array extraction overstates accuracy on exactly the failure mode that does the most damage to a business, the same silent-drop problem in a different form.

Production infrastructure decisions: separation of concerns, scaling, and compliance

Fehlis et al. show what running classification, OCR, and LLM extraction as separate microservices actually buys a production deployment: independent horizontal scaling, with GPU-bound inference decoupled from CPU-bound orchestration. That separation is what makes processing thousands of multi-page documents an hour possible without the most expensive stage of the pipeline dragging everything else down.

Hybrid routing, sending the predictable majority of documents down a cheap deterministic path and reserving agentic LLM calls for the genuinely hard cases, is probably the single design decision with the greatest power to carry a document AI program through its first budget review. Running every document through the expensive path regardless of difficulty makes the unit economics stop making sense fast.

For regulated industries, on-premises deployment is a requirement. It's often a requirement, full stop. OnPrem.LLM (arXiv:2505.07672) demonstrates a full pipeline running entirely on-premises: format-flexible document loaders, automated OCR, table extraction, and a separate dual dense-plus-sparse retrieval store for RAG. Data that legally can't leave the building has a documented path to work with.

Financial services, healthcare, real estate, and logistics all carry compliance obligations, SOC 2, HIPAA, GDPR, PCI, that aren't optional line items. The infrastructure layer underneath a document pipeline has to support self-hosted deployment, zero data retention policies, and HIPAA business associate agreements where healthcare data is involved. Those requirements narrow which tools and vendors can legitimately sit inside the pipeline before a single accuracy number even enters the conversation.

The orchestrating LLM issues its tool calls against APIs that need to be well specified from the start, and typed SDKs matter as much as model choice does here. Skipping that step means a team ends up debugging schema mismatches in production instead of catching them in a type checker, a costly place to find out an API contract was never really enforced.