Stateful Document Processing for Multi-Page Reasoning Tasks

Multi-page document reasoning fails in production for a specific, fixable reason: most pipelines still treat a document as a single static input rather than something a model can search, revisit, and reason about over time. The fix is an external architecture that manages state on the model's behalf, one that lives outside the model and survives across inference calls. That distinction, state managed outside the model versus state crammed into a single prompt, is the entire argument of what follows, and getting it backwards is the single most expensive mistake a team can make when it buys or builds one of these systems.

Why multi-page document reasoning keeps failing in production

The failure isn't a capability problem. Long contracts, reports running several hundred pages, filings where a number on an early page quietly contradicts the number many pages later: none of this defeats the model because the model is dumb. It defeats pipelines that were never built to let a model navigate a document the way a person would, by checking a claim, jotting it down, then going back to see if the next section agrees.

Draw the line early. A stateless pipeline hands the model a document, or some chunk of one, and waits for a single answer back. A stateful pipeline lets the model search on its own terms, read selectively, hold onto what it found, change its mind, and act on evidence built up over several steps. Vendors sell these as the same category of product. They are not, and treating them as interchangeable is the first mistake most procurement decisions make.

The context-window explanation is a convenient distraction, because it lets teams believe the fix is just a bigger window or a better model. Even when a document fits entirely inside the model's window, accuracy still drops, because the relevant evidence is buried among far more irrelevant text. The real bottleneck isn't how many tokens a model can hold. It's whether the interface around the model lets it find the right evidence, remember it, and use it, rather than stare at all of it at once.

Evidence gets scattered across pages and across modalities, text in one place, a table or chart in another. Values conflict and need reconciling rather than just retrieving. Large, messy archives bury the one relevant paragraph among hundreds of irrelevant ones, and multi-hop questions require stitching together findings from sections nowhere near each other in the document. Each of these is a state-management problem, not a reading-comprehension problem, and no amount of model scaling fixes a state-management problem.

Where stateless pipelines break

The dominant shape in production today is conventional retrieval-augmented generation: embed the document into chunks, retrieve a fixed set of pages based on the query, feed those pages to a vision-language model, generate an answer in a single pass. It's a reasonable design for simple lookups, a single page, a single field, evidence that isn't ambiguous. The retriever grabs the right chunk, one pass is enough, and everyone moves on.

The trouble starts once the question requires more than one pass. Retrieval in this setup is static: the model can't decide, mid-task, that it needs to look somewhere else based on what it just read. There's no memory carried from one retrieval to the next, so evidence pulled from page 12 simply doesn't exist anymore by the time the model reasons about page 87. Each call stands alone, answering as if the previous call never happened.

Worse, as a pipeline runs more iterations trying to compensate, everything gets dumped into one growing context: retrieval traces, half-finished reasoning, prior observations, all piling on top of each other until the evidence that actually answers the question gets diluted by noise. A paper (arXiv:2606.05749) calls this the "monolithic context stream," and proposes replacing it with a three-stage loop covered in the next section. Under this kind of pressure, multi-hop reasoning fails first, because the agent can no longer connect two evidence fragments pulled during separate, disconnected steps.

Systems like M3DocRAG, DocAgent, and SimpleDoc share this same limitation: none of them give the model a way to update what it knows partway through a task. They retrieve, they answer, and whatever the model learned along the way evaporates. That's the entire reason multi-hop questions break these systems specifically, while single-hop lookups sail through untouched.

The mutable-state harness: what DocAtlas demonstrates about external state management

DocAtlas takes a different starting position: long-document understanding is a mutable-state interaction to manage. That's a real shift in framing. Instead of trying to retrieve the perfect chunk on the first try, the system assumes the model will need to search, backtrack, and revise, and builds the scaffolding for that up front.

The harness sits outside the model, as a separate environment deciding what document information gets searched, read, stored, reviewed, and shown at each step. It exposes four tool primitives. Search runs over a visual-aware hierarchical tree, enabling targeted retrieval across the document's structure. Read gives the model selective, page-level access rather than dumping the whole document in at once. Note-taking gives the model grounded memory, meaning evidence gets stored along with where it came from, so a later step can check the source. Review lets the agent look back over what it's already found, checking the accumulated evidence before producing a response.

The design principle underneath all four holds the context budget fixed no matter how long the document is; feeding the model more tokens does not solve the difficulty of handling documents spanning hundreds of pages, and discipline about which tokens make it into the model's view is produced by that principle, as the four cases show. Feeding the model more tokens does not solve the difficulty of handling documents spanning hundreds of pages. Discipline about which tokens make it into the model's view does, and that discipline is what most vendor pitches skip past when they lead with context-window size as the headline spec.

MARDoc's three-agent memory architecture as a complementary pattern

MARDoc (arXiv:2606.05749) attacks the same monolithic-context problem from a different angle: split the agent into three roles, each with a narrow job, instead of asking one model to search, reason, and remember all at once.

The Explorer handles multi-granularity multimodal retrieval, deciding not just what to look at but at what resolution: a full page, a region, or a specific figure. The Refiner takes the raw, messy trace of what the Explorer found and distills it into structured evidence memory, stripping out noise before it reaches the reasoning step. The Reflector checks whether what's been gathered so far is actually enough to answer the question, and if it isn't, sends the loop back around with specific feedback about what's missing.

The memory layer does the real work here. Instead of every agent reading through a full accumulated history of everything that happened, the three roles share a structured memory that updates as the task goes, and that design choice is what keeps context pollution from creeping back in through the side door. Compare this to the more common setup, where every retrieval step just appends onto one long, growing context: retrieval scaffolding accumulated across multiple turns buries the fact that actually answers the question, and no one needed to keep that scaffolding around.

The Refiner role generalizes past MARDoc specifically. It's a structured compression step, and nothing about it requires it to live inside the model's behavior. A production system can build it as its own service boundary, a separate stage in the pipeline with its own inputs, outputs, and failure modes to monitor.

The four state components a production harness must manage

From the DocAtlas architecture, four components emerge as things that have to live outside the model, not inside a prompt. Skipping any one of these reverts the harness to a stateless pipeline with extra steps bolted on, no matter how agentic the marketing copy sounds.

A search index with retrieval that improves as it goes returns different results as more is found, unlike a static embedding index returning the same ranked list regardless of what's already been found. Visual-aware indexing matters here specifically, because documents with tables, charts, and figures carry layout and spatial information that text-only retrieval does not capture.

A selective read layer lets the model request specific pages or regions instead of receiving whole documents by default. This enforces the fixed context budget mentioned above, and it's the mechanism keeping attention from getting diluted across irrelevant content.

A grounded note store writes down every piece of evidence with its source and page attached. Notes without that attribution recreate the exact reconciliation problem stateless pipelines already have: two conflicting numbers with no way to tell which page each one came from.

A review step checks whether the evidence gathered so far is enough before the system generates an answer. This is a system-level responsibility, not something to leave to the model. Asking the model to self-police reliably here is asking for trouble.

None of these four hold up as prompt engineering. They need durable data structures and tool interfaces that persist across separate inference calls, not instructions typed into a system prompt and hoped for. And all four depend on a layer that comes before any of them: the document has to be parsed, segmented, and classified correctly first. Get the reading order wrong, or misread the layout, and the search index and note store are corrupted before the model reasons about anything. A stateful harness built on bad parsing doesn't average out its errors over multiple iterations. It compounds them, step after step, until the final answer is confidently wrong.

The stateful reasoning gap understated by benchmark scores on static datasets

The AGORA benchmark found accuracy on authentic workplace documents tops out around 59.4%, with per-domain rankings shuffling substantially compared to aggregate leaderboard rankings. Splitting results out by document type reveals real weaknesses hiding inside systems that look strong on average, and that gap between leaderboard and reality is the whole reason static benchmarks mislead buyers.

DocScope pushes the evaluation further by framing long-document question answering as a structured reasoning trajectory in which the model has to produce the evidence pages it used, the specific regions within them, the supporting statements, and only then the final answer. That's a meaningfully higher bar than end-to-end accuracy, because end-to-end accuracy can't tell you where in the chain a system failed, only that it failed somewhere.

Every vendor in this space claims accuracy north of 95%, and most of those numbers come from general-purpose benchmarks that test reasoning and code generation, a different measure than a model's ability to pull a line-item total off a scanned invoice with a coffee stain across the corner. The number that actually matters in production is what the system does once its own confidence drops below threshold: does it escalate, does it flag, or does it guess and move on anyway.

The best model tested scored 96.50% on clean digital invoices, 92.71% on scanned invoices, and 87.46% on scanned receipts. Same model, same prompt, nearly a nine-point swing based purely on document quality. A stateful pipeline runs into this variance at every single retrieval step along the way, not once at the end, so the errors have far more opportunities to compound than a single-pass system ever would.

Document parsing quality's effect on stateful reasoning

Reading order is where errors start. If the hierarchical tree a document-processing harness builds gets constructed from a document with garbled reading order or missed layout elements, evidence gets routed to the wrong place at every step afterward, and the error doesn't stay contained. It compounds with each iteration of the loop, the same way a bad parse compounds through the note store described earlier.

Vision-language models change the baseline in a real way here. OCR reads left to right and spits out a wall of text with no sense of layout. VLMs interpret layout, spatial relationships, and context together, which matters enormously on anything with tables or multi-column structure. Across a benchmark of four VLMs, accuracy on diverse document types ranged from 42% to 67%, a meaningful step up over OCR on complex layouts, even if the ceiling is still far from perfect.

Fraunhofer IAIS found the same three-tier quality spread for extraction, and inconsistent document formatting produces that same spread in parsing, visible in the same tier breakdown. A model performing well on clean digital PDFs degrades on scanned pages, or other degraded document conditions, and a stateful pipeline sitting on top of that degraded parse carries the errors straight into its note store and its retrieval index. A general-purpose foundation model alone is not a document processing system, no matter how capable it tests on generic benchmarks. The surrounding harness, the layout detection, the reading-order logic, the confidence routing, produces the accuracy a team actually gets far more than which frontier model powers it, and vendors who lead with model choice over harness design are selling the wrong variable.

What this architecture looks like in production document pipelines

Put together, a production stateful pipeline has a recognizable shape. Ingestion handles parsing, layout detection, and reading order, and splits the document by logical unit rather than by an arbitrary fixed page count. Classification comes next, because document type determines which retrieval and extraction strategy applies, and getting this step wrong sends the whole harness down the wrong tool-routing path from the start.

The external state store underlies both of those: structured memory with source attribution, something the agent can query at inference time. This has to be a durable data service, not something living and dying inside a single context window. On top of the state store sits a tool-call interface, search, read, note, and review exposed as endpoints the agent can call, with the harness deciding what comes back and enforcing the context budget on every call.

A sufficiency gate scores the accumulated evidence before the system generates anything, then decides to keep iterating, escalate to a human, or commit to an answer. Workflow orchestration ties the whole loop together, action followed by observation followed by the next action, and exception handling, retries, and escalation rules have to live there too.

Industry research on one accounts payable team found manual review rates dropped from 40% to 4% after a move to this kind of agentic approach, because the system could resolve far more edge cases on its own before handing anything to a human, even though extraction did not become flawless. That's the sufficiency gate working as intended: fewer things need a person's eyes because the system got honest about which cases it can handle alone.

None of this is separable from security. A stateful harness accumulates sensitive document evidence in external stores by design, and those stores need to meet the same compliance bar as everything else touching the data: SOC 2, HIPAA, GDPR, zero data retention where required, self-hosted deployment where the use case demands it. A state store holding extracted contract terms or patient records is not a lower-stakes system just because it sits behind the model instead of in front of it.

Verification steps before building or buying a stateful document harness

Ask first whether the system actually externalizes state or just appends retrieval results onto a growing context and calls it agentic. Most products marketed as agentic document AI right now are the second thing wearing the first thing's name, so that distinction should be insisted on before any contract gets signed. A pipeline that just appends will hit the same context-pollution wall described earlier, just a few steps later than the stateless version it replaced.

Check whether memory carries source attribution. Evidence stored without a page and region attached can't support a real review step, and an answer that can't point back to where it came from isn't auditable, no matter how confident it sounds.

Confirm the sufficiency gate is an actual, explicit decision point, distinct from a fixed number of retrieval steps the system always runs before generating an answer regardless of whether the evidence gathered is enough. A harness that can't tell the difference between having found the answer and having simply run out of steps will hand back polished, wrong answers with the same confidence as correct ones, and that failure mode is far harder to catch downstream than an honest, unglamorous "not sure."