Reading Order Detection in Multi-Column Document Layouts

PDFs store drawing instructions, not sentences. Every glyph on a page gets an (x, y) coordinate and nothing more; nothing in the file format says "this word comes after that one." Reading order has to be rebuilt from geometry alone, and geometry alone is often not enough. That gap is where multi-column documents fall apart, and most pipelines still underestimate how badly.

When a full-width heading or a figure spanning both columns crosses the gutter, the whitespace projection that XY-Cut depends on never finds a clean vertical cut, so the algorithm fails to separate the two reading streams at all. Modern pipelines work around this by running a pre-masking step first, suppressing cross-gutter elements before projection starts, so the gutter gap becomes visible again before any ordering is attempted. The graph-based approach from 2026 sidesteps the problem differently: it scores candidate reading sequences by causal language model likelihood and BERT next-sentence prediction rather than pixel distance, so the spanning element is treated as its own region and the column streams are ordered by whether one line plausibly follows another in meaning. Both strategies share the same underlying discipline: separate the streams first, before any intra-stream or inter-stream ordering begins, because getting the stream boundaries wrong at layer one propagates through everything stacked above it.

What makes multi-column layout a structurally distinct problem

Single-column pages are forgiving. Sort every text element top to bottom, left to right, and the output usually comes out right, because there's only one stream of content to sort in the first place. Try that same coordinate sort on a two-column page and it reads partway down column one, jumps to column two because a line there happens to sit at a similar y-coordinate, jumps back, and stitches together a paragraph that means nothing to anyone, human or model.

Real documents make it worse. Academic papers carry marginalia and side notes that form their own vertical stream, separate from the main columns. Historical and legal texts use interlinear glosses that bridge two columns at once. Figure captions get anchored inside the gutter, the strip of whitespace meant to keep columns cleanly apart, and full-width headings interrupt the grid entirely, resetting the layout mid-page. Each of these makes the gutter disappear at exactly the moment most layout methods need it most, since the gutter is the one clean signal they lean on.

Geometry alone can't settle the question, and pretending otherwise is where most pipelines go wrong. When two reading streams overlap in horizontal range, or an element straddles the line between them, position tells you nothing about which stream a word belongs to. Get that boundary wrong once and the error doesn't stay put: a misjudged column split can duplicate a sentence, drop it, or scramble the token order feeding straight into whatever model reads the output next.

How XY-Cut works and where it reliably breaks

XY-Cut is the classical answer, and it earns its place: fast, easy to read, easy to debug, no training data needed. The algorithm slices a page recursively along horizontal and vertical whitespace projections, cutting at the deepest gap it can find, until it builds a binary tree of regions. Reading order falls out of a simple walk through that tree.

When a gutter runs unbroken between columns, XY-Cut works cleanly and costs nothing, needing no GPU and no labeled data. The trouble starts the moment something crosses that gutter. A wide gloss line, a figure spanning both columns, a horizontal rule running the width of the page: any of these means the whitespace projection never hits pure background across the gap, so the algorithm never finds the cut it needs.

XY-Cut works off pixel-density projections with zero sense of what's actually on the page, so it can't tell a decorative rule from an actual column boundary. Nested text boxes, non-rectangular regions, and content wrapping across a page break all sit beyond its reach. That ceiling is well known, which is exactly why the method still gets used: everyone knows where it fails, so nobody's surprised when it does.

Why early deep learning approaches traded one problem for another

LayoutReader was one of the first serious attempts to swap geometric heuristics for a learned model, pairing visual and positional features in a multimodal network trained to predict reading order directly. Building that training set took real manual labeling, and running it costs meaningfully more than a projection-based cut, which adds up fast once a pipeline has to process documents at volume.

The bigger problem showed up in evaluation, and the result should have killed the approach outright. On the multi-column subset of OmniDocBench, measured by macro edge accuracy on a reading-order graph, LayoutReader scored 25%, while XY-Cut, the decades-old heuristic it was supposed to replace, scored 75% on that same subset. A model built and trained specifically to learn reading order lost to a whitespace projection, on exactly the layouts it was designed to handle. Take that as the verdict on the whole strategy, not just one model: bolting a neural network onto weak positional signal doesn't buy structural understanding, it just buys a more expensive way to get the order wrong.

Adding parameters doesn't hand a model the geometric sense that column layouts actually need. Wrap-around structure punishes a model that never learned to represent columns as columns in the first place, and no amount of extra training data fixes a representation that's missing from the start.

The three technical layers any reading order system must address

Reading order recovery splits into three layers, and most systems fail at one specific layer, not all three at once. Knowing which layer a given method targets tells you almost everything about where it will break, and this is the diagnostic question worth asking before adopting any tool: which layer does it fail at.

Layer one is stream separation: sorting out which regions belong to which independent reading channel, columns, marginalia, captions, before any ordering starts. Layer two is intra-stream sequencing, ordering elements inside a single stream, usually top to bottom but easily thrown off by wrapped lines or oddly shaped regions. Layer three is inter-stream ordering, deciding which stream gets read first when two sit side by side.

Classical methods like XY-Cut fail at layer one, and once the streams never get separated correctly, everything stacked on top inherits that mistake. Neural models tend to fail at layer three on unfamiliar layouts instead, because they carry no explicit notion of stream identity and end up treating spatial closeness as if it meant reading adjacency. Fix only one of these three layers and a system will post local gains on a benchmark, then fall apart the moment it meets a document outside its comfort zone.

How do modern methods handle gutters that disappear behind figures or full-width headings?

The methods gaining ground now share one instinct: separate the streams first, with geometry or masking or clustering, and only then bring in semantic or visual signal to order within and between them. That sequencing, separate first and order second, is the actual lesson buried in every result below, and it's the one design principle worth copying regardless of which specific tool a team picks.

XY-Cut++, from 2026, runs a three-stage pipeline: a pre-mask step that suppresses cross-gutter elements before projection even starts, multi-granularity segmentation, then cross-modal matching. It posts state-of-the-art results on DocBench-100 and OmniDocBench while staying CPU-viable, which matters to anyone running this at scale without a GPU fleet sitting idle.

A training-free, graph-based approach from 2026 takes a different route. Each OCR text line becomes a node; edges between nodes get scored by a weighted mix of causal language model likelihood and BERT next-sentence prediction, and the full reading order comes out as a degree-constrained directed path through that graph, no fine-tuning involved. On the OmniDocBench multi-column subset it hits 88% macro edge accuracy, ahead of XY-Cut's 75% and far past LayoutReader's 25%. That gap says something specific: what actually recovers difficult wrap-around order is judging whether one line plausibly follows another in meaning, not measuring the pixel gap between them. Weigh methods that reason about meaning more heavily than methods that only measure distance; the numbers above aren't close.

FocalOrder, also 2026, is a compact model tuned with preference optimization aimed squarely at the structural ambiguity built into document layouts, and it beats larger general-purpose vision-language models on OmniDocBench v1.0 and Comp-HRDoc, size for size, a result worth sitting with rather than skimming past. A 2024 paper in Pattern Recognition takes yet another angle, framing reading order as pairwise relation prediction between text instances using visual, semantic, and adaptively generated positional features, rather than treating it as a sequence to generate from scratch.

RT-DocLayout, from 2026, folds classification, detection, pixel-level segmentation, and reading order prediction into one 33-million-parameter model running a single forward pass, sidestepping the error propagation that chained single-task pipelines suffer from. For simpler multi-column cases, a lighter geometric trick used in NovaLAD-style pipelines does surprisingly well: normalize the x-centers of bounding boxes, run DBSCAN to cluster columns, then sort each cluster by y-center. Nothing fancy about it, but it makes column assignment explicit and something you can actually check, which counts for a lot when something breaks downstream and someone has to find out why.

How VLM-based parsers absorb reading order implicitly — and what they still get wrong

Traditional OCR pipelines chain separate models for detection, recognition, and layout analysis, and every stage is a place error can sneak in before reading order even gets attempted. Vision-language model parsers collapse that chain: feed in a document image, get structured markdown or HTML out, one pass. Reading order stops being a separate module; it's baked into the order the model spits out tokens in.

The field moved on this fast, and October 2025 alone saw six major open-source VLM-based OCR models released, which says something about how quickly this approach replaced the staged pipeline as the default starting point for new projects.

What VLMs gain is real: no error piling up between stages, complex tables and equations handled in the same forward pass as multi-column body text. What they lose is the ability to be checked, and that trade should worry anyone running these in production. When a VLM gets column order wrong, there's no intermediate layout map to open and fix; the mistake sits sealed inside one opaque generation step. OmniDocBench results confirm the cost: a measurable accuracy drop on multi-column and complex layouts relative to single-column pages, even for the strongest models, which suggests implicit ordering has a real ceiling once layout complexity climbs. One study found only 64% accuracy across 200 annotated pages using a conventional staged OCR pipeline; that's the floor modern methods are working to clear, and it's not a high bar to begin with.

PaddleOCR-VL takes a middle path, and it's the more defensible design of the two extremes: an explicit Document Layout Analysis stage, PP-DocLayoutV2, feeds into a lightweight pointer network that handles reading order. It combines the checkability of a staged pipeline with the representational strength of a learned layout model, rather than betting everything on implicit ordering buried in one generation pass. Anyone choosing between a pure VLM parser and a hybrid like this should default to the hybrid whenever the documents in question carry real column complexity; the ability to inspect where reading order went wrong is worth more than the marginal convenience of a single forward pass.

What current benchmarks measure and what they still miss for multi-column documents

OmniDocBench, from CVPR 2025, covers 981 samples across academic papers, textbooks, and handwritten notes, annotating both layout bounding boxes and reading order, scored with Normalized Edit Distance for text, TEDS for tables, and CDM for formulas. Multi-column documents show up in it, but not as their own evaluation axis, so a model can post a strong overall score while quietly failing on multi-column pages specifically. GLM-OCR and PaddleOCR-VL-1.5 have both cleared 94% on this benchmark, and that alone should raise a flag: a benchmark everyone's already clearing 94% on has stopped discriminating at the difficulty level that matters in production.

olmOCR-bench, from Allen AI in 2025, takes a stricter line: hundreds of PDF files, scored pass or fail on discrete factual properties, including natural reading order and table cell position. It doesn't forgive minor formatting drift, and it cares whether specific facts came out right, not whether the overall output looks polished. Multi-column is one of eight category splits, tested but not the design center.

The structural gap runs across all of these: no major public benchmark treats reading order recovery on genuinely adversarial multi-column layouts as its primary target. DocBench-100, introduced alongside XY-Cut++, comes closest, but it's still a research benchmark, not something built around the failure modes engineers actually hit in production. A system with strong OmniDocBench numbers can still fail on an ordinary two-column journal article with one figure spanning both columns. Match benchmark choice to the real document distribution a system will face, and favor benchmarks built around specific failure modes, row-drop, stream-separation errors, over aggregate scores that average away the exact failures that matter.

Where reading order failures surface in production document pipelines

Semantic incoherence shows up first, and it's the hardest to trace back to its cause. A language model reading mis-ordered text blends content from two different columns into one answer, and the error looks like a reasoning failure when the actual cause sits one layer up, in layout.

Content gets duplicated or dropped next. Column-crossing during extraction can pull the same sentence from both streams, or skip a passage entirely because the extraction logic assumes it's already been captured elsewhere. Tables and figures get misattributed too: a caption gets read as body text, or a figure lands in the middle of a paragraph it has nothing to do with, and any structured extraction running downstream inherits that confusion whole.

The sharpest version of this shows up in agentic workflows, and it's the one worth losing sleep over. A document agent doesn't just extract and hand off; it reads, decides, and acts, often on something with real money or real consequences attached. A silent reading-order error gives that agent no chance to catch itself, because from where it sits, the input looked coherent enough to act on. Static extraction produces a formatting artifact when column order breaks; an agent acting on that same broken order produces a wrong transaction instead, and that's a different category of failure entirely, one that shows up in a ledger or a filing rather than in a QA report. The domains where this bites hardest share a structural trait: two-column clinical trial papers in healthcare, multi-column prospectus documents in financial services, legal contracts carrying marginal annotations, carrier rate sheets with parallel pricing columns in logistics. Each depends on exact column correspondence, and each is exactly the layout that breaks naive ordering.

What engineers should look for when evaluating or building a reading order layer

Test against the actual document distribution a system will face in production, not a benchmark's idea of a representative sample. A pipeline tuned on academic papers can degrade sharply the moment it meets a newsletter layout or a dense insurance table, and there's no substitute for running it against the real thing.

Demand visibility into stream separation, and treat any vendor who can't provide it as a red flag rather than a minor gap. If a system can't show which regions it believes are independent columns, tracking down why it got something wrong turns into guesswork. Push specifically on spanning elements, full-width headings and figures crossing column boundaries, because these are exactly the cases where projection methods and VLMs tend to diverge most sharply from correct output. Treat reading order as its own metric, not something that rides along with text accuracy for free; a system can get every word right and still fail the task completely if it gets the order wrong.

Ask directly for confidence scoring on layout decisions. Knowing when a system is uncertain about a column assignment means a human reviewer gets routed to exactly the documents that need a second look, instead of spot-checking at random and hoping the sample happens to catch the failures. Weigh the architecture's error-propagation profile before committing to one: chained single-task pipelines accumulate mistakes before reading order is even attempted, while unified architectures cut that propagation down at the cost of some modularity. Make that tradeoff on purpose, before launch, not after a two-column prospectus slips through and someone downstream has to explain why the numbers don't match.

For teams running complex, multi-column documents at volume, where accuracy has to clear what general-purpose models deliver out of the box, purpose-built document infrastructure, benchmarked openly against tests like RealDoc-Bench and LongArray-Extract, is worth the switch. The difference isn't marginal: a pipeline that fails silently on a two-column prospectus versus one that flags the page for review before the error ever reaches a downstream agent.