Clinical Document Processing for AI-Assisted Prior Authorization

Prior authorization is, before anything else, a document matching exercise. A request requires gathering diagnoses, clinical notes, labs, and imaging reports, then reconciling that evidence against payer-specific policy criteria, which are themselves documents full of thresholds and qualifying conditions. Treating this as an AI opportunity misses the order of operations: nothing downstream works unless the documents feeding into it are read correctly first. CMS-0057-F tightened the clock on top of that problem. Starting January 1, 2026, standard decisions must be made within 7 calendar days, down from 14, and expedited decisions within 72 hours. The document work that used to happen at a comfortable pace now has to happen fast enough to stay compliant.

Clinical documents in an AI-assisted PA pipeline

An AI-assisted PA system is a multi-stage document pipeline at its core, and every step that follows extraction, from criteria matching to submission to the final decision, depends entirely on what that first layer produced. Finally, it has to assemble a complete request, submit it, and write the resulting decision back into the patient record.

Automated PA platforms describe this as a four-stage sequence: determination, documentation assembly, submission, and status write-back. Extraction errors made at that stage do not stay contained there. They travel forward into the criteria match, and from there into the denial.

Why general-purpose models fail on PA document types

Clinical documents are harder to parse than the clean, well-formatted text that general-purpose language and vision models were primarily trained and tested against. Layouts vary by clinician and by EHR vendor. And many of the fields that matter most are numeric values that have to be reproduced exactly, not approximated. A model that gets the gist of a lab report right but transposes a decimal point has not produced a minor error. It has produced a submission that fails the criteria check.

The obvious objection is that frontier models keep improving, and the gaps described above will close as the models get better. Even the best-performing configuration resolves only a fraction of tasks, and performance falls further once the full workflow is compressed into a single session. That pattern points to something structural in how document-heavy, policy-dense tasks accumulate context, not something that scales away with a slightly larger model.

CHI-Bench and the ceiling for PA agents in realistic conditions

CHI-Bench is the first long-horizon benchmark built specifically to reflect real prior authorization conditions, and the gap it reveals between what vendors report and what agents accomplish is substantial. Presented at NeurIPS 2026 in the Evaluations & Datasets Track, it uses high-fidelity simulators across three domains: Provider Prior Authorization, Payer Utilization Management, and Care Management, each instantiated across MCP servers operating on patient, clinician, and insurer records.

The scale of the benchmark is part of what makes its results credible. Each trial runs an agent for 60 to 80 steps across four to six clinical stages, drawing on more than 200 MCP tools and a 1,279-document operations handbook that stands in for the policy material a real PA team would need to consult. Thirty different agent harness and model configurations were tested against this setup. The best overall resolution rate across all of them reached 28.0% of tasks, and no configuration cleared 20% under the benchmark's strict pass-cubed scoring. When the full workflow was compressed into a single continuous session, rather than split across stages, performance dropped to 3.8%. The top-scoring configuration, built on GL OpenAI Agents running GLM 5.1, reached 18.7% overall accuracy on PA subtasks as of May 2026.

The benchmark's authors attribute these failures to the combination of policy density, multi-role composition, and multilateral interactions baked into realistic PA workflows. Agents struggle most exactly where clinical document understanding and payer-criteria matching are hardest, the same terrain where the extraction-layer failures described earlier occur. CHI-Bench does not suggest that PA automation vendors are misrepresenting their results. It suggests that the conditions vendors test under are considerably more forgiving than the conditions a long-horizon, policy-rich, document-heavy PA workflow actually presents.

The specific document extraction failures that cause PA denials and delays

A handful of recurring extraction failures account for a large share of PA denials and delays, and each one is traceable to a specific point in the document processing pipeline. PA criteria for drugs, devices, and procedures often hinge on threshold values like A1c, eGFR, or tumor markers, and a model that misreads a decimal point or silently drops a row from a lab table will fail the criteria check without any visible error.

Reading-order errors in clinical notes create a different kind of damage. Payer-specific field mismatches round out the list. Each payer's criteria document functions as a structured form with its own field names and value formats, and extracting data against the wrong schema produces a submission that looks complete but is substantively wrong.

The research on AI-generated PA letters found that this administrative scaffolding, the field-level detail that criteria checklists actually depend on, is exactly where AI-generated submissions came in weakest, even in cases where the clinical narrative read well. That points to a mismatch in what these pipelines are optimized to produce. A system tuned for narrative quality, which describes most LLM-based tools in their default configuration, is tuned for the wrong output. Payer criteria checklists reward structured field accuracy, not prose fluency.

Why the document processing layer must be purpose-built

The failure modes above share a root cause: they are the predictable outcome of asking general-purpose models to serve as a document layer instead of building extraction infrastructure designed around the actual structure of clinical documents. A three-stage pipeline, moving from OCR to specialized clinical NLP to an LLM, exists because each stage is suited to a different part of the problem. OCR has to handle fax degradation, handwriting, and scan artifacts before any language model ever sees the text. Specialized clinical NLP models handle entity recognition, pulling out diagnoses, medications, and lab values, more reliably than a general LLM asked to do the same task through a prompt. Only after that upstream work has produced clean, structured input does an LLM add real value, reasoning over policy text and clinical narrative together.

One failure mode deserves particular attention because standard accuracy metrics do not catch it: a system that extracts a lab panel but silently drops values because the table layout was unusual. Generic accuracy scores often do not penalize this kind of omission the way they penalize an outright wrong answer, which makes it dangerous in production. A system built for this environment has to surface missing fields as explicit failures rather than letting them pass through as blanks or zeros that quietly clear a criteria check. Field-level confidence scoring lets the pipeline route uncertain extracted values to a human reviewer before submission to a payer, which is what separates a production system from a demo.

EHR integration does not solve this problem on its own, and in some ways it makes the problem more visible. EHR exports arrive as a mix of structured data, free-text notes, scanned attachments, and PDF-wrapped CCDA documents. The integration layer delivers those documents to the pipeline, but the extraction layer sitting behind the integration determines what the pipeline actually sees and how accurately it reads what it sees.

The "bot wars" dynamic when the document layer is weak

When the extraction layer is unreliable, AI-assisted PA does not eliminate administrative work. It moves that work into review cycles, appeals, and extra verification steps that raise the total cost of getting a single authorization through. Health plans and physician practices are now both deploying AI on their respective sides of the exchange, creating a dynamic where automated systems generate requests and automated systems respond to them with minimal human involvement in between. When a provider's submission is incomplete or inaccurate, that submission trips automated review flags on the payer side, and the result is more cycles between the two systems, not fewer.

This dynamic is strong evidence that document quality, not workflow design, is the binding constraint. The stages of PA automation that work reliably, status tracking, payer portal submission, write-back into the record, have not translated into the cost savings providers expected, and the reason traces back to the documentation assembly step that sits upstream of all three. The CHI-Bench results answer that objection directly: agents working on PA tasks do not fail at communication or routing. They fail specifically at the stages that involve gathering clinical evidence and interpreting payer policy, which are document problems, not coordination problems.

What compliance requirements add to clinical document processing infrastructure

Every point in a PA pipeline where protected health information passes through an AI system is a HIPAA-covered operation, and that covers the document processing infrastructure itself, not only the front-end application a user interacts with. A Business Associate Agreement is required for every vendor whose infrastructure touches PHI, which includes whatever service performs extraction and parsing, not just the platform a care team logs into. Training a model on PHI requires explicit authorization from the covered entity, and zero-data-retention policies represent the highest tier of compliance available, since they ensure patient data never becomes training data for a public foundation model.

State law is adding a further requirement that bears directly on document processing quality. Washington's E2SSB 5395 states that algorithms may be used to process and approve prior authorization requests but may not be used, without human review, to deny care based on a determination of medical necessity. That requirement does not apply to every adverse outcome a system might produce, but it does mean that for medical necessity denials specifically, a human has to review the basis for the decision. That in turn means the extraction layer has to produce outputs a human reviewer can actually check: which document a given value came from, which field it filled, and why it matched or failed to match a given criterion. An opaque confidence number does not meet that bar. Explainability and auditability of extracted evidence are compliance requirements built into the infrastructure, not features added for convenience.

Purpose-built clinical document processing infrastructure in practice

A document processing layer built for clinical PA work has to satisfy accuracy, auditability, compliance, and speed all at once, and that combination is what rules out a pipeline stitched together from prompts as a viable production architecture. Document classification and splitting come first: a PA packet typically bundles several document types together, a referral, a clinical note, a lab report, an imaging study, and the extraction layer has to correctly identify and separate each one before type-specific extraction can even begin. Layout-aware parsing follows, since lab panels, imaging reports, and clinical notes each follow their own structural conventions, and getting values in the correct reading order without dropping rows depends on recognizing those conventions rather than applying one generic template to everything. Field-level confidence scoring has to run throughout, flagging uncertain values for human review before they reach a payer. Extracted data has to map onto the specific field schema of each payer's criteria document rather than a generic clinical ontology, and when a required field simply is not present in the source documents, the system has to surface that as a gap someone needs to address.

On the compliance side, enterprise deployment in healthcare settings calls for HIPAA BAA availability, SOC 2 certification, a zero data retention policy, and self-hosted or bring-your-own-cloud deployment options for organizations handling PHI at scale. Speed matters here in a very concrete way: CMS-0057-F's timelines are already in effect, so a document processing layer that takes months to integrate and validate leaves an organization operating out of compliance for the duration of that rollout. The kind of platform that fits this picture offers APIs for parsing, extraction, classification, and splitting, with SDKs in standard languages, open-source benchmarks teams can use to validate accuracy themselves, and a compliance posture covering HIPAA BAA, SOC 2, GDPR, and self-hosted deployment. That is the shape of infrastructure the rest of a PA workflow can actually depend on.

Evaluating clinical document processing accuracy before committing to a PA pipeline

The most consequential decision in building or buying an AI-assisted PA pipeline is how the document processing layer gets evaluated, and the only evaluation that means anything runs on production-representative documents, not curated demo sets. Testing on an organization's actual document mix is the starting point: PA packets include degraded faxes, handwritten annotations, multi-page bundled PDFs, and EHR exports that vary in format from one referring source to the next, and accuracy measured on clean PDFs says very little about accuracy on what actually arrives in practice.

A single missing or wrong field can fail an entire document even when every other field was extracted correctly. A system that correctly extracts 19 of 20 required fields has still failed that document if the missing field happens to be the diagnosis code or the qualifying lab value. Numeric fields deserve a dedicated evaluation category given the documented gap in accuracy on continuous measures: lab values, dosage thresholds, and imaging measurements should be tested separately from narrative text extraction. Confidence scores need to be checked for calibration, not just presence, because a confidence score that does not track actual accuracy is worse than having no confidence score at all, since it manufactures a false sense of certainty.

Open-source benchmarks built around real-world document layouts, rather than synthetic clean samples, give teams a replicable baseline, but the organization's own documents still need to run through that baseline before any vendor's accuracy claim can be trusted for that organization's specific use case. None of this is a pre-launch checklist to complete once. Payer criteria documents change, new referral sources introduce new document types, and model performance drifts over time. Monitoring extraction accuracy has to continue as an ongoing operational responsibility for as long as the pipeline is in use.