Automation Bias Is the Real Risk Nobody Is Talking About

The AI governance conversation keeps fixating on AI errors. The more dangerous problem is what happens when AI is mostly right.

There's a study published in The Lancet Gastroenterology & Hepatology in August 2025 that doesn't get nearly enough attention in healthcare AI circles. Researchers tracked gastroenterologists over time as they began using AI-assisted colonoscopy detection tools. The AI was good — genuinely helpful, catching adenomas that human eyes miss. Adoption was high. Detection rates improved.

Then the researchers measured something else: what happened to the clinicians' own diagnostic accuracy when the AI was unavailable. The unaided adenoma detection rate plummeted from 28.4% to 22.4% — a 20% relative decline. The AI had made them measurably better at finding polyps with the tool, and measurably worse at finding them without it.

(Budzyń, Romańczyk, et al., The Lancet Gastroenterology & Hepatology, August 2025)

That's automation bias. And it's the governance risk almost nobody in specialty pharma HUB operations is talking about.

We're Measuring the Wrong Thing

The AI governance conversation in HUB operations has been dominated by one question: what happens when the AI gets it wrong? The worry is the false positive on prior authorization criteria, the misclassified benefit eligibility, the clinical appropriateness determination the algorithm got backwards. Governance frameworks are rightly designed around catching those errors.

But there's a second failure mode that's harder to detect, better documented in the literature, and almost completely absent from the governance frameworks seen in specialty pharma. It's not about AI making errors. It's about what happens when AI is mostly right — and the trained professionals reviewing its work stop looking as carefully as they used to.

Automation bias is the tendency of skilled humans to over-rely on automated system outputs — to reduce their own cognitive engagement, accept AI recommendations without sufficient scrutiny, and gradually lose the clinical and operational judgment they've spent years developing. It doesn't announce itself. There's no error log. The AI keeps running, the workflows look smooth, and the performance dashboards are green. The degradation is in the human.

In a HUB environment — where nurse case managers are reviewing AI-assisted prior authorization recommendations, where benefit verification specialists are working off AI-generated eligibility summaries, where clinical appropriateness checks are surfaced by AI tools — this isn't a theoretical risk. It's a structural one. And it gets worse as your AI gets better.

"When automated systems achieve 95% accuracy in prior authorizations, human reviewers experience 'severe vigilance decrement' — transforming from critical clinical evaluators to passive administrative rubber-stampers."

Two Ways Automation Bias Kills Accuracy

The clinical informatics literature identifies two distinct error patterns. Understanding both matters because they require different governance responses — and only one of them responds to training.

Omission errors occur when a clinician or operations professional fails to catch an AI mistake because they weren't actively looking for one. If the AI says the PA criteria are met and the reviewer has been conditioned to treat AI output as presumptively correct, errors pass through unchallenged. The critical finding: training is largely ineffective at reducing omission errors. They're rooted in neurological vigilance fatigue — a deeper cognitive mechanism that policy and training can't simply override.

Commission errors occur when a skilled professional chooses the AI's recommendation over their own clinical judgment, even when the two conflict. They notice the discrepancy. They override their own instinct. A 2012 study in JAMIA found that clinicians overrode their own correct decisions in favor of machine advice in 6-11% of cases — and when the system was wrong, error probability increased 26%. More recently, a 2026 study in Machine Learning for Biomedical Applications found a 7% automation bias rate in AI-assisted pathology workflows, with time pressure intensifying the effect. The better news: training focused on questioning the machine can reduce commission errors by 33-40%.

(Goddard, Roudsari & Wyatt, JAMIA, 2012; Rosbach et al., Machine Learning for Biomedical Applications, 2026)

Both error types share a common root: governance frameworks that treat human review as a formality rather than an active cognitive task. When the workflow says "review and approve" but the training, incentives, and interface design all signal "the AI already handled this," you get omission. When the AI's confidence score is displayed prominently and the reviewer's independent judgment has no documented weight, you get commission.

Why HUB Operations Is Especially Exposed

Every healthcare setting using AI faces automation bias risk. HUB operations faces it with a specific structural vulnerability: volume pressure combined with high-stakes clinical adjacency.

A nurse case manager or PA specialist processing dozens of cases per shift develops workflow rhythms that favor speed. When AI tools enter that environment, they don't just assist — they reshape the cognitive cadence. The human's job transitions, subtly, from "evaluate this case" to "review what the AI found." That transition feels like efficiency. It often is efficiency. And it creates exactly the conditions under which automation bias compounds.

There's a second vulnerability specific to HUB workflows: the handoff chain. Most HUB cases move through multiple reviewers — intake specialists, benefit verification teams, nurse case managers, PA specialists, clinical pharmacists. Automation bias can stack across that chain. If intake accepts an AI eligibility summary without full verification, and benefit verification builds on that summary, and the nurse case manager sees a file that already looks complete — each person's reduced scrutiny multiplies the last person's.

The 60% of patient services programs running hybrid HUB models face an added layer: AI systems often can't access the complete data needed to support the human reviewer — clinical notes, formulary exceptions, payer-specific nuances. The AI's confidence is based on incomplete information, but it may not signal that uncertainty clearly. The reviewer, already primed to trust the output, has no reliable cue to dig deeper.

Governance Lives in the Interface, Not the Policy Document

Here's what Dr. Adam Rodman at the 2025 Penn Medicine Nudges in Health Care Symposium called the "performance paradox": traditional human-in-the-loop design — where a human reviews before AI output becomes action — can actually degrade system accuracy over time as human vigilance deteriorates.

The governance implication isn't to remove humans from the loop. It's to be precise about what kind of loop you're designing. Penn LDI distinguishes "in the loop" (human reviews before action) from "on the loop" (human monitors after AI has acted). Both have roles in HUB operations. The governance failure is when organizations claim "in the loop" while the workflow is functionally "on the loop" — when human review has been reduced to exception-catching after AI has already shaped the decision.

Preventing automation bias requires governance decisions that most organizations haven't made yet:

Active engagement protocols, not passive review. The interface should require reviewers to demonstrate engagement — not just click approve. This might mean requiring independent assessment of the highest-risk element before seeing the AI recommendation, or structured documentation of what the reviewer independently verified. The goal is cognitive activity, not a compliance checkbox.

Explicit uncertainty surfacing. AI systems should display not just their recommendation but their confidence calibration. When a model's confidence is below threshold, or when a case has features outside the training distribution, the reviewer needs a clear signal. Governance must define what "low confidence" looks like in the workflow — not just in the algorithm documentation.

Skill maintenance requirements. The Lancet study is a direct warning: AI can create skill atrophy even as it improves outcomes on the metrics you're tracking. HUBs need to measure staff proficiency over time on cases where AI recommendations are deliberately withheld. Proficiency audits that test independent clinical judgment — not AI-augmented performance — are a governance decision that almost no HUB has made.

Risk-differentiated review depth. Lyell et al.'s research on AI prescribing alerts found that flawed algorithmic recommendations increased prescribing error rates by 33.3%. High-volume, low-variance cases may tolerate lighter review. Cases involving novel presentations, payer-specific nuances, or high-acuity clinical flags require active engagement protocols regardless of AI confidence. Governance must map this by case type — not apply a single review standard across the board.

The Question to Ask Your Operations Team Today

If you're leading HUB operations or patient access, here's the diagnostic question — not "does our AI make errors?" but: "do our people still know how to work the cases the AI is working?"

If the honest answer is "I don't know," that's the governance gap. Because the evidence is unambiguous: automation bias is documented, measurable, and — critically — manageable. But only if the governance framework is designed to address it before the skill atrophy shows up in patient outcomes or regulatory audits.

The organizations that get this right don't have less AI. They have better-designed human engagement in AI-assisted workflows. The AI runs fast and handles the volume. The human review is active, not perfunctory. Performance metrics include independent proficiency, not just augmented throughput. Regulators are beginning to ask for exactly this — which leads directly to where this series goes next.

Building the Governance Architecture That Holds

The first two articles in this series established the framing: governance should be a guardrail, not a gate (Post 1), and the human handoff must be precisely mapped to risk level rather than applied uniformly (Post 2). Automation bias is the dynamic that makes both principles harder to execute as your AI matures.

A well-designed guardrail that isn't maintained erodes. A well-mapped handoff process that doesn't account for skill atrophy drifts toward rubber-stamping. The governance architecture for HUB AI has to be a living system — one that monitors not just AI performance, but human engagement over time. The moment it becomes static, the bias clock starts running.

In the final article in this series, we'll look at the regulatory dimension: what CMS, FDA, and the broader policy environment are signaling about AI governance in prior authorization and clinical decision-making — and why organizations that build defensible human-in-the-loop frameworks now are positioning for what's coming.

If you're building or auditing AI governance for a HUB or patient access program, I'd like to hear where automation bias shows up in your current framework — or where it's conspicuously absent. The conversation in comments is always open.

— Ankur Jain

Key Sources

• Budzyń, Romańczyk, et al. The Lancet Gastroenterology & Hepatology, August 2025.

• Goddard, K., Roudsari, A., & Wyatt, J.C. JAMIA, 2012.

• Rosbach, M., et al. Machine Learning for Biomedical Applications, 2026.

• Lyell, D., et al. E-prescribing automation bias study.

• Rodman, A. Penn Medicine Nudges in Health Care Symposium, 2025.

• Health Affairs, 2025. The AI Arms Race In Health Insurance Utilization Review.

• Penn LDI. In the Loop or On the Loop: The Conundrum of AI Clinical Decision Support.