AI Is Already Inside the Doctor’s Visit. The Question Is Whether It’s Helping or Hunting for Billable Findings.

Picture a primary care visit somewhere in a federally designated physician shortage area. The patient speaks, ambient AI transcribes, and before the clinician has touched the doorknob on the way out, an AI system has already surfaced a differential, flagged a gap in chronic disease management, and suggested a panel of tests. The patient leaves thinking they received better care. The clinician saved forty minutes of after-hours charting. And somewhere in that frictionless encounter, an AI system ordered workup at a cost of $314 per visit against a physician baseline of $97. That figure comes from Daniel Morgan, Professor and Physician at VA Maryland Healthcare System and University of Maryland, citing a study published in NEJM AI examining 200 deidentified primary care notes across 24 AI systems. It is not an outlier. It is a preview.

More than half of physicians now consult AI tools regularly, according to Zeke Emanuel, Vice Provost for Global Initiatives at the University of Pennsylvania. Emanuel is bullish: these tools improve diagnostic accuracy and chronic disease management, and they reach patients in underserved areas who otherwise wait months for an appointment. That argument deserves to be taken seriously. But behind the genuine gains lies a structural incentive problem that medicine has never successfully contained, and AI may be about to supercharge it.

The Accuracy Trap

The conventional wisdom in clinical AI development holds that accuracy is the north star. Build a system that catches what humans miss, validate it against physician performance, and deploy. The FDA has largely accepted this framing: clearances for AI-enabled clinical decision support have focused on sensitivity, specificity, and AUC scores. Regulators ask whether the AI is right. Almost nobody asks what happens when it is right and acts on it.

Morgan‘s data punctures that logic. In the Annals of Internal Medicine study he cites, researchers created synthetic patient scenarios drawn from Choosing Wisely recommendations, the benchmark list of tests and procedures clinicians should stop ordering because evidence does not support them. AI systems showed 5.4% overuse against 0.6% underuse. The asymmetry is not incidental. The models are trained to find, flag, and act. Restraint has to be engineered in deliberately, and so far, almost no one is engineering it.

The NEJM AI study makes this concrete. Physicians recommended no testing in 66% of primary care visits, which is clinically normal. The AI systems tested every patient. Cost-aware prompting brought AI spending down to 2.5 times the physician baseline. Parsimonious prompting got it to 1.4 times. Neither got it to parity. The best prompt engineering available still left AI generating 40% more testing than a trained clinician exercising judgment. That gap, multiplied across a health system, is not a rounding error.

Morgan frames it precisely: “Clinical AI seems very likely to increase overuse. So far studies have looked at it simplistically.” He notes that Choosing Wisely addresses only the clearest cases of always-inappropriate care, and that up to 30% of real clinical medicine falls into genuinely gray territory where overuse is far harder to define and far easier to rationalize. When an AI system operates in that gray zone with a bias toward action, the math gets ugly fast.

When the Scribe Becomes the Physician

Meanwhile, on the documentation side, the gains are real and the evidence is accumulating. Jennifer Stephens, Chief Medical Officer at Jefferson Health, published findings this week in NEJM Catalyst showing that ambient AI reduced documentation time across the Jefferson system, with clinicians closing charts sooner and spending less time on after-hours notes. Advanced practice clinicians saw gains 2.5 times larger than physicians, a finding Stephens rightly flags as a reminder that ambient AI serves the whole care team. The more consistently clinicians used the tool, the greater the benefit. That dose-response relationship is exactly what you want to see in an effectiveness study.

But Stephens also buries the most important variable in her abstract: “thoughtful governance mattered as much as the technology itself.” That single phrase should be the headline of every implementation story. Jefferson built governance structures around its ambient AI rollout. Most health systems deploying these tools at scale are not Jefferson, and most do not have a published governance framework anchoring their implementation. The time savings are real. Whether they hold when governance is absent is a different question entirely.

The distinction between ambient documentation AI and clinical decision support AI is collapsing in practice. Systems that start as scribes are being upgraded to flag abnormal findings, surface care gaps, and recommend next steps. The workflow is the same. The stakes are completely different. A scribe that gets a word wrong delays a note. A clinical AI that flags a finding that a physician then feels obligated to work up has just initiated a cascade.

This is the Oslerian warning that Michel Goldman, Founding President of the I3h Institute at Université libre de Bruxelles, articulates in a new opinion piece in Frontiers in Medicine. Goldman, writing as both physician and cancer patient, argues that AI risks amplifying an existing erosion: clinicians already screen-bound by administrative load and pressured by hospital managers to eliminate unprofitable activities are losing personal contact with patients. Add an AI layer that surfaces more findings, flags more gaps, and recommends more actions, and the physician becomes a validator of machine outputs rather than an observer of a human being. William Osler’s insistence on careful observation, physical examination, and genuine clinical relationship does not survive that transition intact.

What the Guardrails Actually Require

Emanuel draws a sharp line between purpose-built clinical AI, trained on medical literature with specific guardrails, and general-purpose chatbots. He cites a study finding that nearly half of AI chatbot responses to common health questions were inaccurate or incomplete. The distinction is fair. OpenEvidence, the tool Emanuel references, is not ChatGPT with a stethoscope. But the guardrail argument cuts both ways: if clinical AI requires purpose-built guardrails to be safe, then deploying it without explicitly designed stewardship mechanisms is not a neutral act. Morgan makes exactly this point, noting that Microsoft’s Orchestrator framework demonstrates that AI can perform stewardship functions when they are deliberately built in. The operative phrase is “built in.” Post-hoc prompting is not a governance strategy.

For the clinical trials community, the implications arrive faster than most sponsors recognize. As AI diagnostic tools propagate into routine care, the patients entering trials are increasingly patients whose workup was shaped by AI. Incidental findings surfaced by AI systems are driving referrals. AI-flagged chronic disease gaps are generating diagnoses. The eligibility criteria written into a protocol in 2023 assumed a baseline diagnostic practice that no longer describes how patients are being characterized in 2025. If AI overuse is generating 5.4% inappropriate testing even on the clearest Choosing Wisely cases, the contamination of eligibility populations and endpoint baselines is not theoretical.

The FDA has cleared more than 950 AI-enabled medical devices as of the last public count, and the agency’s 2021 action plan for AI and machine learning acknowledged the need for good machine learning practice standards. What that framework does not yet address is value stewardship as a clearance criterion. An AI system can be accurate, sensitive, and specific, and still drive $314 of testing in a visit that warranted $97. Accuracy and appropriateness are not the same axis. Until regulators build a stewardship axis into device evaluation, sponsors designing trials in any therapeutic area touched by AI diagnostics are building on ground that may shift beneath them without notice.

The clinicians raising these concerns this week are not Luddites. Goldman is a cancer patient benefiting from AI-accelerated therapy. Morgan wants stewardship built into LLMs, not LLMs removed from clinics. Stephens published evidence that ambient AI works when governed well. The convergence of their concerns is not a rejection of clinical AI. It is a demand that the field stop treating accuracy as the finish line when the race has barely started, and that patients sitting in those AI-mediated visits deserve to know a machine helped make decisions about their care before they consent to anything.

References

1. Zeke Emanuel, LinkedIn post: “AI is already inside the doctor’s visit, even when patients don’t realize it.” (2026)
2. Jennifer Stephens, DO, MBA, FACP, LinkedIn post: “Ambient AI as a Tool for Returning Time to Care,” NEJM Catalyst Innovations in Care Delivery (2026)
3. Michel Goldman, LinkedIn post: “In the Age of AI, Medicine’s Key Principles Matter More Than Ever,” Frontiers in Medicine (2026)
4. Daniel Morgan, LinkedIn post: “We need to evaluate the benefits and harms of clinical AI” citing Annals of Internal Medicine and NEJM AI studies (2026)