The Oversight Gap Nobody Is Naming: State Sandboxes Are Filling a Void the FDA Has Left Open

Three regulatory developments in the past six months point to a pattern the clinical trials industry has not yet named: the Autonomous AI Accountability Vacuum. The FDA has authorized 1,451 AI/ML-enabled medical devices since 1995, with annual authorizations surging to record levels by the end of 2025 — and yet no federal framework currently mandates independent, prospective oversight of autonomous AI systems operating inside clinical trials. Utah just stepped into that void.

The Utah Department of Commerce’s Office of Artificial Intelligence Policy published its “Regulatory Relief Process” on May 11, 2026, formalizing the state’s AI regulatory sandbox under the Utah Artificial Intelligence Policy Act. What makes this remarkable for clinical trial operators is not the sandbox concept itself — it is what the sandbox reveals by existing: that sponsors deploying AI-driven clinical decision support are currently navigating a patchwork of state-level experimentation because the federal infrastructure for prospective AI oversight in research has not materialized.

That vacuum has operational consequences no one at the sponsor level is fully pricing in yet.

The Signal Coming Out of Salt Lake City

Open Utah’s Regulatory Relief Process document and you find a structured mechanism: AI developers operating within the sandbox receive defined regulatory relief — protection from certain liability and enforcement actions — in exchange for transparency obligations, structured reporting, and state-level monitoring. The program explicitly covers healthcare AI applications. What it creates, functionally, is a supervised deployment environment where an independent authority watches an AI system operate in real conditions and requires developers to document what breaks.

That structure should sound familiar to anyone who has run a Phase II adaptive trial under a Data Safety Monitoring Board. The principle is identical: you do not wait for harm to occur and then investigate. You install a watching mechanism before deployment and give it authority to intervene. The MRCT Center’s “Framework for Review of Clinical Research Involving AI,” published in June 2025, makes precisely this argument — that IRBs need structured, practical protocols for evaluating AI research involvement, anchored in the same foundational principles that govern human subjects protections.

The uncomfortable question that follows: if a sponsor today deploys an autonomous AI system to make real-time dose-modification recommendations inside a Phase III oncology trial, who is the DSMB equivalent for the algorithm? Not the FDA — there is no standing federal requirement. Not the IRB — most lack the technical frameworks to evaluate algorithmic decision logic. And not the CRO, which is contractually downstream of the sponsor. Utah’s sandbox model, by contrast, makes the state itself that watching mechanism. That is not a complete solution. But it is a solution, and right now it is the only one with formal authority behind it.

What the FDA’s PCCP Guidance Misses

The FDA did move on AI governance — just not on trial oversight. In December 2024, the agency finalized guidance on Predetermined Change Control Plans for AI-enabled devices, covering De Novo, PMA, and 510(k) pathways. A PCCP allows a manufacturer to pre-specify how their AI system may evolve post-market without requiring a new marketing submission for each algorithmic update. The intent is regulatory efficiency.

But the PCCP framework is a post-market lifecycle tool. It governs how a cleared device changes after approval — not how an AI system behaves inside a clinical trial before it has any clearance at all. Consider the operational gap this creates: a sponsor using an AI-driven patient stratification engine during a Phase II basket trial is operating outside the PCCP framework’s scope entirely. The device has no clearance. The PCCP has no purchase on it. The FDA’s draft guidance on AI-enabled device lifecycle management explicitly requires manufacturers to address transparency and algorithmic bias throughout the device lifecycle — but “lifecycle” begins at marketing authorization, not at first patient enrolled.

Clinical trial AI does not start at market clearance. It starts at protocol design. That twelve-to-thirty-six month gap between first deployment and potential clearance is precisely when AI systems are most likely to behave unexpectedly — operating on training data distributions that do not match the enrolled population, making recommendations in therapeutic contexts they were not validated for, and doing so without any independent watching mechanism required by federal regulation.

A published cluster-randomized trial in npj Digital Medicine found that an autonomous AI diagnostic system increased specialist clinic productivity by 40%, reaching 1.59 encounters per hour versus 1.14 in controls. That is a meaningful efficiency gain. But productivity metrics are not safety metrics. The same system architecture that accelerates throughput can also accelerate systematic misclassification — and the trial reported productivity, not adverse decision rates.

The Operational Reckoning Ahead

Here is the counterintuitive framing most sponsors are getting wrong: they believe the regulatory risk in clinical AI sits at the submission stage — that the FDA will scrutinize the AI methodology when they review the NDA or BLA. That belief is structurally incorrect. By the time a sponsor submits, the AI’s role in the trial is historical record. If the algorithm introduced systematic bias in randomization, stratification, or endpoint assessment during the trial, that bias is baked into the dataset. No post-hoc audit corrects it; it surfaces as unexplainable heterogeneity in the statistical analysis, and the FDA reviewer who notices it has no framework to distinguish AI-induced confounding from natural variance.

The oversight failure, in other words, does not appear at submission. It appears three years earlier, silently, in the trial data.

For CROs, the implication is structural. Contracts currently assign AI validation responsibility to sponsors while CROs execute operations — a division that made sense when “AI” meant EDC auto-coding and risk-based monitoring flags. It does not hold when the AI is making real-time clinical recommendations that alter patient management. The MRCT Center’s AI Framework explicitly places IRBs in the evaluation chain, but IRBs reviewing a protocol in 2026 are being asked to assess transformer-based decision architectures with frameworks written for human investigator judgment. The gap between the IRB’s current competency and the task being assigned to it is not a training problem — it requires structural redesign.

Technology vendors face the sharpest near-term pressure. The FDA has cleared over 950 AI/ML-enabled medical devices as of early 2026 — roughly a doubling from the approximately 520 cleared in 2022. That pace of authorization has created a market expectation that clearance equals validation. Vendors selling AI-powered clinical decision support into trial environments are trading on cleared device credentials for systems operating in uncleared contexts. When the first major trial failure is traced to an autonomous AI recommendation — and the probability of that event rises with every new deployment — the liability question will not wait for federal rulemaking.

Utah’s sandbox, imperfect as it is, offers a template that sponsors should be studying now rather than after the first crisis. The program’s core mechanism — a defined regulatory authority, transparency obligations, structured reporting, and pre-specified intervention criteria — maps directly onto what a trial-level AI oversight framework would require. The FDA has proposed a pilot program for AI-enabled optimization of early-phase clinical trials, but a proposal is not a requirement, and sponsors with Phase III timelines cannot wait for a federal pilot to mature.

Twelve months from now, the sponsors who treated Utah’s sandbox as a curiosity will be the ones scrambling to retrofit oversight documentation onto trials already underway. The ones who read the May 2026 Regulatory Relief Process document as a design specification — and built equivalent watching mechanisms into their trial protocols proactively — will be the ones whose datasets survive regulatory scrutiny. The FDA will eventually close this gap. The only question is whether it does so before or after a high-profile trial failure makes the closure mandatory.

References

1. Nature Medicine — “What Utah’s clinical AI sandbox reveals about independent oversight”
2. Utah Department of Commerce, Office of Artificial Intelligence Policy — “Regulatory Relief Process” (May 11, 2026)
3. American Hospital Association — “FDA Finalizes Recommendations Simplifying Approval Process for Medical Devices Using AI” (December 2024)
4. MRCT Center — “Framework for Review of Clinical Research Involving AI” (June 2025)
5. MedDeviceGuide — “FDA Draft Guidance: AI-Enabled Device Software Functions Lifecycle Management and Algorithmic Bias”
6. npj Digital Medicine — Cluster-randomized trial: autonomous AI system increases specialist clinic productivity by 40%
7. Intuition Labs — FDA AI/ML Medical Device Tracker (1,451 cumulative authorizations through end of 2025)