Routine Health Data Is Outperforming Trial Data for AI Training. The FDA Hasn’t Caught Up.

Three converging developments in the past eight months point to a trend that clinical operations teams have not yet named: the Routine Data Inversion. The assumption that controlled, curated clinical trial datasets produce superior AI models is quietly collapsing, and the regulatory architecture designed to validate those models was never built to handle what comes next.

The evidence started arriving in the pages of Nature Medicine. A study published in 2026 demonstrated that neuroimaging AI models trained on routine health system data, the kind collected during ordinary clinical care across diverse, unselected patient populations, delivered better diagnostic performance than models trained on the carefully curated datasets that typically power clinical trial-grade AI. The operational implication is significant enough to warrant a pause: the messiest data may be the best data.

That finding does not arrive in a vacuum.

What Routine Data Actually Reveals

The conventional wisdom in clinical AI development holds that training data quality means controlled acquisition protocols, homogeneous patient populations, and rigorous exclusion criteria, essentially replicating the conditions of a Phase 3 trial. Health system data fails all three tests. Scanners vary by site. Technician protocols drift. Patient populations include comorbidities, age ranges, and demographic compositions that no IRB-approved enrollment criteria would ever tolerate. The standard assumption is that this noise degrades model performance.

The Nature Medicine study inverts that assumption with specificity. Neuroimaging AI models trained on routine clinical data, collected as a byproduct of normal care delivery rather than as a deliberate research artifact, did not just match curated-dataset models. They outperformed them. The mechanism is straightforward once you accept it: a model trained on diverse, variable, real-world inputs learns to generalize. A model trained on pristine trial data learns to perform beautifully within conditions that most clinical sites will never replicate.

This connects directly to a documented problem in diagnostic AI. Research published in PLOS Medicine on dermatology AI, analyzed by Daneshjou et al. and available via PubMed Central, found that AI algorithms trained on non-representative datasets showed measurably reduced diagnostic accuracy on darker skin tones compared to lighter skin tones. The neuroimaging finding in Nature Medicine is the affirmative version of the same phenomenon: train on the real distribution of patients, and your model performs on the real distribution of patients. That should be obvious. In practice, it has not been how AI development in clinical research works.

The Regulatory Gap Taking Shape

Here is where the regulatory architecture becomes relevant, and where the gap becomes uncomfortable to look at directly.

The FDA updated its guidance on Real-World Evidence for medical devices in December 2025, expanding how both FDA staff and sponsors may use Real-World Data to support regulatory decisions. According to IQVIA’s analysis of that update, a key provision allows the FDA to accept RWE without always requiring the traditional randomized controlled trial construct. For medical devices, including software as a medical device, this matters enormously, because the approval pathway for AI diagnostic tools runs through 510(k) clearances and, increasingly, through Predetermined Change Control Plans.

The FDA has also established a framework for Class II radiological machine learning-based quantitative imaging software through its PCCP rule, which allows AI developers to pre-specify how their models will be updated after clearance without triggering a new review cycle. On paper, this is forward-looking. In practice, it leaves an operational question unanswered: if the highest-performing neuroimaging AI is trained on routine health system data rather than protocol-controlled trial data, what validation framework governs the submission? The PCCP rule tells sponsors how to update a cleared model. It does not tell them how to demonstrate that a routine-data-trained model meets the evidentiary standard for clearance in the first place.

That gap is not theoretical. The ONC’s 2023-2024 data brief found that 71% of hospitals reported using predictive AI integrated with their EHR systems, up from 66% the prior year. Those models exist. They are being used clinically. Many of them were never submitted to the FDA because sponsors made a risk calculation: the evidentiary pathway for routine-data-trained AI is ambiguous enough that clearance is uncertain, and the commercial deployment pathway through general wellness or non-device classifications is faster. The Nature Medicine finding accelerates that calculation.

What Sponsors, CROs, and Vendors Must Reckon With

The counterintuitive implication here is that sponsors who have invested most heavily in controlled data acquisition infrastructure for AI training may now hold a competitive liability, not an asset. A bespoke imaging acquisition protocol designed for a Phase 2 neurology trial produces clean, auditable, reproducible data. It also produces a training set that the real clinical world will never replicate at scale. If the Nature Medicine finding holds across validation cohorts, that investment in control is an investment in brittleness.

For CROs, the operational shift is more immediate. The emerging model is not a clinical trial that generates training data. It is a health system data partnership that generates training data, supplemented by a prospective validation study designed to satisfy regulatory reviewers. The clinical trial becomes the validation instrument, not the primary data source. CROs that have built their AI capabilities around site-level data collection will need to develop health informatics competencies that most of them do not currently hold at any serious depth.

The FDA and its international counterparts have begun acknowledging this pressure. The joint FDA, Health Canada, and MHRA document on Good AI Practice in Drug Development, which outlines 10 guiding principles for AI evidence generation across the drug product lifecycle, emphasizes transparency in data sourcing and representativeness of training populations. Those principles are directionally consistent with the Nature Medicine finding. But principles are not submission requirements. A sponsor who reads that document and then attempts to file a 510(k) for a routine-data-trained neuroimaging tool will find that the gap between principle and procedure is still wide enough to lose a program in.

The global AI in medical imaging market was valued at USD 3.39 billion in 2024, according to Market Research Future, with neurology holding the largest application segment. That market is growing into a regulatory framework that was designed before the best evidence confirmed that routine data beats curated data. Somewhere in that mismatch, a sponsor is about to file a submission that forces the FDA to answer a question it has so far avoided: does a model trained on 200,000 routine MRI reads from 47 health systems require the same evidentiary package as one trained on 800 protocol-controlled scans from a single-site academic trial? The answer will define the next decade of neuroimaging AI development. The agency should start drafting it now, before the first rejection letter writes it for them.

References

1. Nature Medicine — “Learning from routine health system data builds better neuroimaging AI models”
2. IQVIA — “FDA Updates Guidance on Real-World Evidence for Medical Devices” (January 2026)
3. Encord — “FDA Predetermined Change Control Plan (PCCP) Rule for AI/ML Medical Devices”
4. FDA, Health Canada, MHRA — “Guiding Principles of Good AI Practice in Drug Development”
5. ONC — “Hospital Trends in Use, Evaluation, and Governance of Predictive AI, 2023–2024”
6. PubMed Central — Daneshjou et al., AI diagnostic accuracy and skin tone disparities
7. Market Research Future — “AI in Medical Imaging Market, 2024 Valuation”