Mohanish Anand, TA Head for Global Trial Management, ran a simple test: he asked ChatGPT, Claude, Gemini, Perplexity, DeepSeek, and Copilot to pick a random number between 1 and 30. Every single one said 17. When he pressed each model to explain itself, some claimed the result came from a genuinely random process. Others admitted 17 just “looks” random. The explanations contradicted each other. The number never changed. What Anand captured in that single experiment is the central failure mode of AI in clinical development: a system that produces confident, plausible, internally consistent output, trained on the patterns of what has already happened, presented to users as if it were reasoning about what is happening now.

The stakes attached to that failure mode are not abstract. Clinical trials depend on decisions that contain something no historical pattern can fully encode: a site population behaving differently than expected, a protocol burden that reads fine on paper and feels punishing to the patient living it, a standard of care that shifted after the last comparable study closed. Per Anand’s post, “AI tells us what usually happens. Human judgment asks: What is different this time?” That question does not get asked when the workflow is designed to rubber-stamp AI output at speed.

The Button That Costs a Fortune

Doug Bain, a consulting partner focused on trial technology, put it with surgical directness: “If your AI makes 10,000 decisions and a human clicks Approve 10,000 times, you haven’t necessarily created a control. You may just have created a very expensive button.” The human-in-the-loop construct has become the industry’s primary answer to AI governance anxiety. Regulators ask for it. Sponsors build it into SOPs. Ethics review committees accept it as sufficient. But the behavioral reality of high-volume AI-assisted review is that humans rationalize rather than evaluate when the cadence is fast enough and the outputs look coherent enough. The 17 problem, scaled to 10,000 protocol deviation flags or 10,000 site risk scores, becomes invisible precisely because each individual approval feels reasonable.




This is the counterintuitive truth the field keeps sidestepping. The assumption embedded in most AI governance frameworks is that more human touchpoints produce more human judgment. The evidence from cognitive science runs the opposite direction: high-volume, low-variance approval workflows produce automation bias, the documented tendency of people to accept machine output without independent verification when the machine rarely appears to be wrong. A system designed to catch AI errors can, under the wrong conditions, systematically suppress the human capacity to find them.

Rudy Malle, who trains and places clinical research professionals, identifies the specific capabilities that AI cannot independently own: patient safety, data integrity, risk interpretation, site coaching, exception management, and regulatory accountability. Every item on that list requires exactly the kind of judgment that automation bias erodes. The workflow design question that sponsors and CROs have not yet answered honestly is: at what approval volume does the human-in-the-loop stop being a safeguard and start being a liability shield?

Pattern Recognition Is Not Clinical Reasoning

Serhii Vakal, Computational Drug Discovery Architect at Orion Pharma, draws the line precisely where the engineering analogy for AI breaks down: every AI-generated molecule, every predicted property, every computationally guided experiment must still "confront the complexity of living biology, laboratory validation, and clinical trials." That confrontation is not a remaining inefficiency to be optimized away. It is the substance of drug development. Biology does not behave like a dataset. Patients are not rows.

Morgan Levine, building at a stealth startup, pushes the critique deeper than execution. Even if AI solved every operational bottleneck in the find-a-target, develop-a-drug, run-a-trial pipeline, the underlying paradigm would still be insufficient for most major chronic diseases. The pipeline architecture assumes a single actionable target. Aging does not have one. Most complex inflammatory and neurodegenerative diseases do not have one. AI applied to an epistemically flawed model of disease accelerates the production of answers to the wrong question. Per Levine's post, "we continue plowing ahead with the therapeutic pipeline built around finding a target and developing a drug against it. That will never be enough."

Jon Bondebjerg, a clinical drug development leader, offers the most operationally grounded account of what useful AI actually looks like in practice. He used AI tools to build and stress-test a scenario-planning model for a clinical development program, testing which study sequence preserved capital and protected timelines when both were constrained. The useful output was not the first model. It was the challenge process: tracing every assumption, benchmarking every number, verifying that formula logic matched scenario labels. That process surfaced errors that "looked convincing on the surface." His conclusion reads like a corrective for the entire field: "AI is most useful as a demanding analytical partner, not as an answer machine."

Subhajit Sengupta, Associate Director of Data Science at Cytel, points to the specific technical conditions under which AI earns trust in statistical and adaptive design workflows: validated knowledge bases, transparent context engineering, and formal evaluation criteria before output reaches a decision-maker. These are not exotic requirements. They are the minimum conditions for any analytical tool used in GCP-regulated environments. The gap between that standard and what most organizations are actually deploying is where risk accumulates quietly.

Whose Evidence Is the AI Summarizing?

The governance problem compounds when AI operates closer to the patient. Eden Brownell, a behavioral scientist focused on human-AI interaction, examines the OpenAI-Epic integration with a precision that most breathless coverage of that announcement skipped entirely. Giving an AI read access to a patient's electronic health record does not update the model on current medicine. The training data still encodes publication bias, underrepresentation in clinical trials, and the documented patterns of who received less care despite equivalent need. Real-world data embedded in EHR systems carries the history of healthcare inequity alongside the clinical signal. Per Brownell's post: "Published research can exclude people from the evidence. Real-world data can encode how the system failed them. And AI can compress both into an answer that sounds complete."

For sponsors running decentralized or hybrid trials who are integrating AI-assisted site selection, recruitment screening, or eCOA interpretation, this is not a distant philosophical concern. It is a protocol design question. If the model's training data systematically underrepresents the populations a trial is designed to enroll, AI-assisted recruitment tools will replicate and potentially amplify existing diversity gaps while producing outputs that read as optimized and evidence-based. The FDA's 2020 Action Plan for the Diversity of Participants in Clinical Trials and the subsequent guidance on diversifying clinical trial populations placed the burden of diversity on sponsors. AI does not redistribute that burden. It can quietly reinforce the structural conditions that created it.

What the convergence of these voices describes is not a technology that has failed to mature. It is an industry that has not yet built the institutional discipline to use the technology it already has without mistaking pattern recognition for clinical judgment, or approval workflows for oversight. Bondebjerg found real value in AI-assisted scenario planning, but only because he treated every model output as a hypothesis to be contested, not a recommendation to be accepted. Anand's six models all said 17. The question every trial operations leader needs to answer before their next AI implementation is not whether their vendor's system is smarter than ChatGPT. It is whether the humans in their loop are actually asking why.

References

  1. Mohanish Anand, "What the models can't see," LinkedIn
  2. Doug Bain, "Can we stop pretending that putting a human-in-the-loop with AI makes it safe?" LinkedIn
  3. Rudy Malle, "AI is not eliminating clinical operations," LinkedIn
  4. Serhii Vakal, "Again and again, we hear bold claims from Big Tech," LinkedIn
  5. Morgan Levine, "A lot of people are rightly pointing out," LinkedIn
  6. Jon Bondebjerg, "Using AI to guide clinical program planning," LinkedIn
  7. Subhajit Sengupta, "Clinical trial teams are being asked for more statistical rigor," LinkedIn
  8. Eden Brownell, "OpenAI is now connected to Epic," LinkedIn
Website |  + posts

Moe Alsumidaie, MBA, MSF, is founder and Chief Editor of Vanguard Publications, which publishes Clinical Trial Vanguard, Pharma Vanguard and BullScope, and Head of Research at CliniBiz. He has two decades in clinical trial operations and data science, with earlier roles at Genentech, Abbott Vascular and Stanford University Medical Center, and is a guest lecturer in clinical trial sciences at Rutgers University.