What the Evidence Says About Trusting an AI Suggestion Too Much

Controlled experiments now show a consistent pattern: when a decision-support tool offers confident but incorrect advice, clinicians frequently adopt it, and their decisions get worse than if they had received no help at all. In a randomized trial of physicians who had already completed AI-literacy training, those exposed to deliberately flawed model advice reasoned less accurately than colleagues who received sound advice. The phenomenon has a name, automation bias, and the newer evidence lets us describe its size and its mechanism rather than just assert that it exists. This article appraises that evidence. It is educational and not medical advice.

What automation bias means

Automation bias is the tendency to over-trust an automated recommendation, either by acting on a wrong suggestion (a commission error) or by failing to act because the system stayed silent (an omission error). A related idea, automation complacency, describes the slower erosion of vigilance that sets in when a tool is usually right. The distinction matters for study design. Bias shows up on individual hard cases where the machine is wrong, while complacency shows up as a gradual drop in independent monitoring across many easy cases where the machine is right. Both push in the same direction: the human stops fully checking the machine's output.

The strongest recent experiment

The most direct evidence comes from a randomized study of 44 physicians who had all completed roughly 20 hours of formal AI-literacy training. Each worked through six clinical vignettes with optional access to a large language model. For half of the participants, the model's recommendations were left unmodified; for the other half, the recommendations were deliberately corrupted on three of the six cases. The design isolates the variable that matters, which is not whether AI is present but whether the AI is wrong.

The results are sobering because the participants were, by construction, a best case. Overall diagnostic-reasoning scores fell from 84.9% in the sound-advice group to 73.3% in the flawed-advice group, an adjusted difference of about 14 percentage points. Accuracy of the single top diagnosis dropped further, from roughly 90.5% to 76.1%, an adjusted difference near 18 percentage points. Training on how AI works did not immunize these clinicians against a confident, fluent, wrong answer. The authors also reported exploratory signals that heavier habitual LLM users showed larger performance drops, which is the pattern you would predict if routine reliance dulls independent scrutiny. Those subgroup findings come from a small sample and should be read as hypotheses, not conclusions.

Two design features make this study informative. First, the errors were plausible rather than absurd, which mirrors the real failure mode of modern language models: subtle, well-worded mistakes rather than obvious nonsense. Second, the comparison group received the same tool giving correct advice, so the measured harm is attributable to the erroneous content and the tendency to defer to it, not to the mere presence of a computer.

A converging result in trainees

A separate randomized study of 111 medical students found the same asymmetry in a population with less clinical experience. Misleading AI explanations significantly degraded diagnostic accuracy, while correct AI explanations produced no significant improvement over a no-explanation control. In other words, the downside from wrong guidance was larger and more robust than the upside from right guidance. The study also found that confidence was poorly calibrated: students' stated confidence did not reliably separate their correct answers from their wrong ones, so self-assurance was not a usable signal for when to trust the machine. For learners still building pattern recognition, an incorrect but authoritative-sounding suggestion appears especially costly.

Why fluent systems are harder to resist

A conceptual analysis in the journal AI and Ethics offers a mechanism that fits these numbers. Its attentional-integration account argues that when a clinician offloads part of a task to an automated aid, attentional resources shift away from the decision itself. Situational awareness narrows, and what the authors call learned complacency sets in, so the person monitors the automation's output less closely over time. The concern is not only a single wrong call but a gradual thinning of the vigilance and independent reasoning that make oversight meaningful.

Large language models raise the stakes here. Older clinical tools tended to output a discrete label with a numeric confidence score, which at least flags uncertainty. Language models instead produce polished narrative recommendations that read as expert prose while sometimes embedding a clinically significant error. Fluency is doing persuasive work that accuracy has not earned, and that mismatch is precisely what automation bias exploits.

What the evidence supports, and what it does not

These studies are experimental and mostly use vignettes rather than live practice, with modest sample sizes, so the exact percentages should be read as direction and magnitude rather than fixed constants. What replicates across settings is the qualitative finding: confident, wrong AI advice measurably degrades human decisions, and users do not reliably catch the error on their own. The practical implication drawn by the investigators is structural. Safety depends on workflow design, verification steps, and genuine human oversight, rather than on an assumption that a trained professional will simply notice when the machine is wrong. Evidence that a tool improves accuracy on average is not evidence that it is safe when it fails, and those are separate questions that deserve separate testing.