A Prediction Model Can Be Accurate and Still Not Help: The Case for Impact Studies

Showing that a prediction model discriminates and calibrates well tells you it makes accurate estimates, not that using it helps anyone. A model earns its place only when an impact study, ideally a randomized comparison, shows that acting on it improves decisions or outcomes. A perfectly accurate model can still be useless if clinicians already judge well, if it changes no management, or if no effective action follows.

Accurate is not the same as useful

A prediction model goes through development, where it is built, and validation, where it is shown to discriminate and calibrate in new data. Passing both is necessary and reassuring, but it answers only one question: does the model produce accurate risk estimates? It does not answer the question patients care about, which is whether using the model leads to better decisions and better outcomes.

What an impact study is

An impact study compares what happens when the model is used against what happens when it is not. The strongest design randomizes clinicians or clinics to have the model or not, and then measures downstream decisions, patient outcomes, costs, or resource use. Weaker before-and-after designs can be swamped by trends over time. The point is that impact is measured on care and outcomes, not on the model's own statistics.

Why an accurate model can still fail

Several things can break the link between accuracy and benefit. If clinicians already estimate risk well without the tool, a model that agrees with them changes nothing. If the model flags a risk for which no effective action exists, knowing the risk does not help. If the output arrives at the wrong moment or is ignored, it never touches a decision. And if acting on the model carries its own harms, the net effect can even be negative.

The bridge from accuracy to value

Between validation and a full impact trial sits a useful halfway analysis: net benefit, the quantity behind decision curve analysis. It weighs the benefit of true positives against the harm of false positives at the risk threshold where a real decision flips, and it can suggest whether a model is likely to help before anyone runs a costly trial. It is not a substitute for measuring impact, but it filters out models that cannot plausibly earn their keep.

Reading the claim behind a model

When a model is promoted as ready for practice, ask what stage of evidence supports it. Development alone is the weakest claim. External validation is stronger. Evidence that using it actually improved decisions or outcomes is the real prize, and it is far rarer than the marketing implies. A model can be genuinely accurate and still be waiting for the study that would show it helps.