Reclassification Metrics NRI and IDI, and How They Mislead

Net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) were built to measure how much a new marker adds to an existing risk model, beyond a barely-moving area under the ROC curve. They can be useful, but both are easy to inflate: a miscalibrated model, an over-optimistic category-free version, or the wrong confidence interval can make a useless marker look valuable. Read them by asking whether the model was well calibrated, whether events and non-events are reported separately, and whether the reclassification was actually correct.

The problem these metrics were built to solve

When researchers add a promising new biomarker to an established risk model, the area under the ROC curve often barely moves, even for a marker that clearly matters. That flat curve frustrated the field, so statisticians proposed measures that look at whether the new information moves people into more correct risk categories. The two best-known are the net reclassification improvement and the integrated discrimination improvement.

What IDI actually measures

The integrated discrimination improvement compares the two models by their discrimination slope, which is the average predicted risk among people who had the event minus the average predicted risk among those who did not. A better model pushes those two averages further apart. IDI is the increase in that gap when you add the new marker, and it can be read as the average gain in sensitivity minus the average loss in specificity across all possible thresholds.

What NRI counts, and its two flavors

The net reclassification improvement asks a different question: when you switch to the new model, how many people move to a more appropriate risk category? Categorical NRI needs clinically meaningful cutoffs, and it adds the net fraction of events that move up to the net fraction of non-events that move down. Category-free NRI drops the cutoffs and counts any change in predicted risk in the right direction, which sounds cleaner but turns out to be the most easily inflated version.

Where they mislead

A critical review of these indices found several traps. Category-free NRI can overstate a marker's value even in independent validation data, so a positive number is not proof of real added value. The commonly published variance formulas give confidence intervals that are too narrow, so significance can be claimed where none exists, and bootstrap intervals are the safer choice. And with three or more categories, NRI ignores how far people move, treating a trivial shift the same as a large, decision-changing one.

Calibration is the hidden dependency

Both measures assume the models are well calibrated. If the extended model's predicted risks are systematically off, NRI and IDI can look impressive while reflecting miscalibration rather than genuine information. This is why a reader should check calibration first, then treat a reclassification number as a secondary, supporting piece of evidence rather than the headline.

How to read a reclassification claim

Ask three questions. Was the model calibrated in the data where the metric was computed? Are results reported separately for events and non-events, so you can see which group actually improved? And is there evidence the reclassification was correct and clinically meaningful, not just movement? The reviewers' own recommendation is telling: when a single summary is wanted, the improvement in net benefit, the same quantity behind decision curve analysis, is more trustworthy than NRI.