How to Read a Calibration Plot, and Why a Confident Model Can Still Be Wrong

A calibration plot answers one question: when a model says thirty percent, does the event actually happen about thirty percent of the time? Predicted probability runs along the horizontal axis, observed frequency up the vertical axis, and a perfectly calibrated model traces the diagonal from corner to corner. The reason a confident model can still be wrong is that confidence and correctness are separate properties. A model can sort people beautifully, putting the sicker ahead of the healthier, while attaching numbers to that ordering that are systematically too high or too low. The plot is where that gap becomes visible.

What the plot actually shows

Start with the axes, because they are easy to confuse. The horizontal axis is what the model claims: the probability it assigns to each case. The vertical axis is what reality delivers: among cases given a similar predicted probability, the fraction in which the event occurred. To build the curve, you group predictions into bins or fit a smooth flexible line, then plot the average prediction in each group against the average outcome in that group.

The diagonal line, where predicted equals observed, is the target. Points that sit on it mean the numbers can be taken at face value. Points that drift off it mean the numbers need translation before anyone acts on them.

Two departures matter, and they look different on the page. When the curve sags below the diagonal, the model is overconfident: it predicts high probabilities that outcomes do not support. It says eighty percent when the truth is closer to sixty. When the curve rises above the diagonal, the model is underconfident: it predicts low and the world delivers higher. Often a single model does both, overstating risk at the top and understating it at the bottom, which produces a curve flatter than the diagonal, like a line that has been pressed toward the horizontal.

Two numbers that summarize the picture

Reading the shape by eye is useful, but two summary measures make the assessment precise, and they answer different questions.

Calibration-in-the-large is the coarsest check. It asks whether the average predicted probability across everyone matches the overall event rate. If a model predicts a mean risk of ten percent in a population where the event happens fifteen percent of the time, the whole set of predictions is shifted, regardless of how well individuals are ranked. This is the first thing to break when a model trained in one setting is dropped into another where the underlying rate differs. The math has not changed; the base rate has.

The calibration slope describes how the predictions spread out. Think of it as the slope of the line relating predicted to observed risk. A slope of one is ideal. Below one is the signature of overfitting: extreme predictions are too extreme in both directions, high risks overstated and low risks understated, so the fitted line tilts flatter than the diagonal. Above one, which is less common, means predictions are too timid and could be pushed further from the average. Taken together, calibration-in-the-large and the slope tell you whether the model is off, in which direction, and for whom.

Why good discrimination is not enough

Here is the trap that gives this topic its bite. A model can discriminate well and still be badly calibrated. Discrimination, often summarized by a concordance statistic, measures ranking: given one person who had the event and one who did not, how often does the model score the first higher? That is a question about order. Calibration is a question about level. You can preserve order perfectly while sliding every number up or down.

Picture a model that assigns risks that are all exactly double the truth. Everyone who should be at ten percent is labeled twenty, everyone at thirty is labeled sixty. The ranking is untouched, so discrimination looks excellent. Yet every probability is wrong by a factor of two, and any decision that keys off a probability threshold will fire at the wrong moment. A discrimination metric alone would never reveal this. Only the calibration plot does.

This is why I treat calibration and discrimination as two required readings rather than interchangeable ones. A screening tool with strong ranking but a slope well under one will flag people whose stated ninety percent risk is really nearer fifty. The tool is not useless. It is mislabeled, and the label is what a clinician reads off the screen.

How miscalibration misleads a decision

Probabilities earn their keep when they meet a threshold. Start a treatment above a certain risk. Order a confirmatory test. Escalate monitoring. Each of those rules assumes the number means what it says. When the curve departs from the diagonal near the threshold, the decision moves even if accuracy overall looks fine.

An overconfident model near a treatment threshold pushes people over the line who do not belong there, inviting overtreatment and its harms. An underconfident model in the same region leaves people below the line who should have crossed it, and they go untreated. The average error can look small while the error precisely where you make choices is large. That is why I look hardest at the part of the curve nearest the decision point, not at the fit as a whole.

A practical way to read one

When a calibration plot lands in front of you, work through it in order. Check calibration-in-the-large first: does the curve sit around the diagonal overall, or is the whole thing shifted? Read the slope next: is the curve parallel to the diagonal, or flattened toward horizontal? Then look local: near the probabilities where a real decision gets made, is the model over or under? Finally, ask where the data came from, because a model calibrated in its development sample often drifts once the population or the base rate changes, and calibration tends to be the property that degrades first.

This piece is educational and not medical advice; decisions about your own care belong with your own clinician. The broader point holds for any predictive tool. A confident number is a claim, and the calibration plot is how you check whether reality has agreed to honor it.