Propensity Scores: What They Balance, What They Miss, and How to Read One

A propensity score is the estimated probability that a person receives the treatment being studied, given their measured characteristics. Researchers use it to make treated and untreated groups comparable on those measured characteristics, through matching, weighting, or stratification, so an observational comparison behaves a little more like a randomized one. The essential limitation is that a propensity score can only balance what was measured, so it does nothing about unmeasured confounders, and the honest way to read one is to check the balance it achieved and remember what it could never see.

The one number it collapses

Imagine trying to compare two groups that differ in dozens of ways: age, sex, disease severity, other conditions, and more. A propensity score collapses all of those measured differences into a single number, the estimated probability that a given person would receive the treatment.

Two people with the same propensity score are, on the measured characteristics, similar in how likely they were to be treated. That single number is what lets researchers rebuild comparable groups out of messy observational data.

Four ways studies use it

There are four common ways to put a propensity score to work. Matching pairs each treated person with an untreated person of a similar score. Weighting, often called inverse probability of treatment weighting, gives each person a weight so that the treated and untreated groups come to resemble the population of interest. Stratification sorts people into bands of similar scores and compares within each band. Covariate adjustment enters the score into an outcome model.

Each approach has its own trade-offs, but all of them share the same goal of balancing the measured characteristics between the groups.

Balance is the thing to check, not prediction

A frequent misunderstanding is that a propensity model should predict treatment accurately. It should not be judged that way. The test of a propensity score is whether, after matching or weighting, the two groups actually look alike on their measured characteristics.

Careful studies report this with standardized mean differences for each variable before and after adjustment, and a difference under about one tenth is usually taken as good balance. If a paper shows you the balance table, it is doing the work. If it only shows the model, you cannot tell whether the groups ended up comparable.

The blind spot: unmeasured confounding

A propensity score can only balance the characteristics that were measured and entered into it. Anything that was not recorded, such as frailty that never made it into the chart or a subtle symptom that nudged a doctor toward one treatment, remains unbalanced.

This is the deep limit that separates even the best propensity analysis from a randomized trial, where randomization balances the unmeasured factors too. A trustworthy study states which confounders it could not measure and considers how much they might have mattered.

Common ways it goes wrong

Beyond unmeasured confounding, a few problems recur. There must be genuine overlap, meaning some treated and some untreated people at each level of the score, or the comparison is built on people who have no counterpart in the other group.

Variables that are actually consequences of treatment, rather than causes of it, should not go into the score, because adjusting for them introduces bias. And the timing of when the score is measured matters, since mixing up the order of treatment and follow-up can smuggle in immortal time bias.

Reading a propensity-score study

Look for a balance table with standardized mean differences after adjustment, not just a description of the model. Check that the authors discuss overlap and what happened to people who could not be matched.

See whether they name the confounders they could not measure and probe how sensitive their result is to that gap. A propensity score is a serious tool, and the studies that use it well tell you exactly what it did and did not fix.