Quantifying Heterogeneity: What I-Squared and Tau-Squared Each Really Mean

When trials in a meta-analysis disagree, reviewers reach for statistics to describe that disagreement, and the two most common ones answer different questions. I-squared is a proportion, the share of the variation between study results that is more than chance, while tau-squared is an amount, the estimated spread of the true effects on the scale of the outcome. Because I-squared depends on how precise the included studies are, a high value does not by itself mean the effects differ in a way that matters, so tau-squared and a prediction interval tell you more about real-world spread.

Two questions: is there heterogeneity, and how much

Studies pooled in a meta-analysis rarely agree perfectly. Some of that scatter is just sampling noise, the ordinary luck of who ended up in which trial. Some of it is real: the true effect genuinely differs between populations, doses, or settings. Untangling those is the whole point of measuring heterogeneity.

There are really two separate questions. Is there any real heterogeneity beyond chance, a yes or no? And if so, how much is there, a matter of degree? The common statistics do not all answer the same question, and readers get into trouble by treating one as if it answered the other.

Cochran's Q and why its p-value misleads

The oldest tool is Cochran's Q, which adds up how far each study sits from the pooled estimate, weighted by precision, and compares that total against what chance alone would produce. It yields a p-value for the null hypothesis that all studies share one true effect.

That p-value is treacherous. When only a few small studies are pooled, Q has little power, so real heterogeneity slips past as nonsignificant. When many large studies are pooled, Q becomes so powerful that it flags tiny, unimportant differences as statistically significant. So a nonsignificant Q does not mean the effects agree, and a significant Q does not tell you the disagreement matters. Q answers the yes-or-no question, poorly, and cannot speak to degree at all.

I-squared is a proportion, not an amount

I-squared was designed to be more interpretable. It expresses the percentage of the total variation in study estimates that is due to genuine heterogeneity rather than chance. Rough conventions read low, moderate, substantial, and considerable bands as the value climbs, and those labels have become a reflex in how reviews are summarized.

The reflex hides a crucial point: I-squared is relative. It is a ratio of real variation to total variation, so it depends on how precise the studies are. Pool a set of enormous, very precise trials, and even trivial true differences fill up the numerator and push I-squared toward high values. Pool small, noisy trials, and real heterogeneity can be swamped by sampling error, leaving I-squared deceptively low. A big I-squared therefore does not mean the effects are far apart in any way you would care about clinically. It means real differences make up a large share of a total that itself depends on study size.

Tau-squared measures the spread on the effect scale

Tau-squared answers the how-much question directly. In a random-effects model it is the estimated variance of the true effects across studies, and its square root, tau, is their standard deviation, expressed in the units of the effect measure such as the log odds ratio or the mean difference.

Because it lives on the scale of the outcome, tau-squared is an absolute quantity. It does not shrink or swell just because your studies happen to be large or small. Two meta-analyses with identical tau-squared describe the same real spread of effects, even if one shows a high I-squared and the other a low one because their studies differ in precision. That is why methodologists increasingly ask for tau-squared, and for what it implies, rather than leaning on I-squared alone.

Putting them together

The most useful expression of tau is the prediction interval, which uses the between-study spread to show the range within which the effect in a new, similar setting is likely to fall. A pooled estimate can carry a reassuringly narrow confidence interval while the prediction interval, widened by tau, reveals that the next setting could see anything from clear benefit to none.

When you read a meta-analysis, do not stop at the I-squared label. Ask whether the review reports tau-squared or a prediction interval, and whether it investigated the sources of heterogeneity rather than merely quantifying it. Numbers describe the disagreement; they do not explain it. The reviews worth trusting treat a high heterogeneity statistic as the start of an inquiry into why the studies differ, not as a box to tick before pooling anyway.