When Not to Pool: Deciding a Meta-Analysis Would Mislead

Not every systematic review should end in a single pooled number. When the studies differ too much in who they enrolled, what they did, or how they measured outcomes, forcing them into one average can manufacture false precision and hide a real disagreement. In those cases a careful review uses a structured synthesis without meta-analysis, describing the pattern of results transparently instead of collapsing them into a misleading summary effect.

Pooling is a decision, not a reflex

It is tempting to treat a systematic review as a machine that must end in one number. Collect the studies, run the meta-analysis, report the pooled effect. But combining studies is a judgment call, and sometimes the honest judgment is not to combine them at all.

A summary effect built from studies that have little in common can look precise and authoritative while describing nothing that exists in the real world. The number is real; what it represents is not.

The three kinds of heterogeneity

Reviewers weigh three sorts of difference before pooling. Clinical heterogeneity is variation in the participants, interventions, and outcomes, whether the studies are really about the same thing. Methodological heterogeneity is variation in how the studies were designed and run, such as risk of bias or length of follow-up. Statistical heterogeneity is the observable scatter in the results themselves, often summarized with the I squared statistic.

Statistics can flag a problem, but the deeper question is clinical. If the studies enrolled different diseases or tested different treatments, a small I squared does not rescue the comparison, and a large one is a symptom rather than the disease itself.

What breaks when you pool anyway

Force diverse studies into one average and two things can go wrong. The first is false precision. The pooled estimate arrives with a narrow confidence interval that implies a certainty the underlying evidence does not support.

The second is concealment. A genuine split, a treatment that helps one population and does little in another, can average out to a bland middle that matches neither group. The single number does not resolve the disagreement; it buries it under a tidy result.

The honest alternatives

When a summary effect would mislead, a review still has rigorous options, set out in the reporting guideline for synthesis without meta-analysis and in the Cochrane guidance on other synthesis methods. A team can summarize the effect estimates and their direction across studies, in a structured and transparent way, without claiming a single pooled value.

Where variances are missing, they can combine p values to ask whether there is evidence of an effect in at least one study. They can count studies by the direction of their effect. Each of these is a defined method with known limits, not a vague narrative dressed up as analysis.

The vote-counting trap

One popular shortcut deserves a clear warning. Counting how many studies reached statistical significance, and declaring that side the winner, is a discredited method. It quietly rewards large studies and penalizes small ones regardless of the actual effect sizes, and it can point in the wrong direction.

Counting by the direction of effect, setting significance aside, is the defensible version of the idea, and even that is a weak tool best reported honestly as such. A review that leans on significance counting is signaling that it skipped the harder question of whether the studies belonged together.

How to read a review that chose not to pool

When you meet a review with no forest-plot diamond, do not assume it failed. Ask whether the authors explained why they did not pool, and whether they named the synthesis method they used instead.

A review that says plainly that the studies were too diverse to combine, and then lays out the pattern of results along with its limits, is doing better science than one that produced a neat number by ignoring the differences. The refusal to pool, stated clearly, is often the more trustworthy move.