Standardized Mean Differences: Comparing Studies That Used Different Scales

A standardized mean difference, or SMD, divides the gap between two group means by a standard deviation, turning a result measured in the units of one questionnaire into a unit-free number that can sit beside a result from a different questionnaire. That is its whole reason to exist: when several trials measure the same underlying construct, say depression or pain or physical function, on instruments that do not share a scale, an SMD puts them on common ground so a meta-analysis can pool them. Cohen's d and Hedges' g are the two you will meet most often. They are useful, and they mislead in specific, learnable ways. This article is general education about reading evidence, not medical advice; any decision about your own care belongs in a conversation with a qualified clinician.

Why trust this walk-through? Dr. Damon Tojjar is a physician-scientist whose peer-reviewed meta-analysis of ethnic differences in insulin sensitivity, published in Diabetes Care, had to reconcile effects reported in different units across many studies. Pooling forces you to be honest about what a standardized number does and does not carry.

What a standardized mean difference actually is

Picture two trials of the same kind of therapy for the same problem. One measures the outcome on a scale that runs from 0 to 27; the other uses a scale from 0 to 63. A raw difference of "4 points" means something entirely different in each, so you cannot average 4 and 4 and learn anything.

The SMD fixes this by rescaling. Take the difference between the treatment and control means, then divide by a standard deviation drawn from the study. The standard deviation measures how spread out the individual scores were, so dividing by it re-expresses the effect not in points but in how many standard deviations apart the groups were. An SMD of 0.5 means the group means differed by half a standard deviation, whatever the original scale. The two trials now speak the same language, and a pooled estimate becomes possible.

Cohen's d and Hedges' g, briefly

Cohen's d is the basic version: the mean difference divided by a pooled standard deviation of the two groups. It does its job well in large samples. In small ones, though, d carries a known upward bias, tending to report the effect as slightly larger than the truth.

Hedges' g applies a small correction factor that removes most of that small-sample bias. In large studies the two are nearly identical; in small ones g is the more honest choice, which is why most careful meta-analyses report it. The distinction is real, but it is not what causes the deeper trouble. That lives in the denominator.

A rough interpretation, held loosely

A convention that has traveled far calls an SMD near 0.2 small, near 0.5 medium, and near 0.8 large. Treat it as a reasonable first orientation, no more. The author who proposed that rubric offered it as a rough guide for fields with no better benchmark, and warned against treating the labels as fixed law.

Hold them loosely because a standardized number strips out context on purpose, and context is often what a decision needs. A "small" 0.2 on a widely used outcome affecting millions can matter enormously at the population level. A "large" 0.8 on a scale nobody recognizes may mean a change no patient would notice. When you can, convert an SMD back into the units of a familiar instrument, so a clinician can judge whether the change crosses a meaningful threshold.

Where the standardized mean difference misleads

The elegant part of the SMD is also its weakness: the answer depends on the standard deviation you divide by, and that number is a property of the sample, not of the treatment.

Different populations, different spread. The same treatment with the same real effect can yield a larger SMD in one trial than another only because the second enrolled a more variable group. A wide, heterogeneous sample has a large standard deviation, so a fixed effect divided by it looks smaller. A narrow, carefully selected sample has a small one, so the identical effect looks larger. Pool SMDs across trials that differ in who they enrolled, and some of the spread between studies is not disagreement about the treatment. It is disagreement about the denominators.

Restricted range. This is the sharpest version of the same problem. If a trial recruits only people within a narrow band, say only those with moderate baseline severity, the standard deviation shrinks because the extremes were screened out. A shrunken denominator inflates the SMD. The treatment did not become more powerful; the measuring stick got shorter. Broad eligibility does the reverse, widening the spread and deflating the SMD. Two trials can report honest, correctly computed effect sizes that disagree mostly because their entry criteria differed.

Measurement reliability. A noisy instrument, one that would give somewhat different scores if the same stable person were measured twice, inflates the observed standard deviation and pulls the SMD toward zero. So an effective treatment measured with a shaky scale can look weaker than the same treatment on a precise one. The instrument's quality leaks into a number meant to be about the treatment.

Direction and sign. A quieter hazard: scales do not all point the same way. On some, higher is better; on others, higher is worse. Pooling without first aligning the direction of every scale can turn a real benefit into apparent harm. This is bookkeeping rather than statistics, and a common source of error.

How to read a pooled SMD without being fooled

A few questions do most of the work. Ask which standard deviation the authors divided by, and whether the choice was consistent across studies. Ask whether the trials enrolled similar people, since differences in range quietly move the numbers. When a forest plot of SMDs shows high heterogeneity, resist reading all of it as real conflict about the treatment, because part can come from the denominators. And ask whether the team translated the pooled SMD back into a familiar clinical scale, since that is where a standardized number becomes human.

Keep the model small. An SMD rescales an effect into standard-deviation units so different instruments can be compared, Hedges' g is the form to prefer in small samples, the 0.2, 0.5, 0.8 labels are a first glance rather than a verdict, and the whole thing rests on a standard deviation that belongs to the sample, not the treatment. Hold that, and a wall of standardized effect sizes stops looking authoritative and starts looking like something you can question.