Bayesian and Frequentist Results: Two Different Questions a Trial Can Answer

A frequentist analysis asks how often you would see data this extreme if the treatment truly did nothing, which is what a p value and confidence interval summarize. A Bayesian analysis starts from a stated prior belief, updates it with the trial data, and reports the probability that a benefit is real and roughly how large it is. Neither is more correct; they answer different questions, so the first thing to check is which question a given result is actually answering.

The question each framework asks

Start with what the two approaches are trying to compute, because that is where they part ways. A frequentist analysis treats the true treatment effect as a fixed but unknown number and asks a question about the data: if this treatment really did nothing, how often would a trial produce a result at least this extreme? The p value is the answer to that question, and the confidence interval is the range of effects the data do not rule out under repeated sampling.

A Bayesian analysis turns the arrow around. It treats the unknown effect as something you hold a probability distribution over, starts from a prior distribution that captures what was believed before the trial, and uses the data to update that belief into a posterior distribution. The output is a direct statement about the effect itself, such as the probability that the treatment helps, or the probability that it helps by more than some chosen amount.

What a prior is, and why it is not cheating

The prior is the part that makes people uneasy, because it sounds like putting a thumb on the scale. In practice a well conducted Bayesian analysis does the opposite of hiding assumptions: it states them and then tests how much they matter. Priors range from skeptical, which lean toward no effect and demand strong data to be moved, to enthusiastic, which start closer to benefit.

The honest way to present a Bayesian result is to show the posterior under a range of priors, including a deliberately skeptical one. If the conclusion holds even when you begin from doubt, that is persuasive. If it appears only under an optimistic prior, the prior is doing the work rather than the data, and you should read the claim in that light.

Posterior probability versus the p value

A p value is easy to misread as the probability that the treatment does not work. It is not. It is the probability of data like these under the assumption that the treatment does nothing. The Bayesian posterior is the quantity people often thought the p value was: the probability that the treatment works, given the data and the prior.

That difference changes how a result reads. A frequentist trial reports whether it crossed a significance threshold. A Bayesian trial can say something closer to the decision a reader faces, such as a 96 percent probability of any benefit and a 90 percent probability of a benefit big enough to matter.

Credible intervals versus confidence intervals

The two frameworks also produce similar looking intervals that mean different things. A 95 percent confidence interval is a frequentist object: if you repeated the trial many times, about 95 percent of the intervals built this way would contain the true effect. It does not say there is a 95 percent probability that this particular interval contains the truth, even though it is constantly read that way.

A 95 percent credible interval is the Bayesian version, and it does carry the intuitive meaning: given the data and the prior, there is a 95 percent probability that the effect lies inside it. When a paper reports a credible interval, you are finally allowed to read it the way most people already read confidence intervals.

A trial that was called negative but was not

A concrete case shows why this matters. In a trial comparing two resuscitation strategies in septic shock, the difference in twenty eight day mortality favored the newer approach, but the frequentist p value came in at 0.06, just past the usual line, so the trial was widely filed as negative. A Bayesian reanalysis asked a different question: given these data, how probable is it that the strategy reduces mortality?

Across a range of priors, the posterior probability of benefit stayed above 90 percent. The data had not changed. What changed was the question, from did we cross a threshold to how likely is a benefit, and the second question gave a less discouraging and arguably more useful answer. That is the practical payoff of reading both framings rather than treating one threshold as a verdict.

How to read a Bayesian claim critically

Bayesian results are not automatically more trustworthy; they can be spun too. When you meet one, ask a few plain things. What prior was used, and was a skeptical prior among those tested? Is the reported probability for any benefit at all, which is a low bar, or for a benefit big enough to matter clinically? And does the analysis describe its model rather than quoting one comforting number?

Held to those questions, the two styles of reporting complement each other. The frequentist p value guards against being fooled by noise; the Bayesian posterior tells you how probable the benefit is and how big. A reader who can hold both is much harder to mislead than one who knows only the threshold.