Prespecified or Post Hoc: Why the Timing of an Analysis Decides How Much to Trust It

An analysis written into the protocol before anyone saw the results is prespecified; one chosen after looking at the data is post hoc. The distinction matters because the eye is drawn to whatever pattern happens to look striking, and testing many possibilities after the fact manufactures false positives. Prespecified findings are meant to be confirmatory, while post hoc findings are hypothesis generating, and a careful reader keeps the two in separate mental boxes.

What prespecification means in practice

To prespecify an analysis is to write down, before the data are unblinded, exactly what you will measure and how you will test it. That includes the primary outcome, how it is defined and when it is assessed, the statistical test, and the way missing data and multiplicity will be handled. The point is not bureaucracy. It is that a plan fixed before the results are visible cannot be shaped by them.

Prespecification is what lets a test count as a real test. Once you have seen the data, almost any analysis you choose is influenced, consciously or not, by the patterns already in front of you. Committing to the plan in advance is the discipline that keeps a confirmatory claim honest.

Why post hoc findings are easy to fool yourself with

A post hoc analysis is one decided after looking at the data, and the danger is not dishonesty; it is human vision. The eye is drawn to whatever looks striking, and in any real dataset something always does. If you are free to pick the outcome, the subgroup, and the cut point after seeing the numbers, you can nearly always find a combination that reaches significance, even in data with no true effect.

One case makes the scale of the problem vivid. When researchers took a single trial's outcome and analyzed it under eight defensible definitions, the odds ratios ranged from about 0.23 to 0.94 and the p values ran from below 0.001 to 0.89. Same patients, same event, very different verdicts, turning only on a choice made after the fact. That is the freedom a post hoc analysis quietly enjoys.

Outcome switching: the quiet version

The most common way prespecification fails is not a dramatic fishing expedition but a quiet substitution. Outcome switching is when the primary outcome named in the protocol or registry is changed, dropped, demoted, or joined by a new one by the time the paper appears. It often looks innocuous, and it is common.

It is also not neutral. A study comparing trial registrations with the eventual publications found that about a third of trials had at least one primary outcome change, and that trials which switched reported effect sizes roughly sixteen percent larger on average. Statistically significant outcomes are more likely to be reported fully than nonsignificant ones. The net effect is a published literature that looks more positive than the underlying trials actually were.

How registries and protocols make timing checkable

The reason any of this is verifiable is that trials are meant to be registered before they begin and to publish their protocols. A public registry entry timestamps the primary outcome and the analysis plan before the results exist, so a reader can compare what was promised with what was delivered.

This is why prospective registration became a condition of publication in many journals and a legal expectation in several jurisdictions. It does not stop outcome switching, but it makes it visible. When a paper's primary outcome differs from its registered one with no explanation, the registry is the document that lets you notice the gap.

Confirmatory versus hypothesis generating

The cleanest way to hold all of this in mind is to sort every result into one of two buckets. A confirmatory finding is prespecified, tested inside a controlled analysis plan, and meant to support a firm conclusion. A hypothesis generating finding is exploratory or post hoc, valuable for suggesting what to study next but not for settling the question.

Neither bucket is bad. Exploratory analysis is how good ideas are found, and dismissing it would be foolish. The error is category confusion: treating a post hoc, exploratory result as if it were a confirmatory one. A great deal of overreaching in medical claims is, at heart, that single mistake.

How to check timing as a reader

You can check the timing without any statistics. Look for whether the paper names a single prespecified primary outcome and whether it matches the registry entry, which is usually linked or searchable by the trial's identifier. Notice the language: words like prespecified and primary carry weight, while exploratory, post hoc, and subgroup are honest flags that a finding is a lead rather than a conclusion.

Then ask the plain question behind all of it. Was this analysis planned before the data were seen, or chosen after? If the paper does not make that answer easy to find, that in itself is worth noting.