P-Hacking: The Hidden Choices That Manufacture Significance

P-hacking is what happens when the many small, defensible choices inside a study (which outcome to report, when to stop collecting data, which cases to exclude, whether to adjust for a covariate) get quietly steered toward whatever pushes a p-value below the .05 threshold. In a widely cited 2011 paper in Psychological Science, Joseph Simmons, Leif Nelson, and Uri Simonsohn showed that exploiting just four of these "researcher degrees of freedom" can raise the false-positive rate from the advertised 5% to roughly 61%. The behavior is rarely fraud. It is ordinary motivated reasoning, and the correction is disclosure rather than better intentions, backed by its firmer cousin, preregistration.

The choices that hide inside a p-value

Every empirical study is a chain of decisions. Should we gather more participants? Should we drop the responses that look too fast, or too slow, and where exactly is the cutoff? Which of the measures we collected counts as the outcome? Should the analysis control for baseline age, or sex, or nothing at all? Simmons and colleagues called this latitude researcher degrees of freedom, and they made an uncomfortable observation: for most of these forks there is no single correct answer, which means almost any choice can be justified after the fact. When the person making the choice also wants the result to reach significance, the justifications reliably line up on the significant side. The authors stress that this is not usually malicious. A large literature on motivated reasoning shows that people reach the conclusions they want while sincerely believing they followed the evidence.

The popular name for the pattern, p-hacking, came later. The 2011 paper's contribution was to measure how much damage the ordinary version does.

A false result, reported honestly

To make the point unmissable, the authors ran a real experiment designed to prove something that cannot be true. Twenty University of Pennsylvania undergraduates listened either to "When I'm Sixty-Four" by the Beatles or to a control track. Afterward they gave their birth date, and the analysis controlled for their father's age. The result: people were nearly a year and a half younger after hearing the Beatles song than after the control, F(1, 17) = 4.92, p = .040. A song had, statistically speaking, changed how old the listeners were.

Every step in that sentence is a legitimate analysis honestly reported. What the published version hid were the choices behind it. The team had collected several other measures and reported only father's age as the control; they ran the analysis with that covariate and did not show the version without it; and they had monitored the p-value as data accumulated, stopping once it crossed .05, with no sample size fixed in advance. Report the same study transparently and the effect dissolves: without the covariate it is not significant. The finding was manufactured entirely by undisclosed flexibility.

How 5 percent becomes 61 percent

The experiment is a parable; the simulations are the proof. Simmons and colleagues generated thousands of purely random datasets, containing no real effect at all, and let a hypothetical researcher use common degrees of freedom. Analyzing two related outcomes instead of one nearly doubled the chance of a false positive. "Optional stopping," collecting ten observations per group and then peeking after every additional subject until significance appeared, produced a spurious result about 22% of the time. Adding a covariate, or dropping one of three conditions, each added more. Combine all four moves in a single study and the false-positive rate reached about 61%. A researcher using these routine tools was more likely to find a significant effect that was not real than to correctly report that nothing was there.

This is why a lone significant p-value, stripped of context, carries so little information. John Ioannidis had argued a few years earlier that findings are less trustworthy wherever there is "greater flexibility in designs, definitions, outcomes, and analytical modes" (PLoS Medicine, 2005). The False-Positive Psychology paper turned that warning into a number.

The remedy is disclosure, then preregistration

The authors' fix is deliberately modest. They proposed six requirements for authors: decide and report the data-collection stopping rule before starting; collect at least twenty observations per cell or justify fewer; list every variable collected; report every experimental condition, including failed ones; if any observations are excluded, also report the result with them included; and if a covariate is used, also report the result without it. Four parallel guidelines ask reviewers to enforce that transparency and to stop rewarding suspiciously perfect results. None of this forbids exploration. It only asks that exploratory work be labeled as such rather than dressed up as a confirmed prediction.

Disclosure has a limit the authors acknowledged: it cannot reveal the studies that were run and quietly filed away. That gap is what preregistration closes. As Brian Nosek and colleagues describe it (PNAS, 2018), writing down the hypothesis and the analysis plan before seeing the outcomes separates genuine prediction from after-the-fact storytelling, so a confirmatory claim can be told apart from an exploratory one. Registered reports and public preregistration have since spread across psychology, medicine, and clinical trials for exactly this reason.

Reading a study with this in mind

For anyone weighing a headline finding, a few questions do most of the work. Was the analysis preregistered, and does the paper report that plan? Does the outcome match the one the study was designed to measure, or a substitute that appeared later? Are sample size, exclusions, and covariates stated plainly, with the alternative versions shown? A study that answers these openly has earned more trust than a tidier one that stays silent. The lesson of the Beatles experiment is not that researchers are dishonest. It is that a p-value means little until you know how many roads were available to reach it. This is a guide to appraising evidence, not medical advice.