When More Data Is Not Better: Why a Bigger Sample Can Be Confidently Wrong

A bigger dataset is not automatically better evidence, because size fixes one kind of error and is helpless against another. More data shrinks random noise, so the answer becomes more precise. It does nothing about bias, so if the data was tilted to begin with, the answer stays wrong and extra rows only make it look sharper. The most dangerous result in my field is not a small study with wide error bars. It is a vast study with tight ones pointing the wrong way, because a tight error bar reads as confidence to almost everyone who sees it. This is a method explainer, not medical advice; decisions about your own care belong with your own clinician.

What is the difference between precision and accuracy?

Accuracy is whether you are aiming at the right target. Precision is how tightly your shots cluster, wherever they land. The two are independent, which is the whole story: a dataset can be enormously precise and badly inaccurate at once.

Picture an archer with a bent sight. Every arrow lands in a tight group, a hand's width left of the bullseye, and handing him more arrows tightens the group without ever drifting it toward the center, because shooting more does nothing about the bent sight. A large biased sample is that archer. The cluster is the confidence interval, narrowing as the sample grows. The distance from the center is the bias, untouched by any amount of extra shooting. We reach for more data when an estimate is jumpy, and that instinct is right when the trouble is random scatter. It misfires when the trouble is systematic, because then more data buys precision around the wrong number, which is worse than honest uncertainty.

Why does a biased dataset get more dangerous as it grows?

Bias does not shrink with sample size. That sentence undoes a lot of intuition. We are taught that estimates improve as we collect more, and they do, but only the random part improves while the systematic lean built into how the data was gathered sits unmoved.

Here is the quotable version. Random error averages out as a sample grows; systematic error does not, so a large biased sample produces a precise estimate of the wrong quantity. A small biased study at least has wide intervals that overlap the truth and invite doubt. Scale that study up and the intervals pull away from the truth while the result starts to look authoritative. The bias was always there. Size simply stripped away the uncertainty that had been giving the truth somewhere to hide.

This is confidence outrunning correctness. We all read a narrow interval as a strong result, treating precision as a proxy for being right. With a clean sample that shortcut works; with a tilted one it inverts, and the most precise number in the room becomes the one most likely to mislead.

How does this show up in clinical AI?

Modern clinical models are trained on the largest datasets medicine has ever assembled, and scale gets sold as the reason to trust them. The pitch is that volume confers reliability. Volume confers precision. Whether it confers reliability depends on whether those records represent the patients the tool will actually meet, and they often do not, because the data was generated by a care system with its own habits about who gets seen, tested, and coded.

When I review a model with a stunning performance figure, my first question is not how the number was computed but where the data came from and who is absent. A held-out test set is carved from the same source as the training data, so it inherits the same tilt, and model and evaluation can both describe a population the clinic does not contain.

A worked example: the confidently wrong screen

Suppose a screening model is trained on a huge archive from a handful of large referral centers. Patients reach those centers because someone already suspected disease, so the archive is dense with sicker, more thoroughly worked-up cases than a community clinic ever sees. The model learns that referred population superbly, and because the dataset is huge, every metric arrives with a tight interval and the report radiates certainty.

Deploy it where most people actually live, in primary care, and it meets earlier, milder, differently documented patients. It is precisely calibrated to a world that clinic is not, so it will flag and reassure with the same crisp confidence it showed in validation, and be wrong more often than the numbers ever warned. Had the sample been smaller, someone might have paused at wide error bars. Size removed the hesitation.

Does this mean small studies are better?

No, and that is the trap on the other side. The answer to a precise wrong number is not a vague right one. A tiny sample is noisy and easy to fool by chance, which is exactly what more data fixes. The point is sequence, not size. First ask whether the sample is aimed at the right target, then ask how tightly it clusters. Get the aim right and more data is among the best things you can have.

How do you tell precision from accuracy in practice?

You cannot read accuracy off a confidence interval, because the interval describes scatter, never aim. So you interrogate the source instead of the statistic. How did each record come to exist, and is that path connected to the outcome being measured. Who could have been in this dataset and is not. Has the result been checked against a different source collected a different way, since agreement across methods is a real test of accuracy a single dataset can never supply on its own.

The deepest tell is what a narrow interval does to the people reading it. A tight result ends conversations, so the room stops asking where the data came from precisely when that question matters most. Some of my own research has examined how a physiological relationship can differ across populations rather than holding as one curve, and the lesson generalizes: a result averaged over the wrong mix of people can be precise and still describe no one in the room.

What this means for reading evidence

Hold two ideas at once. Size is a real strength, and it is the answer to noise, not to bias. Give a large dataset full credit for the precision scale earns, then withhold judgment on accuracy until you understand how the data was assembled. None of this is a knock on the people building these systems, since representative data is hard to gather when care itself is uneven. What is not defensible is letting size stand in for whether the data was aimed correctly. The error bar tells you how steady the hand was. It says nothing about whether the sight was bent, and a steady hand on a bent sight is how good evidence goes wrong.