“A new study proves” — how many times have you seen a headline like that, only to find out that the “study” was an experiment on 20 students at one university? Sample size and confidence intervals are two statistical concepts that determine whether a study’s result can be trusted at all — and yet they almost never make it into the headlines. As a result, a single, small study is sometimes presented in the media as universal scientific truth, even though in reality it says very little about what happens outside the room in which it was conducted.
In this article, I explain step by step what sample size and confidence intervals are, why a small research sample generates a wide margin of uncertainty, and how — without being a statistician — you can judge for yourself whether a given study deserves a headline, or just a footnote saying “further research is needed.”
What sample size is and why it matters
Sample size (denoted by the letter n) is the number of people, animals, cases, or measurements included in a study. Scientists almost never study an entire population — instead, they take a sample and use it to estimatewhat the situation looks like in the whole population. The key word is precisely estimate: every such estimate carries uncertainty, and the size of that uncertainty can be calculated exactly.
According to a basic law of statistics, the standard error of an estimate decreases in proportion to the square root of the sample size (√n). This has a very practical consequence: to halve the uncertainty of a result, the sample has to be not two, but four times larger. That’s why going from a study of 25 people to a study of 100 people doesn’t give a “four times better” result, just an uncertainty interval half as wide — and yet each of these studies can be described in the media with an equally strong headline.
What a confidence interval is
A confidence interval (most often 95%) is a range of values within which — assuming the sample was properly selected — the true value in the whole population lies with a specified probability. When a poll says a candidate has 42% support ± 3 percentage points, it means: we are 95% confident that true support in the whole population lies somewhere between 39% and 45%. That “± 3 points” is the margin of error — half the width of the confidence interval.
The problem is that journalists (and sometimes the authors of press releases themselves) very often leave out the confidence interval and give only a point value — e.g., “42% of Poles believe that…”. Without the confidence interval, the reader has no information about how precise this result is.
__CI_IMG__
Case study: how a small study became a global phenomenon — and fell apart
One of the most instructive examples in the recent history of the social sciences is the story of so-called “power posing.” In 2010, researchers Dana Carney, Amy Cuddy, and Andy Yap published a paper in the journal “Psychological Science” showing that people who adopted an expansive, “powerful” body posture for a minute later had higher testosterone levels, lower cortisol levels, and were more willing to take risks than people who adopted a “weak” posture. The study included just 42 participants.
This result became the basis of one of the most popular talks in TED history (viewed tens of millions of times) and entered popular culture as a “scientifically proven” way to boost self-confidence before a job interview or an important presentation.
The problem arose in 2015, when a team of researchers led by Eva Ranehill replicated this experiment on a much larger sample — 200 participants, almost five times more than in the original study. The results of the replication didn’t confirm any of the hormonal or behavioral effects described in the 2010 paper. In the following years, more than a dozen further independent studies — together covering more participants than almost all of the earlier literature on the subject — also found no effect of “power posing” on hormone levels or risk-taking.
This doesn’t mean the 2010 researchers cheated — it means that an effect observed in 42 people, even if statistically significant in that sample, simply didn’t hold up with a larger, more precise sample. In other words: the small sample size produced a wide, uncertain confidence interval that had room for an effect that isn’t a real phenomenon in the population.
The replication crisis — a systemic problem, not an isolated case
The story of “power posing” isn’t an isolated one. In 2015, an international team of researchers gathered around the Open Science Collaboration project published the results of one of the largest replication projects in the history of the social sciences in the prestigious journal “Science.” The team attempted to repeat 100 psychological studies previously published in three leading scientific journals.
The results were sobering: while 97% of the original studies reported a statistically significant effect, on replication only 36% of themkept their statistical significance. What’s more, even in the studies that were successfully repeated, the average effect size was half as large as in the original study. Analysts point out that one of the main systemic causes of this phenomenon is research samples that were too small in the original experiments, combined with pressure to publish only “positive,” impressive results.
Why small samples more often produce “spectacular” results
Paradoxically, it’s precisely small studies that are statistically more prone to generating very strong, impressive results — not because the effect is stronger, but because with a small sample, random variation has a much greater impact on the final result. A large sample “averages out” random deviations; a small sample lets them dominate the picture. That’s why it’s precisely small studies with very strong conclusions that so often make the front pages — not because of the quality of the evidence, but because of how attractive the result itself is.
How to judge for yourself whether a given study is worth trusting
The checklist below lets you judge in under a minute whether a press article reporting the results of a scientific study deserves your trust, or rather a healthy dose of skepticism.
__CHECKLIST_IMG__
1. Check the sample size
Look in the text for the number “n=” or phrases like “among X participants.” If the article doesn’t give the sample size at all, that in itself is a warning sign.
2. Check whether a confidence interval or margin of error is given
Reliable polls and studies publish a margin of error. Its absence doesn’t always mean bad faith — sometimes it’s just a journalistic simplification — but it’s always worth tracking down the original scientific publication, where such information will be found.
3. Check whether the result has been replicated
A single study, even a large and well-designed one, is still just one data point. Science works cumulatively — only repetition of the result by independent teams builds real confidence.
4. Assess the representativeness of the sample
A large sample that isn’t representative (e.g., only students from one university, or people who volunteered to take part in the study) can still lead to wrong conclusions about the whole population, despite its seemingly impressive size.
5. Compare the headline with the actual strength of the evidence
Ask: does the headline say “may suggest” or “proves”? Reliable studies on small samples are usually described by the authors themselves as preliminary, pilot studies, or as requiring confirmation — it’s the media, simplifying the message, that often turn cautious wording into categorical claims.
Frequently asked questions
Is a small study always worthless?
No. Small pilot studies play an important role in the early stages of science — they let you check whether a given research direction is worth pursuing at all before investing resources in a large-scale study. The problem only arises when the result of a small pilot study is presented publicly as final proof.
What sample size is “enough”?
There’s no single universal number — it depends on how strong and how variable the effect being studied is. That’s why well-designed studies publish a so-called power analysis in advance, which indicates what sample size is needed to detect a given effect with a specified probability.
Does a wide confidence interval mean a study is bad?
Not necessarily — sometimes it’s simply honest, transparent information about the real uncertainty of the measurement. A worse sign is the opposite situation: a result given without any confidence interval at all, suggesting false precision.
Summary
Sample size and confidence intervals aren’t dry concepts reserved for statistics textbooks — they’re practical tools that let you distinguish solid scientific evidence from a passing media sensation. The story of “power posing” and the results of the large Open Science Collaboration project show that even prestigious scientific journals have published results that didn’t survive a confrontation with a larger, more precise sample. Before you believe the next headline beginning with “a new study proves,” check how many people it was conducted on — that one question can save you more disappointment than any other reading habit.