How to read a study without being fooled

A single paper is a claim, not a finding. Understanding what a p-value is not, what happened when researchers tried to replicate a hundred studies, and how "doubles the risk" can mean almost nothing will make you harder to mislead than most journalism.

There's a reason science reporting reads the way it does. Papers are written for reviewers, press releases are written for clicks, and both compress exactly the information you'd need to judge the claim: how many people, studied how, over what period, compared to what, with what size of effect.

You can't fix that by trusting institutions more or less. You can fix it by learning a handful of habits, most of which take about ninety seconds per claim.

The uncomfortable base rate

In 2005, John Ioannidis published a paper titled "Why Most Published Research Findings Are False." It was not a polemic against science; it was a mathematical argument. When studies are small, when effects are modest, when many teams chase many hypotheses, and when journals prefer positive results, a large share of what gets published will be noise dressed as signal. He laid out the conditions under which the false-positive rate climbs above fifty percent, and noted that most fields met them.

The prediction was tested a decade later. In 2015, the Open Science Collaboration repeated one hundred published psychology experiments with high-powered designs. Ninety-seven percent of the originals had reported statistically significant results. When repeated, thirty-six percent did. Effect sizes in the replications averaged about half of the originals.

Not thirty-six percent "wrong." Thirty-six percent produced a significant result again — and the effects were roughly half the size when they did.

Two things follow. First, a single study in a soft field is weak evidence, no matter how prestigious the journal. Second, the direction of a finding is far more likely to survive than its magnitude. If you must take something from one paper, take the sign, not the number.

What actually causes the problem

Four mechanisms do most of the damage, and knowing them is most of the defence.

Underpowered studies

Statistical power is the probability of detecting a real effect if it exists. Conventionally you want eighty percent. Katherine Button and colleagues surveyed the neuroscience literature in 2013 and found a median power of about twenty-one percent — and lower still in some imaging subfields. In an underpowered study, a "significant" result is more likely to be an exaggeration of the true effect, because reaching significance with a small sample requires a lucky large estimate. Small studies don't just fail to find true things; they inflate the ones they do find.

Publication bias

Daniele Fanelli's analyses of thousands of papers found that over ninety percent of articles in psychology and psychiatry reported positive results, versus a much lower rate in fields where negative findings are publishable. If null results sit in drawers, the published literature is a biased sample of all the times the hypothesis was tested, and a meta-analysis that averages only the published ones inherits the bias.

Researcher degrees of freedom

If you can choose when to stop collecting data, which covariates to include, which subgroup to report, and which of several outcome measures is "primary," then any dataset will eventually yield a p-value under .05. Andrew Gelman calls this the garden of forking paths: the researcher doesn't need to intend fraud, only to make reasonable-looking choices at each fork. Preregistration — publishing the analysis plan before seeing the data, and ideally the whole design as a registered report — exists almost entirely to close this door.

The measurement problem

Questionnaires that ask people to rate their own conscientiousness, mood, or sleep quality are not measuring the thing they name in the headline. They're measuring a self-report, which is affected by mood, memory, and what the participant thinks the study wants. This isn't fatal — a lot of useful work uses self-report — but the leap from "scored higher on a scale" to "is actually more disciplined" is where most reporting goes wrong.

The single most useful number in the article

When you see a risk described as a percentage increase, stop and find the base rate. This is the most common way a trivial finding becomes a headline, and it works because relative risk sounds enormous while absolute risk is what actually affects a person.

"Increases risk by fifty percent" could mean an increase from 2 in 100 to 3 in 100 — a real effect, worth knowing about, and nothing like the alarm the sentence produces. It could also mean 2 in 10,000 to 3 in 10,000. Gerd Gigerenzer's work on medical risk communication has shown repeatedly that both doctors and patients misread this format, and that converting to natural frequencies — "three out of every hundred" — removes most of the error.

The mirror rule applies to benefits. A drug that reduces relative risk of a rare event by forty percent may prevent one event per thousand people treated. That can still be worth it. It is a different decision, and it depends on the number the press release didn't print.

Correlation, cause, and the missing variable

Observational studies are the overwhelming majority of what you'll read, and they can only show that two things move together. The question to ask is always: what else moves with both?

The classic teaching example is ice cream sales and drowning deaths, both driven by summer. Less obvious versions are everywhere. People who eat breakfast are healthier — but people with chaotic mornings are also poorer, more stressed, and more likely to be shift workers. Studies that adjust statistically for income and stress can weaken the breakfast effect considerably. A study that adjusts for nothing is describing a correlation among people who differ in a hundred ways.

This is why researchers care about natural experiments, randomised trials, and designs like regression discontinuity or instrumental variables: they create comparisons where the groups are plausibly alike except for the variable in question. When a headline claim rests on an observational association with modest adjustment, the honest reading is "consistent with," not "causes."

What the finding cannot tell you

A study of one population rarely generalises cleanly. Joseph Henrich, Steven Heine and Ara Norenzayan pointed out in 2010 that a huge share of psychology's evidence came from Western, educated, industrialised, rich, democratic populations — samples that are unusual on a long list of measures, including some of the most basic perceptual ones. Findings about what motivates people, what feels fair, and how much choice matters have repeatedly failed to transfer.

Then there's the timescale. A four-week trial tells you about four weeks. Effects that matter most — on learning, health, savings — play out over years, which is precisely the duration that trials are least able to afford. Any long-term claim built by extrapolating a short trial is an argument, not a measurement.

Where to look when you actually need to know

Start with the synthesis, not the latest paper. Systematic reviews and meta-analyses pool studies, exposing whether a result holds across labs and designs. Cochrane reviews are the gold standard for medical questions and are readable in their plain-language summaries. For anything replicated, the pattern is consistent: many small studies show a big effect, then one high-powered study shrinks it, then the field settles somewhere lower.

Check whether the work was preregistered, which you can often do in minutes by searching the authors' names on the Open Science Framework. A preregistered study that confirms its plan is more informative than a larger one that doesn't, because the plan removes the freedom to fish.

And accept that most individual results, even good ones, will be revised. That's not a reason for cynicism; it's the reason the instructional format matters more than any single study. A mechanism supported by converging lines of evidence — different methods, different samples, different eras — is what you're actually entitled to believe.

Updating in proportion

The failure mode on both sides is the same: treating one paper as decisive. Believers add it to the pile; sceptics find the flaw and discard the field. The better practice is ecological — ask how much this study should move your confidence, given what you already knew, and move it that much and no more.

Concretely: a single small observational study on a contested question should move you almost not at all. A preregistered replication in a large sample should move you a lot. A Cochrane review pooling twenty trials should move you most of the way, and then you wait ten years and see.