Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

OBJECTIVES: To show the effects of chance on meta-analyses, and the potential dangers of being prompted to do a meta-analysis by one favourable trial. DESIGN: In total, 100,000 trials were simulated and combined into 10,000 meta-analyses, using data from the control group of a cancer trial. Each participant record was randomly coded to simulate allocation to 'treatment' or 'control'. SETTING: Simulated study. PARTICIPANTS: De-identified records for 578 patients from the control group of a cancer trial, of whom 147 had died. MAIN OUTCOME MEASURE: Time to death from any cause. RESULTS: Of the 100,000 trials, 4897 (4.9%) were statistically significant at 2p < 0.05 and 123 (1.2%) of the 10,000 meta-analyses were significant at 2p < 0.01. The most extreme result was a 20% reduction (99% CI: 0.70-0.91; 2p = 0.00002) in the annual odds of dying in the 'treatment' group. If a meta-analysis contained at least one trial with a statistically significant result (at 2p < 0.05), the likelihood of the meta-analysis being significant (at 2p < 0.01) increased strikingly. For example, among the 473 meta-analyses in which the first trial in a batch of 10 was statistically significant (at 2p < 0.05), 18 (3.8%) favoured treatment at 2p < 0.01. CONCLUSIONS: Chance can influence the results of meta-analyses regardless of how well they are conducted. Researchers should not ignore this when they plan a meta-analysis and when they report their results. People reading their reports should also be wary. Caution is particularly important when the results of one or more included studies influenced the decision to do the meta-analysis.

Original publication




Journal article


J R Soc Med

Publication Date





116 - 119


Systematic reviews, chance, meta-analysis, randomized trials, statistical analysis, Bias, Biometry, Clinical Trials as Topic, Computer Simulation, Humans, Meta-Analysis as Topic, Outcome Assessment (Health Care), Review Literature as Topic