Objectives: To show the effects of chance on meta-analyses,and the potential dangers of being prompted to do a metaanalysisby one favourable trial. Design: In total, 100,000 trials were simulated and combinedinto 10,000 meta-analyses, using data from the controlgroup of a cancer trial. Each participant record wasrandomly coded to simulate allocation to 'treatment' or 'control'. Setting: Simulated study. Participants: De-identified records for 578 patients fromthe control group of a cancer trial, of whom 147 had died.Main outcome measure: Time to death from any cause. Results: Of the 100,000 trials, 4897 (4.9%) were statisticallysignificant at 2p < 0.05 and 123 (1.2%) of the 10,000meta-analyses were significant at 2p < 0.01. The mostextreme result was a 20% reduction (99% CI: 0.70-0.91;2p 1/4 0.00002) in the annual odds of dying in the 'treatment' group. If a meta-analysis contained at least one trial with astatistically significant result (at 2p < 0.05), the likelihood ofthe meta-analysis being significant (at 2p < 0.01) increasedstrikingly. For example, among the 473 meta-analyses inwhich the first trial in a batch of 10 was statistically significant(at 2p < 0.05), 18 (3.8%) favoured treatment at2p < 0.01.Conclusions: Chance can influence the results of metaanalysesregardless of how well they are conducted. Researchersshouldnotignorethiswhentheyplanameta-analysisandwhentheyreporttheirresults.Peoplereadingtheirreportsshouldalsobewary.Cautionisparticularlyimportantwhentheresultsofoneormoreincludedstudiesinfluencedthedecisiontodothemeta-analysis. © The Royal Society of Medicine.

Original publication

DOI

10.1177/0141076813514574

Type

Journal article

Journal

Journal of the Royal Society of Medicine

Publication Date

01/03/2014

Volume

107

Pages

116 - 119