A recent study by Davidson et al (METHODS team) finds no compelling evidence that preprints have different summary treatment effect estimates than peer-reviewed journal publications. The results are published open access in the Journal of Clinical Epidemiology.
Inclusion of preprint trials within a meta-analysis is not largely endorsed since these manuscripts have not undergone formal peer review and may overestimate the treatment effects. The authors analyzed data from a large living systematic review and meta-analysis (COVID-NMA, www.covid-nma.com), and selected 37 pharmacological treatment meta-analyses that included 44 preprints and 70 peer-reviewed publications and found that there was no statistically significant difference in summary effect estimates between the two publication types – ratio of odds ratio (ROR), 0.88; 95% CI, 0.71–1.09; I2 = 17.8%; τ2= 0.06 (ROR < 1 indicated larger effects in preprint trials).
The findings were consistent across all the post-hoc sensitivity analyses but the authors caution that their analysis is limited to the COVID-19 context and the small number of trials per meta-analysis which increased uncertainty around the estimation.
The authors conclude that, based on their data and especially within the landscape of a fast-moving pandemic, considering the results of preprint trials may be reasonable. They recommend that systematic reviewers and meta-analysts assess preprint inclusion on an individual level, accounting for risk of bias and completeness of reporting.
