Post-publication peer review and the identification of methodological and reporting issues of COVID-19 trials
In a recent study led by Mauricia Davidson (CRESS-METHODS) and published in BMJ Evidence-Based Medicine, researchers sought to investigate the extent to which systematic reviewers and post-publication peer review identified methodological and reporting issues in COVID-19 trials. The study focused on randomized controlled trials (RCTs) evaluating pharmacological treatments for COVID-19 and analyzed how frequently issues were addressed—and whether they could have been easily resolved by trial authors.
The study utilized data from the COVID-NMA living systematic review (covid-nma.com), as well as commentary on preprint servers such as medRxiv, Research Square, and SSRN, along with PubPeer, an online post-publication peer review platform. By employing qualitative content analysis, Davidson et al. developed themes and domains of methodological and reporting issues found within these trials.
The researchers found that systematic reviewers identified methodological and reporting issues in 446 of the 500 eligible RCTs (89%). In 391 trials (78%), the identified issues could have been easily addressed by the authors, including incomplete reporting (49%), selection of reported results (52%), and the absence of access to a pre-specified trial plan (25%). These findings suggest that systematic reviewers, without changing their standard process, already detect key issues that could be easily resolved by authors.
However, despite the potential of post-preprint and post-publication peer review as a quality control mechanism, only 74 RCTs (15%) had received at least one comment on PubPeer or preprint servers, with a total of 348 comments, and only 46 trials (9%) had received comments addressing issues that could have been easily resolved by trial authors. These included incomplete reporting (6%), errors (5%), statistical analysis flaws (3%), inconsistent reporting of methods and analyses (2%), spin (2%), selection of reported results (1%), and lack of access to raw data or pre-specified plans (1%).
Davidson et al. concluded that while systematic reviewers play a critical role in highlighting methodological and reporting issues, the absence of an established feedback mechanism between them and trial authors represents a missed opportunity for improving research quality. The study underscores the need for direct engagement between systematic reviewers and trial authors to supplement traditional peer review and ensure rigorous scientific reporting.
Additionally, the study highlights the need to foster a culture within the research community that values post-publication peer review. Despite its potential as a feedback mechanism, the study found that post-publication peer review remains underutilized and relatively ineffective in identifying key issues.
The study’s findings offer valuable insights into how the research ecosystem can evolve to better support transparency and continuous improvement in trial reporting, ensuring high-quality evidence reaches all stakeholders.
- Davidson M, Korfitsen CB, Riveros C, Chaimani A, Boutron I. Post-publication peer review and the identification of methodological and reporting issues in COVID-19 trials: a qualitative study. BMJ Evidence-based Medicine. 2025 Feb 20; https://doi.org/10.1136/bmjebm-2024-113068
By Mauricia Davidson
