Poor reproducibility of systematic review results is a common challenge in both primary and secondary research, often leading to reduced confidence in this type of study among the scientific community and the public.
To address this limitation, it has been proposed to introduce a final verification step within the systematic review process, with the aim of testing the reproducibility of results and increasing the overall robustness of the study. According to this proposal, systematic reviews should include the following stages: defining the research question; establishing eligibility criteria; searching for, collecting, and selecting data; assessing the methodological quality of studies and the statistical strength of the evidence; presenting and discussing the results; and, finally, testing their reproducibility.
Reproducibility testing
Reproducibility testing can be conducted using two different approaches.
A first option involves having a researcher from the same team, but not directly involved in the project, repeat the processes carried out by the original authors before the systematic review is finalized. This functions as a form of quality control. If discrepancies from the original findings are identified, the authors have an opportunity to review and revise the work before publication, clarifying and resolving the causes of the discrepancies.
A second approach consists of having an independent group of authors review the results after the systematic review has been completed. At the conclusion of this process, both groups can discuss any discrepancies identified and amend the work accordingly. The cycle of review, discrepancy identification, and resolution may then be repeated for each stage of the study.
Greater quality and transparency
The results of reproducibility testing should be documented in a report and published as a supplement to the systematic review, promoting complete and transparent communication and helping readers better understand the quality and reliability of the work.
It is worth noting that systematic reviews are already evaluated through tools such as AMSTAR 2 and ROBIS, which are designed to assess methodological quality and the risk of bias. However, these tools do not directly evaluate whether the results can be independently reproduced.
Reproducibility testing could therefore become a new publication requirement for systematic reviews. The implementation of such a requirement could reduce the number of reviews with poorly reproducible or non-reproducible findings while encouraging the publication of higher-quality research.
Source
Faggion CM Jr. Should a systematic review be tested for reproducibility before its publication? Journal of Clinical Epidemiology. 2019 Jun;110:96. doi:10.1016/j.jclinepi.2019.02.008.