What factors might influence the trustworthiness of non-randomized studies of interventions?

As researchers, we regularly read scientific publications and experience varying levels of trust in the reported results. This trust is based on our expertise in the planning, conduct, and reporting of studies. Beside possible biases, our trust in the study might be influenced by other factors. These factors could be related to the study itself but also to external factors such as the research field, the topic, where it is published (e.g., journal, platforms), who conducted the study, etc

Overall, the trust in the reported results is based on implicit factors which, to our knowledge, have never been explored.

Let us illustrate with this example:

Two non-randomized (observational) studies assessing the effect of a pharmacologic treatment on patients’ health have been critically appraised using a specific risk of bias tool (i.e., ROBINS-I) which explores risk of bias due to confounding; risk of bias in selection of participants; risk of bias in classification of interventions; risk of bias due to deviations from intended interventions; risk of bias due to missing data; risk of bias in measurement of outcomes; and risk of bias in selection of the reported result. Similar issues were identified in both studies and they were both rated as having an overall moderate risk of bias.

However, when you read the reports, you believe that one study is more trustworthy than the other. This judgment is based on factors, other than the risk of bias, that influence your trust.

With this in mind, please think about recent reports of NRSI that you have read, and identify the factors, other than the risk of bias, that might influence your trust the results of the study.

You may list factors that pertain to the study itself (e.g., quality of the dataset), who led, conducted, analyzed or reported the study (e.g., reputation of the research centers), where it is published (e.g., predatory journals), the field where it is performed, the circumstances under which it is performed, the research practices (e.g., data sharing), the effect estimates (e.g., unexpectedly large effect estimates) or any other factors that you may consider influential.

Please list all factors in your own words and provide a brief explanation and specify whether the factor might increase or decrease your trust in non-randomized studies of interventions. We invite you to cite as many factors you think might influence your trust in the study.

Of note, some factors might be rarely reported in non-randomized studies of interventions, but should nevertheless be considered. If you have any doubt on whether the factor is covered in the risk of bias, then include it in the list of factors below.