Study results are put to the test at the “Münster Reproduction Games” – An interview

Reproducibility is a key factor in determining how conclusive the results of scientific studies are. At the “Münster Reproduction Games” on 25 September (Friday), researchers in all career stages from the master's level onwards will gather to put already published studies to the test. The organisers are the Münster Center for Open Science (MüCOS), the Center for Data Science and Complexity, the Service Center for Digital Humanities and the Institute of Political Science at the University of Münster. In the following interview with Christina Hoppenbrock, MüCOS managing director Dr Lukas Röseler offers insights into the event and describes strategies for meeting the requirement of reproducibility.

Which scientific papers are going to be put to the test?

Reproduction means checking the results of a study against the same data used in the study. In principle all studies for which data are available are eligible. This is primarily the case with quantitative studies. We focus on findings that are relevant to the disciplines represented at the event, such as political science, psychology, digital humanities or physics, and which claim to be reproducible. This could, for example, include all the studies from a prestigious journal.

And how does such a check proceed during the event?

The teams choose a study in advance. We ensure that the data are available and, whenever possible, document which software was used to analyse the data in the original study. If the analysis code is also available, a so-called numerical reproduction is carried out. This means checking whether the same values are calculated using identical data and identical code. If the code is not available, the analysis must be reconstructed. Subsequently, the participants carry out robustness analyses – they use an alternative analysis path to check whether they can support the same conclusions as in the target study. We document the results in a report and include them in the world's largest curated database of reproduction and replication attempts. Other researchers are then made aware of the reproduction attempts via various software tools, such as a plugin for the literature management programme ‘Zotero’.

All the papers involved here have undergone a peer-review process, i.e. an assessment by experienced scientists. Did the reviewers who assessed the studies fail to do their job properly?

Checking findings for reproducibility is not part of quality assurance in most disciplines. When evaluating research, the names of the journals, publication numbers and citation numbers are often what matter. Publishing a lot is seen as something positive. However, this also leads to a higher demand for reviews. The number, quality and detail of reviews play no role in research evaluation. In this situation, requiring reviewers to check data and results would make it harder to find volunteers and would lengthen assessment times. However, some journals, especially those independent of publishers, have already successfully implemented editorial reproducibility checks. This includes our own new journal, “Replication Research”.

Current studies show that newer papers are more likely to be reproducible than older ones, and that the accessibility of experimental data improves reproducibility. Is this a sign that something is changing in the field?

Yes, on almost all levels of the scientific system, we are seeing a shift towards more transparency and higher quality. Scientific societies are introducing guidelines for data sharing, funding providers are requiring data management plans and institutions are providing infrastructure. University centres offer workshops – and researchers are demanding that data be handled responsibly. As a result, shared, licensed and well-documented data in some fields – which used to be a luxury – have become a standard requirement of scientific work within a few years.

The issue of reproducibility has had an impact on most researchers. What advice would you give them?

Discuss with your colleagues what role reproducibility has in your own field of research and what role it should have. If it is not necessary or is not factored into the evaluation, this could at worst cause others who work more thoroughly to be disadvantaged. In that case, one should address the issue together with scientific communities and organisations. My second piece of advice: if you build on other findings that claim to be reproducible, then verify that claim. If the results cannot be confirmed with the original data, then they might not be confirmable in a new study either.

Further information

• Münster Center for Open Science
• Center for Data Science and Complexity
• Service Center for Digital Humanities
• The world's largest curated database of reproduction and replication: FORRT Library of Reproduction and Replication Attempts (FLoRA)
• The journal “Replication Research”
• Save the date: A three-day reproducibility workshop for early-career researchers