74% of academic research code fails to run — research says
What can we learn from this setback?
As we are witnessing an ongoing replication crisis in various fields of science, it seems like data-science and software engineering are not an exemption.
How can we learn from the academic crisis as practitioners?
Intro
In a study recently published on Scientific Data Journal, the research team conducted an automatic process where in 2109 code packages had been pulled from the Harvard Dataverse archive between the years 2010 and 2020, these contain 9078 R code files in total.
Then, the process created 3 clean runtime environments for versions 3.2, 3.6 and 4.0 of R interpreter and then attempted to execute the scripts on all 3 environments.
Re-execution is counted as successful if at least 1 of these attempts completed without errors.
Then, the process conducted a cleaning routine to all scripts.
The cleaning routine included replacing absolute paths, file encoding and identification of file imports.
A time limit of 5 hours had been imposed on the execution of each script.
Such that in a case when time limit is exceeded (TLE), the script is excluded from the results and noted as TLE in the results table.
Results
Of these ~9000 scripts, only 25% successfully executed without cleaning.
After cleaning process conducted, 40% executed successfully, best effort of both with and without cleaning stood at 56%.
Take homes
The study lists several ‘best practices’ based on the re-execution results:
- Libraries should be
- Use relative paths
- Create a workflow pipeline that specifies the correct execution sequence.
- Use containerization solutions such as Docker.
- Test your code on a clean environment before shipping (it works on my machine).
- Avoid using proprietary software.
- Capture re-execution commands.
Personal interpretation
To my knowledge, this is the first large scale study on quality and reproducibility of academic code.
Code reproducibility is crucial in software businesses as it reduces barriers to entry and makes it easier for new team members to be involved.
Taking these best practices will immediately make your code execution reproducible and boost up the productivity of your team mates and also your own.
Code reproducibility isn’t just a skill, it’s a culture.
Create a healthy coding cloture in your organization and enjoy the benefits of highly cohesive research and development team.
Study:
Trisovic, A., Lau, M.K., Pasquier, T. et al. A large-scale study on research code quality and execution. Sci Data 9, 60 (2022). https://doi.org/10.1038/s41597-022-01143-6
