What is an executable paper?

At the Open Science Fellows Program, I had the pleasure to meet Dr. Jana Lasser. In her “open science” fellowship she is working on how executable papers can be a tool for more reproducibility and transparency in the natural sciences. The topic is initially irritating for a social science blog (with a qualitative orientation). For me it is important to enable interdisciplinary exchange and to keep thinking outside the box. Therefore, I am very pleased that Jana has written three blog posts about her experience with executable papers:

1. What is an executable paper?
2. How to create an executable paper and make it accessible?
3. Tips and tricks for creating an executable paper

Dr. Jana Lasser received her PhD in physics at the Max Planck Institute for Dynamics and Self-Organization in 2019 and is now working as a postdoctoral researcher at the Complexity Science Hub Vienna. The work presented in this article was made possible by the Wikimedia foundation, Stifterverband and Volkswagen foundation. @janalasser

What is an executable paper?

Enhanced by the current Covid-19 pandemic, Open Science practices have taken an even more central role in the way we conduct research. This does not stop at the data being available, but includes transparency of the methods employed to reach conclusions, and the dissemination of the results. Too often, even if data sets are openly available, analysis code is hidden away in cryptic scripts that are “available upon request” and not thought for other researchers to read and use them. A prominent example of this problem is the simulation based research that informed large parts of the UK’s policy decisions regarding Covid-19. The research received heavy criticism because the underlying simulation code was not accessible and not understandable at first. A way to remedy this problem and support data availability, transparency of methods is the “executable paper” – i.e. formatting a scientific publication as a dynamic piece of software that combines text, raw data, and the code used for the analysis, that a reader can interact with and that makes the process of the generation of insights transparent.

In principle, the idea is to enable the reader to reproduce each step taken to arrive at a publication’s conclusion, from the raw data to the polished plot. Therefore, a fundamental part of producing an executable paper is making all material, from the actual paper to the data, available to readers by hosting it on an accessible platform. The executable paper itself should be linked to a repository containing the raw data and should be able to

• display nicely formatted text, including references and links (just like a journal article),
• display figures, plots, and possibly also videos and interactive elements,
• display the code used for creating such plots from data,
• interpret such code and user input in case of interactive elements, and
• be composed of completely Open Source components and hosted under a free license to be easy to share and re-use.

Traditional journals increasingly support uploading online-only supplements, including analysis codes, images and even videos. Nevertheless, these materials are sometimes incomplete and clearly separated from the main publication. This makes the reproduction of the research from the raw data to the polished figure shown in the paper difficult or even impossible. This can be remedied by an executable paper that collects and combines all relevant information in one master document. This way, following the details necessary to reproducing the research does not require multiple switches between different documents and software, thus becoming more transparent.

Sadly, the concept of executable papers is not established in traditional journals and only a handful of computer science and math journals support a variation of the format – the “reproducible paper”. This publication type is centered around a simulation code that can be run by the reader to generate data and perform their analysis, thus reproducing the whole research process down to the results. The main difference to the executable paper is that, while the executable paper relies on empirical data, the reproducible paper’s code generates the data, thus putting a larger emphasis on the description of the code.

Creating an executable paper takes a significant amount of time, but I can only recommend the experience. It gave me new confidence about my own research, and forced me to revisit choices I had made in the past, sometimes in a rush and with less understanding than I have today. Creating an executable publication also gave me the chance to use many open tools and workflows I already knew about, and to pull them together into a more open and transparent way of academic working. In retrospective, creating an executable paper is a journey through almost all aspects of Open Science: from open and accessible data to open and transparent tools and software, ending with an open publication composed of open and transparent content.

Examples

• To show how executable papers look likehere is an example of my own work: Lasser, J. (2020). JanaLasser/salt-polygons-are-caused-by-convection: Final version for open science fellowship project “executable papers” (Version v1.0). Zenodo. https://doi.org/10.5281/ZENODO.3831237. A publication that explains why patterns form on the surface of salt deserts.
• Another very good example by a different research group is: R. Abbott et al. (2019) LIGO tutorial. The publication walks the reader through the detection of gravitational waves based on data from the LIGO experiment.

In the next part of this series, I will give an introduction how to create an executable paper.

---

---