At the Open Science Fellows Program, I had the pleasure to meet Dr. Jana Lasser. In her “open science” fellowship she is working on how executable papers can be a tool for more reproducibility and transparency in the natural sciences. The topic is initially irritating for a social science blog (with a qualitative orientation). For me it is important to enable interdisciplinary exchange and to keep thinking outside the box. Therefore, I am very pleased that Jana has written three blog posts about her experience with executable papers:
- What is an executable paper?
- How to create an executable paper and make it accessible?
- Tips and tricks for creating an executable paper
Dr. Jana Lasser received her PhD in physics at the Max Planck Institute for Dynamics and Self-Organization in 2019 and is now working as a postdoctoral researcher at the Complexity Science Hub Vienna. The work presented in this article was made possible by the Wikimedia foundation, Stifterverband and Volkswagen foundation. @janalasser
In the first, and second post of this series, I explained what an executable paper is and introduced tools to create executable papers and make them accessible. In the following post, I will share a few more insights on how to use the format of an executable paper effectively to make research more transparent.
Choosing the right level of detail
Making every single analysis step transparent and reproducible is desirable for the sake of openness. Nevertheless, an article should still be readable and readers should not be discouraged by the level of detail given. Every scientific publication therefore has to find the right balance between granularity and the big picture and the executable paper is no exception. Still, to this end, the executable paper format gives more options than a regular publication: For example in Jupyter Notebooks all code cells can be collapsed, leaving only their output (figures or tables) and the formatted text. This way, clicking on a code cell and diving into the details of the analysis becomes an additional option the reader has. Additionally, Jupyter Notebooks offer the possibility to add direct links to more detailed descriptions or separate analysis scripts either at the bottom of the document or in a separate file. Links can be followed without downloading additional materials or firing up a different software, which makes reproducing the research easier and more transparent.
Including intermediate steps
In the process of creating insights from raw data, data is often analyzed in multiple steps. Nevertheless, the reader might only be interested in reproducing a single one of these steps and being forced to reproduce all steps might become frustrating. A solution is to provide both the complete analysis pipeline as well as the data that is created in each step. This way, a reader has the choice of reproducing an analysis step himself or herself or falling back on the already analyzed data that is supplied within the repository.
Clean and shareable data
The executable paper relies on the availability of the raw data which means raw data has to be well-ordered, well described, and uploaded somewhere publicly accessible. Formatting data in such a way is very time consuming. While going through the process of cleaning up my data to create an executable publication, I realized I had made countless implicit assumptions that guided the data processing. Particularly relevant points in this process turned out to be
- using telling and unique file names that describe the data content, and possibly the dates when the data was collected or processed;
- documenting all relevant meta-information in a dedicated README file any time data are modified;
- storing a copy of the original raw data and
- having multiple and synchronized backups of the processed data.
Beside being fundamental in terms of open science practice, formatting data can bring further benefits. For example, I could publish my reformatted data as six independent datasets [1, 2, 3, 4, 5, 6] on PANGAEA, a platform to publish data sets, and write a dedicated descriptive publication (preprint available here). Publishing data as a scientific artifact in itself is increasingly being recognized as valuable contribution to research and integrates seamlessly into the process of creating an executable paper.
In my opinion, one of the major benefits of executable papers in terms of supporting the transparency of the research process, is the possibility to extend the analysis beyond what is typically reported in a traditional publication. Usually, only a single (often “the successful”) way of analyzing the data is described in a paper, and things like sensitivity analysis, or the results obtained with different parameters are reported in the supplementary information, or omitted altogether. Conversely, the addition of widgets to a Jupyter Notebook allows for user input in the code cells. This way, plots can be interactively modified and re-computed, taking the reader input into account (for example from a dropdown menu, as shown in the Figure below). The reader can then redo the calculation to check to what extent the conclusions drawn from the analysis still hold using different parameters.
To summarize, executable papers offer a lot of possibilities, such as interactive elements, to add additional content to a scientific publication. These possibilities need to be used carefully to strike a balance between detail and the bigger picture of the research. Executable papers rely on clean and accessible data and adequately preparing data is not a trivial task. Nevertheless, data prepared in such way can easily be re-used for dedicated data publications.
With this I conclude my series of posts about executable papers and hope it motivates readers to try this format and make their research even more transparent and accessible.
 Nield, J. M., Lasser, J., & Goehring, L. (2020). Temperature and humidity time-series from Owens Lake, central California, measured during one week in November 2016. [Data set] https://doi.org/10.1594/PANGAEA.911139
 Nield, J. M., Lasser, J., & Goehring, L. (2020). TLS surface scans from Owens Lake and Badwater Basin, central California, measured in 2016 and 2018 [Data set]. https://doi.org/10.1594/PANGAEA.911233
 Lasser, J., & Goehring, L. (2020). Subsurface salt concentration profiles and pore water density measurements from Owens Lake, central California, measured in 2018. [Data set] https://doi.org/10.1594/PANGAEA.911059
 Lasser, J., Goehring, L., & Nield, J. M. (2020). Images and Videos from Owens Lake and Badwater Basin in central California, taken in 2016 and 2018. [Data set] https://doi.org/10.1594/PANGAEA.911054
 Lasser, J., & Goehring, L. (2020). Grain size distributions of sand samples from Owens Lake and Badwater Basin in central California, collected in 2016 and 2018. [Data set] https://doi.org/10.1594/PANGAEA.910996
 Lasser, J., & Karius, V. (2020). Chemical characterization of salt samples from Owens Lake and Badwater Basin, central California, collected in 2016 and 2018. [Data set] https://doi.org/10.1594/PANGAEA.911239