Reproducible Research with Code Notebooks
Bjørn Peare Bartholdy
Data Steward, Delft University of Technology
Agenda
- Short intro to reproducibility
- Introduction to Quarto
- DEMO
- Q&A
What is reproducibility?
Definitions
Turing Way
Reproducibility
The ability for an independent researcher to obtain the same results using the same data, methods, and procedures as described in a study.
Computational Reproducibility
The ability to obtain the same results by running the same code and using the same data in the same computing environment.
Why is reproducibility important?
Selfish Reasons
- Avoid disaster
- Good for your reputation
- Save time in the future
- Good for continuity
- Halp reviewers see your way
Markowetz, F. Five selfish reasons to work reproducibly. Genome Biol 16, 274 (2015). https://doi.org/10.1186/s13059-015-0850-7
Non-selfish Reasons
- Enables others to verify and trust your results
- Facilitates cumulative, self-correcting science
- Enhances transparency and accountability
- Helps mitigate errors, fraud, or misinterpretation
Path towards reproducibility
Reproducibility is a Spectrum
Marwick et al. 2017 https://doi.org/10.31235/osf.io/72n8g
What you can do?
- Write code for your research
- Documentation: Explain what steps are needed to reproduce the results
- List dependencies
- Use version control
- Use reproducibile workflow tools
- Publish your code and data
About Quarto
Quarto: What is it?
A free, open-source scientific and technical publishing system built on Pandoc.
Documents are authored using markdown.
Did I mention free and open source? 😎
Quarto Language support
Code can be seamlessly integrated in Quarto documents
- Python
- R
- Julia
- Observable
Quarto Output formats
- Documents
- HTML
- DOCX
- MD
- Jupyter
- Presentations
- reveal.js
- beamer
- powerpoint
- Books
- HTML
- EPUB
- DOCX
- AsciiDoc
- Websites
- HTML (Bootstrap 5)
- publish directly to Netlify
And many more.
Quarto Output formats
- Documents
- HTML
- DOCX
- MD
- Jupyter
- Presentations
- reveal.js
- beamer
- powerpoint
- Books
- HTML
- EPUB
- DOCX
- AsciiDoc
- Websites
- HTML (Bootstrap 5)
- publish directly to Netlify
And many more.
No, seriously, see the full list here
Quarto Extensions
Extensions by both developers and the community
- shortcodes
- fontawesome
- social-embeds
- …
- filters
- lightbox
- shinylive
- include-code-files
- …
- formats
- letter-pdf
- rladies-revealjs
- blackboard-revealjs
- journal templates
- …
Bring your own IDE
- RStudio
- VSCode
- Neovim
- Jupyter
Extensive documentation
Compared to existing options
RMarkdown for R users and Jupyter for Python users
Quarto is more streamlined and specifically tailored to scientific output
- built-in functionality
- cross-refs
- author metadata
- more output options
- including Jupyter
- formatting features
- columns
- callouts
- footnotes
Note
I’m a callout
Compared to existing options
RMarkdown for R users and Jupyter for Python users
Quarto is more streamlined and specifically tailored to scientific output
- built-in functionality
- cross-refs
- author metadata
- more output options
- including Jupyter
- formatting features
- columns
- callouts
- footnotes
Important
I’m an important callout
Compared to existing options
RMarkdown for R users and Jupyter for Python users
Quarto is more streamlined and specifically tailored to scientific output
- built-in functionality
- cross-refs
- author metadata
- more output options
- including Jupyter
- formatting features
- columns
- callouts
- footnotes1
Important
I’m an important callout
Active development
GIPHY
Quarto examples
Example projects
Example websites
Example presentation
GIPHY
Join us
Join an upcoming R Café at TU Delft!
Join the Rbanism community for meet-ups, workshops, challenges and more around R in urbanism research!
We run these activities in close collaboration with the TU Delft Digital Competence Centre, TU Delft Research Data Services, Open Science Community Delft, and the Netherlands eScience Center.
The hands-on part (if you missed it)
Quarto: the basics
We’ll start out with a blank Quarto document (my-article.qmd)
First we need to populate the YAML header
Quarto: the basics
We’ll start out with a blank Quarto document (my-article.qmd)
Then add some markdown content
Quarto: the basics
We’ll start out with a blank Quarto document (my-article.qmd)
Then add some markdown content
---
title: Catchy title or pun: the actual title
author: Me and Al
---
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
Quarto: the basics
Now render the document.
You can either render from your IDE,
or using the command line
Quarto: the basics
Dynamic variables- don’t repeat yourself!
metadata shortcode
{{< meta name-of-variable >}}
Include external content (e.g. qmd or md files)
{{< include name-of-file.ext >}}
Include custom variables from _variables.yml
{{< var name-of-variable >}}
Important
Using _variables.yml only works with Quarto projects
Otherwise you will get the error ?var:<varname>
Quarto: the basics
You can add keywords to the yaml header and include them in the text
---
title: Catchy title or pun: the actual title
author: Me and Al
---
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
Quarto: the basics
You can add keywords to the yaml header and include them in the text
---
title: Catchy title or pun: the actual title
author: Me and Al
keywords: this; is; science!
---
{{< meta keywords >}}
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important thingsQuarto: the basics
And maybe you already wrote the materials and methods section,
which you have in another file called *_matmet.qmd*
---
title: Catchy title or pun: the actual title
author: Me and Al
keywords: this; is; science!
---
{{< meta keywords >}}
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
{{< include _matmet.qmd >}}Quarto: the basics
Of course the real power lies in seemlessly combining text and code (with output).
Using any of the supported languages, like R
---
title: Catchy title or pun: the actual title
author: Me and Al
keywords: this; is; science!
---
{{< meta keywords >}}
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
{{< include _matmet.qmd >}}Quarto: the basics
Of course the real power lies in seemlessly combining text and code (with output).
Using any of the supported languages, like R
---
title: Catchy title or pun: the actual title
author: Me and Al
keywords: this; is; science!
---
{{< meta keywords >}}
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
{{< include _matmet.qmd >}}
## Results
```{r}
#| label: fig-airquality
#| fig-cap: "Temperature and ozone level."
#| warning: false
#| echo: true
library(ggplot2)
ggplot(airquality, aes(Temp, Ozone)) +
geom_point() +
geom_smooth(method = "loess")
```Quarto: the basics
Of course the real power lies in seemlessly combining text and code (with output).
Using any of the supported languages, like R, or Python
---
title: Catchy title or pun: the actual title
author: Me and Al
keywords: this; is; science!
---
{{< meta keywords >}}
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
{{< include _matmet.qmd >}}
## Results
```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"
import numpy as np
import matplotlib.pyplot as plt
r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
subplot_kw = {'projection': 'polar'}
)
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```Quarto: the basics
Added bonus: built-in cross-referencing
---
title: Catchy title or pun: the actual title
author: Me and Al
keywords: this; is; science!
---
{{< meta keywords >}}
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
{{< include _matmet.qmd >}}
## Results
```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"
import numpy as np
import matplotlib.pyplot as plt
r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
subplot_kw = {'projection': 'polar'}
)
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```
Blah, blah, blah, something important. Don't believe me? Check out @fig-polar!Quarto: the basics
---
title: Catchy title or pun: the actual title
author: Me and Al
keywords: this; is; science!
---
{{< meta keywords >}}
## Introdution
This paper will make an important impact in a specific field of science,
despite having "negative" results.
The aims of this paper are:
- something important
- some other important things
- maybe a few less important things
{{< include _matmet.qmd >}}
## Results
```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"
import numpy as np
import matplotlib.pyplot as plt
r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
subplot_kw = {'projection': 'polar'}
)
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```
Blah, blah, blah, something important. Don't believe me? Check out @fig-polar!
## Conclusion
Quarto is awesome!
DOI