index.qmd

---
format: 
  revealjs:
    theme: ["styles.scss"]
    footer: "<https://github.com/pythonhealthdatascience/gw4_prize_presentation>"
    preview-links: true
    height: 800
    width: 1250
    preload-iframes: true
    controls: true
    controls-layout: bottom-right
    revealjs-plugins:
     - fullscreen
---

# Improving Reproducibility in Open Healthcare Simulation

<hr>

:::: {.columns}

::: {.column width="75%"}
<br>
<h3>Amy Heather <a href="https://orcid.org/0000-0002-6596-3479" aria-label="View ORCID record"><img src="images/orcid.png" alt="ORCID iD" style="width:32px; margin-left:15px; margin-right:6px; vertical-align:middle;"></a>0000-0002-6596-3479</h3>
<br>
<br>
Part of the "STARS" project:
:::

::: {.column width="25%"}
![](images/exeter.png)
:::

::::

![](images/stars_banner.png)

## Discrete event simulation (DES)

<p align="center">
  <img src="images/simple_des.gif"/>
</p>

<div style="text-align: center; font-size: 0.8em">
Animation created using web app developed by <strong>Sammi Rosser</strong> <a href="https://orcid.org/0000-0002-9552-8988" aria-label="View ORCID record"><img src="images/orcid.png" alt="ORCID iD" style="width:16px; margin-left:6px; margin-right:6px; vertical-align:middle;"></a>0000-0002-9552-8988
<br>
Available at <img src="images/GitHub_Invertocat_Dark.png" alt="GitHub" style="width:16px; height:16px; margin-left:6px; vertical-align:middle;"> [hsma-programme/Teaching_DES_Concepts_Streamlit](https://github.com/hsma-programme/Teaching_DES_Concepts_Streamlit) (2024) (MIT Licence).
</div>

## Sharing simulation models

<br>

**Reproducibility**: Generating reported results using the provided code and data.

<br>

:::: {.columns style='display: flex;'}

::: {.column width='40%'}

```{.python}
# Simulation code...
# etc.

# Plotting code...
fig = px.line(
    df, 
    x='Number of Nurses', 
    y='Mean Wait Time (min)', 
    color='Inter-arrival Time',
    markers=True,
    color_discrete_sequence=['#0000FF', '#FFA500', '#FF0000']
)

fig.update_layout(
    title='Impact of Nursing Staff on Patient Wait Times',
    xaxis_title='Number of Nurses',
...
```

:::

::: {.column width='10%' style='display: flex; align-items: center;'}

![](images/thin-long-arrow-right-icon.png)

:::

::: {.column width='50%'}

```{python}
import plotly.express as px
import pandas as pd
import numpy as np

# Create sample simulation data
nurses = np.tile([1, 2, 3, 4, 5, 6], 3)
inter_arrival_times = np.repeat(['5 min', '10 min', '15 min'], 6)
wait_times = [
    # 5 min inter-arrival time
    120, 70, 50, 40, 35, 33,  
    # 10 min inter-arrival time
    80, 45, 30, 22, 18, 16,   
    # 15 min inter-arrival time
    40, 22, 15, 10, 8, 7      
]

# Create dataframe
df = pd.DataFrame({
    'nurses': nurses,
    'iat': inter_arrival_times,
    'mean_wait': wait_times
})

# Create line chart with multiple lines
fig = px.line(
    df, 
    x='nurses', 
    y='mean_wait', 
    color='iat',
    markers=True,
    color_discrete_sequence=['#0000FF', '#FFA500', '#FF0000'],
    height=400,
    width=600
)

# Customize layout
fig.update_layout(
    title='Impact of Nursing Staff on Patient Wait Times',
    xaxis_title='Number of Nurses',
    yaxis_title='Mean Wait Time (minutes)',
    legend_title='Inter-arrival Time'
)

fig.show()
```

:::

::::

## Why does reproducibility matter?

:::: {.columns}

::: {.column width='50%' .fragment fragment-index=1}

<p align="center">
  For other modellers...
</p>

:::

::: {.column width='50%' .fragment fragment-index=4}

<p align="center">
  For modeller...
</p>

:::

::::

:::: {.columns}

::: {.column width='25%' .fragment fragment-index=2}

<p align="center">
  ![](images/partner-handshake-icon.png){width=200}<br>
  Trust
</p>

:::

::: {.column width='25%' .fragment fragment-index=3}

<p align="center">
  ![](images/recycle-sign-arrow-icon.png){width=200}<br>
  Reuse
</p>

:::

::: {.column width='25%' .fragment fragment-index=5}

<p align="center">
  ![](images/recycle-sign-arrow-icon.png){width=200}<br>
  Reuse
</p>

:::

::: {.column width='25%' .fragment fragment-index=6}

<p align="center">
  ![](images/quality-badge-checkmark-icon.png){width=200}<br>
  Quality
</p>

:::

::::

<br>

:::: {.columns}

::: {.column width='45%' .fragment}

Reasons for non-reproducility include:

* Missing code
* Lost parameters
* Software changes

Troubleshooting **time-consuming** and maybe **impossible** if information is lost.

:::

::: {.column width='5%'}
:::

::: {.column width='50%'}

::: {.my-callout .fragment}
::: {.my-callout-header}
Research aim:
:::
::: {.my-callout-container}
Generate a set of actionable recommendations to improve reproducibility of healthcare simulation models.
:::
:::

:::

::::

## Protocol

<p align="center">
  <img src="images/stars_wp1_workflow.png"/>
</p>

## Pre-registration

::: {.scroll-container style="overflow-y: scroll; height: 700px;"}

<p align="center">
  <img src="images/zenodo.png"/>
</p>

:::

## GitHub template

::: {.scroll-container style="overflow-y: scroll; height: 700px;"}

<p align="center">
  <img src="images/github_template.png"/>
</p>

:::

## {#study-websites fullscreen=true}

<iframe class="stretch" data-src="https://pythonhealthdatascience.github.io/stars-reproduce-kim-2021/evaluation/reproduction_report.html"></iframe> 

## {#summary-website fullscreen=true}

<iframe class="stretch" data-src="https://pythonhealthdatascience.github.io/stars_wp1_summary/pages/reflections.html"></iframe> 

## Recommendations

:::: {.columns}

::: {.column width='50%'}

![](images/reproduction_wheel.png)

:::

::: {.column width='50%'}

![](images/troubleshooting_wheel.png)

:::

::::

## Preprint

<p align="center">
  <img src="images/preprint.png"/>
</p>

## Ongoing / future work

::: {.scroll-container style="overflow-y: scroll; height: 700px;"}
<p align="center">
  ![](images/python_template.png)
</p>
:::

## Ongoing / future work

:::: {.columns}

::: {.column width='20%'}

::: {.callout-warning icon="false"}

## Re-runnable

Code runs
<br><br><br>&nbsp;

:::

:::

::: {.column width='20%'}

::: {.callout-warning icon="false"}

## Repeatable

Code can produce the same results more than once
<br>&nbsp;

:::

:::

::: {.column width='20%'}

::: {.callout-warning icon="false"}

## Reproducible

Running code regenerates the published results
<br>&nbsp;

:::

<p align="center">
  ↓<br>
  This work
</p>

:::

::: {.column width='20%'}

::: {.callout-warning icon="false"}

## Reusable

Code can be adapted and used in new contexts
<br>&nbsp;

:::

<p align="center">
  ↓<br>
  STARS ^[Monks, T., Harper, A., & Mustafee, N. (2024). Towards sharing tools and artefacts for reusable simulations in healthcare. Journal of Simulation, 1–20. [10.1080/17477778.2024.2347882](https://doi.org/10.1080/17477778.2024.2347882)]
</p>

:::

::: {.column width='20%'}

::: {.callout-warning icon="false"}

## Replicable

New code based on described methods produces consistent results

:::

<p align="center">
  ↓<br>
  STRESS ^[Monks, T., Currie, C., Onggo, B., et al. (2019). Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines. Journal of Simulation, 13(1), 55–67. [10.1080/17477778.2018.1442155](https://doi.org/10.1080/17477778.2018.1442155)]
</p>

:::

::::

::: {.incremental}

* Working with journals "Simulation" and "Journal of Simulation".
* Developing training materials with HDR UK.

:::

## Acknowledgements

:::: {.columns}

::: {.column width='50%'}

Thanks for their supervision, support, and involvement to:

* **Tom Monks** <a href="https://orcid.org/0000-0003-2631-4481" aria-label="View ORCID record"><img src="images/orcid.png" alt="ORCID iD" style="height:30px; margin-left:15px; margin-right:6px; margin-top:-5px; margin-bottom:-5px; vertical-align:middle;"></a>0000-0003-2631-4481</h3>
* **Alison Harper** <a href="https://orcid.org/0000-0001-5274-5037" aria-label="View ORCID record"><img src="images/orcid.png" alt="ORCID iD" style="height:30px; margin-left:15px; margin-right:6px; margin-top:-5px; margin-bottom:-5px; vertical-align:middle;"></a>0000-0001-5274-5037</h3>
* **Nav Mustafee** <a href="https://orcid.org/0000-0002-2204-8924" aria-label="View ORCID record"><img src="images/orcid.png" alt="ORCID iD" style="height:30px; margin-left:15px; margin-right:6px; margin-top:-5px; margin-bottom:-5px; vertical-align:middle;"></a>0000-0002-2204-8924</h3>
* **Andy Mayne** <a href="https://orcid.org/0000-0003-1263-2286" aria-label="View ORCID record"><img src="images/orcid.png" alt="ORCID iD" style="height:30px; margin-left:15px; margin-right:6px; margin-top:-5px; margin-bottom:-5px; vertical-align:middle;"></a>0000-0003-1263-2286</h3>

:::

::: {.column width='50%'}

Thanks to the **authors** who made their code available under open licences, facilitating this research.

This project is supported by the Medical Research Council [grant number MR/Z503915/1].

:::

::::

:::: {.columns}

::: {.column width='50%'}

<p align="center">
  <img src="images/exeter.png" alt="University of Exeter" style="width:380px; margin-top:-40px; margin-bottom:-20px;">
  <img src="images/or_society.png" alt="The OR Society" style="width:380px; margin-top:-20px; margin-bottom:-20px;">
</p>

:::

::: {.column width='50%'}

<p align="center">
  <br>
  <img src="images/hdr_uk.png" alt="HDR UK" style="width:330px; margin-top:-20px; margin-bottom:-40px;">
  <img src="images/nhs_somerset.png" alt="Somerset NHS Foundation Trust" style="width:380px; margin-top:-20px; margin-bottom:-20px;">
</p>

:::

::::