Experiment new workflow with Jupytext and more integrated pyodide directive

[maximlt]

Warning! This is a hacky experimentation, no serious code review please 🙃

---

The goal is to manage to seamlessly combine:

1. Having markdown files as the source of the docs
2. Allowing the pyodide directive to be used in these files
3. Making it easy to update/create these source files, as if they were notebooks

In this PR I have quickly hacked something that seems to work well but needs more experimentation:

• Jupytext allows to edit MyST Markdown files as if they were notebooks (right-click on a md file and open as a notebook, it's pretty nice!). New code cells are saved in the notebook with a code fence like {code-cell} ipython3. I have added the jupytext.md file in the example site.
• Added a source parser that converts all {code-cell} to {code-block}.
• Changed the pyodide directive so that it inherits from the CodeBlock directive, and overrides it when Sphinx runs. Code blocks ({code-block} python) will be treated normally, those that have the pyodide flag will be treated exactly as the pyodide directive implemented by Philipp.

What is missing is a way to declare that all the {code-cell} of a markdown file should have the pyodide flag. This could probably be declared in the jupytext header and parsed by the pyodide extension. Right now the flag has to be added manually in the markdown file.

With this approach the markdown files look something like that (it's likely the header can be reduced, I haven't really tried, and yes the :pyodide: null is ugly, ignore that it can be improved for sure):

---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.0
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

# Jupytext

This Markdown file can be opened as a Notebook in JupyterLab with Jupytext installed. New code cells are created with the `code-cell` directive.

```{code-cell} ipython3
:pyodide: null

print('pyodide code-cell are turned into code-blocks and parsed with the PyodideCodeBlock directive')
```

```{tip}
Install jupyterlab_myst to display MyST markdown :)
```

This is how it will render when opened as a notebook in JupyterLab:

---

Pretty satisfied overall with the outcome of this little experiment so far!

[maximlt]

Jupytext supports multiple "notebooks as markdown" formats. In the first post I used the MyST format, that IIUC exists to make it easier to work with MyST-NB (and preserve tags like hide-output?). {code-cell} looks like a directive but in practice in MyST code it isn't one, it's used to collect all cells and "turn them into a notebook". It might not be such a good idea to rely on it? OTOH we use MyST-NB to build the docs so relying on this could give us extra benefits, i.e. convert all notebooks to Markdown, even those that are not meant to use the Pyodide directive (Datashader).

There's also the Jupytext notebook format that looks more conventional. Notebook tags, can be declared as such, we could use that to declare that this is code fence must be processes with the pyodide directive (first internally converting the code fence to a myst directive). An advantage of this approach is that it's closer to raw markdown (some renderers ignore whatever comes after python so highlight the code correctly, verified with the md preview in VS Code):

```python tags=["pyodide"]
param = 5
```

Still some analysis to do! Some question to answer too:

• How's the experience using JupyText?
• How much do we want to use it? I.e. would we be ready to convert all notebooks to that format?

[maximlt]

The explored workflow is meant to work best in Jupyterlab with Jupytext and Jupyterlab-MyST. VSCode doesn't yet have the same capabilities (built-in or via extensions), i.e. opening a markdown file as a notebook (well, Quarto does that but we're not using it). Refs: mwouts/jupytext#143, microsoft/vscode-jupyter#1240

[MarcSkovMadsen]

Feedback

Conclusion: I would not be the one pushing this as its not clear to me if Jupytext will help us in the long run. And I'm afraid the workflow will never be great in VS Code.

• ✔️ It improves the current workflow by enabling you to quickly move from .md to .ipynb format and back.
• ❌ It does not solve other issues with our documentation.

I believe there are other solutions like quarto with Panel plugin and mkdocs with pyodide extension that are worth exploring before switching to jupytext because they might solve more problems.

If jupytext does not get in the way of solving the rest of docs issues, then its ok for me to introduce. But it will require someone to do the work. And I don't get enough benefit to be pushing this across Panel or HoloViz.

Ideas for jupytext improvements

Pyoidide and Panelite

It might be an idea to support requirements in the meta data

And create python scripts that can convert

• a single .md file to a single pyodide .html file for preview. It does not matter its the full sphinx rendered .html file. Something simpler will also help you test if pyodide works. This will be very valuable.
• a Panelite notebook that includes the code to install the requirements. Then you can quickly open the notebook in https://panelite.holoviz.org/lab/ and test that it works. This assumes you are not using new features not yet released.

If this is implemented then my workflow would be improved and I would look more favorably at jupytext.

Review questions

Can this

• ✔️ Support a combination of pure Python code blocks and Pyodide code blocks?
• ✔️ Quickly get a preview of the markdown file.
• ✔️ render a standalone file in an efficient way as I update the file?
  • I think its possible to get live reload with rendering. How fast it is depends on how fast the notebook can be executed.
• ✔️ support some kind of automated testing to make sure our docs works.
  • I think its the same as today. We can render the markdown file and look for exception.
• ❓ Make the pyodide code interactive in the docs? I.e. can a user change the code and rerun it? Shiny Python docs supports this.
• ❓ Quickly get a pyodide powered preview? Currently I have to build the entire site (1½ hours) and then test.
• ❓work in a on a Jupyterhub where all other users don't use Jupytext?
• ❓ Make it fast to build our docs
• ❓ Work for teams. One open question I have is whether the kernelspec can get in the way. With pip that I use, its not that obvious how to control that name.
• ❌ Support developing our docs with auto reload
• ❌ work efficiently with VS Code? It would be that bad but its manual.
• ❌ be used for my own site awesome-panel.org or the PANEL-CHAT-EXAMPLES. No. nbsite is too complicated for that. And mkDocs just looks much better than Sphinx docs no matter the theme. It sells better

Tips and Tricks

Convert to notebook or html file

Convert your .md to a rendered notebook and/ or html file.

jupytext --to ipynb --execute doc/playground/pyodide/jupytext.md
jupyter nbconvert --execute --to html doc/playground/pyodide/jupytext.ipynb

VS Code Markdown Preview

Installation

See also https://discord.com/channels/1075331058024861767/1076148201326907472/1187350817905311794.

git clone https://github.com/holoviz-dev/nbsite.git
cd nbsite
git checkout experiment_directive_jupytext
cd site
python -m venv .venv
source .venv/Scripts/activate # windows git bash
pip install -r requirements.txt 

Start jupyter lab to render a notebook.

[MarcSkovMadsen]

Comparison to Quarto

With quarto you can create a document like

With an output of

I.e. you have two, live streamlit applications in the document.

There is a dedicated VS Code extension enabling you to run the code

There is a preview with live reload.

The quarto CLI is very fast. It can also render jupyter notebooks. Even with live reload quarto preview document.ipynb.

Its just light years ahead of jupytext.

quarto-vs-code.mp4

I would like a similar experience using some Python/ Jupyter tool chain.

[maximlt]

Thanks for your feedback Marc, it's very useful, I'll look more into it after the holidays. My quick feedback is that I think you're mixing too many things. My goal was trying to see whether it was possible to have all our files as Markdown files without changing our stack drastically and build new custom stuff. It seems what you are exploring is far far beyond the scope of what I've done.

Its just light years ahead of jupytext.

That seems a little unfair, the goal behind Quarto and Jupytext is very different, I've been amazed with the capabilities offered by Jupytext.