How to run multiple notebooks in parallel

[ddreyfus]

The documentation states: The general workflow when using papermill is parameterizing a notebook, executing it, as well as storing the results. In addition to operating on a single notebook, papermill also works on a collection of notebooks.

How would one execute a collection of notebooks?
Are they executed in series?
Is papermill thread safe such that I can execute multiple via a thread pool?

Thank you

[MSeal]

Sorry for the slow response.

How would one execute a collection of notebooks?
That documentation is a little dated. We should probably remove the statement or clarify how to use papermill in parallel.

Are they executed in series?
No, but that's done sometimes by queuing commands.

Is papermill thread safe such that I can execute multiple via a thread pool?
Yes, though there's a known issue if you set the cwd that is not threadsafe: #473. It's also multi-process safe, which is usually what I use.

[ddreyfus]

Thank you for the response. Yes, an example in the documentation would be great.

…

On Feb 3, 2021, at 12:39 PM, Matthew Seal ***@***.***> wrote: Sorry for the slow response. How would one execute a collection of notebooks? That documentation is a little dated. We should probably remove the statement or clarify how to use papermill in parallel. Are they executed in series? No, but that's done sometimes by queuing commands. Is papermill thread safe such that I can execute multiple via a thread pool? Yes, though there's a known issue if you set the cwd that is not threadsafe: #473 <#473>. It's also multi-process safe, which is usually what I use. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#578 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4GHGC4G6WEGRJC3OPAL53S5GC4XANCNFSM4W26ZTPQ>.

[retorquere]

I'm actually looking to execute notebooks in series, but they seems to get started in a sort of interleaved fashion; I can see output indicating notebooks starting while others are still running (again, going by their output). I currently have

import papermill

notebooks = [
  'LeeftijdsgroepenLandelijk',
  'RegioV2',
  'DagOverzicht',
]

for nbname in notebooks:
  print('Updating', nbname)
  papermill.execute_notebook(f'{nbname}.ipynb', '/tmp/output.ipynb', stdout_file=sys.stdout, stderr_file=sys.stderr)

edit: I should have noted that all the notebooks being executed have async/await procedures. I think it's those sections that get overlapped, but replacing this:

  papermill.execute_notebook(f'{nbname}.ipynb', '/tmp/output.ipynb', stdout_file=sys.stdout, stderr_file=sys.stderr)

with this:

  from subprocess import run
  result = run(f'papermill --log-output {nbname}.ipynb /tmp/output.ipynb', shell=True, capture_output=True, text=True)
  print(result.stdout)
  print(result.stderr)

did have them executed one by one.

[edublancas]

You can use Ploomber for this. I built it on top of papermill to allow assembling multi-notebook pipelines. If you have independent notebooks, it'll run them in separate sub-processes. Here's an example.

I'm the maintainer so feel free to open an issue on ou repository if you have questions!

[aiqc]

Airflow has a papermill operator.

You need to coerce airflow to allow parallel runs:

airflow.cfg [core] parallelism, max_active_runs_per_dag