test(notebooks): run the demo notebook end to end on both backends - #1214
Draft
ggprior wants to merge 3 commits into
Draft
test(notebooks): run the demo notebook end to end on both backends#1214ggprior wants to merge 3 commits into
ggprior wants to merge 3 commits into
Conversation
Executes all 86 cells of the demo notebook in a real kernel, once per backend, and checks the ROC AUC and RMSE it prints still clear a floor. Nightly, on dispatch, and on PRs that touch the notebook. The notebook installs its own dependencies, so the workflow seeds the venv with an editable install of the checkout first: `uv pip install tabpfn` audits an already-satisfied requirement rather than pulling PyPI's wheel over the top, which is what makes the run test this commit. Three things stood between the notebook and an unattended run: the backend prompt, now skippable with `TABPFN_DEMO_BACKEND`; a token prompt that ignored a `TABPFN_TOKEN` already in the environment; and `gdown`, which the text-data cell calls with a flag current versions dropped and which nothing installs outside Colab.
ggprior
force-pushed
the
georg/colab_notebook_ci
branch
from
August 27, 2026 14:20
bea09d9 to
5092ca1
Compare
Some notebook dependencies build from source, and the tool-cache Python on the GPU runner image reports an include prefix that is not where it was installed, so the build cannot find Python.h. A uv-managed CPython keeps its headers and pkg-config prefix in one tree.
Cells that shell out to an installed command resolve it on PATH, not through the kernel's interpreter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: TODO
Claude generated description
The demo notebook had no execution coverage — only the nightly check that its
pip installlines resolve to the latest release. It had drifted: thesynthetic-data cell raised on the client backend, and three separate bugs sat
behind that one symptom, in three repos.
CI
tests/test_notebook_execution.pyruns all 86 cells in a real Jupyter kernel,once per backend, and checks the ROC AUC and RMSE the notebook prints still
clear a floor. Nightly, on dispatch, and on PRs touching the notebook. Failures
on the nightly open or comment on a tracking issue, matching the existing
install check; the executed notebook is uploaded as an artifact either way.
The notebook installs its own dependencies, so the workflow seeds the venv with
an editable install of the checkout first —
uv pip install tabpfnaudits analready-satisfied requirement instead of pulling PyPI's wheel over the top,
which is what makes the run test the commit rather than the release.
Runtimes on a GPU runner: 4m30s local, 9m client.
Notebook fixes
Three things blocked an unattended run:
TABPFN_DEMO_BACKEND=local|clientnow skips it;anything else still prompts, so interactive use is unchanged.
straight to
getpass, ignoring aTABPFN_TOKENalready in the environment.Environment first, then Colab secret, then prompt.
gdown. The text-data cell calls it with--fuzzy, a flag currentversions dropped, and nothing installs it — it only worked because Colab
ships an older one preinstalled. Now installed, and called with
-Oso theoutput filename does not depend on Drive metadata.
Depends on
The client-backend job stays red until two upstream fixes ship, both surfaced by
this workflow:
PriorLabs/tabpfn-extensions#390— outlier detection calls.clone()onlogits, which is a tensor locally and an array from the client.PriorLabs/tabpfn-client#369—output_type="full"returnsNaNwhere alocal prediction has
-inf, and raises above the full-output row cap insteadof splitting.
Merge those, release both, then this.
Also checked
No notebook path needs a Hugging Face token. All five
Prior-Labs/*model reposare ungated and resolve anonymously, and a fresh
HF_HOMEwithHF_TOKENunsetdownloads and predicts on
TABPFN_TOKENalone. The notebook only mentionsHugging Face in prose.