Skip to content

test(notebooks): run the demo notebook end to end on both backends - #1214

Draft
ggprior wants to merge 3 commits into
mainfrom
georg/colab_notebook_ci
Draft

test(notebooks): run the demo notebook end to end on both backends#1214
ggprior wants to merge 3 commits into
mainfrom
georg/colab_notebook_ci

Conversation

@ggprior

@ggprior ggprior commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

TL;DR: TODO

Claude generated description

The demo notebook had no execution coverage — only the nightly check that its
pip install lines resolve to the latest release. It had drifted: the
synthetic-data cell raised on the client backend, and three separate bugs sat
behind that one symptom, in three repos.

CI

tests/test_notebook_execution.py runs all 86 cells in a real Jupyter kernel,
once per backend, and checks the ROC AUC and RMSE the notebook prints still
clear a floor. Nightly, on dispatch, and on PRs touching the notebook. Failures
on the nightly open or comment on a tracking issue, matching the existing
install check; the executed notebook is uploaded as an artifact either way.

The notebook installs its own dependencies, so the workflow seeds the venv with
an editable install of the checkout first — uv pip install tabpfn audits an
already-satisfied requirement instead of pulling PyPI's wheel over the top,
which is what makes the run test the commit rather than the release.

Runtimes on a GPU runner: 4m30s local, 9m client.

Notebook fixes

Three things blocked an unattended run:

  • The backend prompt. TABPFN_DEMO_BACKEND=local|client now skips it;
    anything else still prompts, so interactive use is unchanged.
  • The token prompt. The local branch checked the Colab secret and then went
    straight to getpass, ignoring a TABPFN_TOKEN already in the environment.
    Environment first, then Colab secret, then prompt.
  • gdown. The text-data cell calls it with --fuzzy, a flag current
    versions dropped, and nothing installs it — it only worked because Colab
    ships an older one preinstalled. Now installed, and called with -O so the
    output filename does not depend on Drive metadata.

Depends on

The client-backend job stays red until two upstream fixes ship, both surfaced by
this workflow:

  • PriorLabs/tabpfn-extensions#390 — outlier detection calls .clone() on
    logits, which is a tensor locally and an array from the client.
  • PriorLabs/tabpfn-client#369output_type="full" returns NaN where a
    local prediction has -inf, and raises above the full-output row cap instead
    of splitting.

Merge those, release both, then this.

Also checked

No notebook path needs a Hugging Face token. All five Prior-Labs/* model repos
are ungated and resolve anonymously, and a fresh HF_HOME with HF_TOKEN unset
downloads and predicts on TABPFN_TOKEN alone. The notebook only mentions
Hugging Face in prose.

Executes all 86 cells of the demo notebook in a real kernel, once per
backend, and checks the ROC AUC and RMSE it prints still clear a floor.
Nightly, on dispatch, and on PRs that touch the notebook.

The notebook installs its own dependencies, so the workflow seeds the venv
with an editable install of the checkout first: `uv pip install tabpfn`
audits an already-satisfied requirement rather than pulling PyPI's wheel
over the top, which is what makes the run test this commit.

Three things stood between the notebook and an unattended run:
the backend prompt, now skippable with `TABPFN_DEMO_BACKEND`; a token
prompt that ignored a `TABPFN_TOKEN` already in the environment; and
`gdown`, which the text-data cell calls with a flag current versions
dropped and which nothing installs outside Colab.
@ggprior
ggprior force-pushed the georg/colab_notebook_ci branch from bea09d9 to 5092ca1 Compare August 27, 2026 14:20
Some notebook dependencies build from source, and the tool-cache Python on
the GPU runner image reports an include prefix that is not where it was
installed, so the build cannot find Python.h. A uv-managed CPython keeps its
headers and pkg-config prefix in one tree.
Cells that shell out to an installed command resolve it on PATH, not through
the kernel's interpreter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant