Skip to content

chore(deps): bump renderers (tokenizer config-build fallback for Laguna)#2657

Merged
rasdani merged 1 commit into
mainfrom
chore/bump-renderers-tokenizer-rope-fallback
May 27, 2026
Merged

chore(deps): bump renderers (tokenizer config-build fallback for Laguna)#2657
rasdani merged 1 commit into
mainfrom
chore/bump-renderers-tokenizer-rope-fallback

Conversation

@hallerite
Copy link
Copy Markdown
Member

@hallerite hallerite commented May 27, 2026

Bumps the deps/renderers submodule from 2ec28a889ab3f0, pulling in renderers#72.

Why

Loading a tokenizer for the Laguna model series (poolside/Laguna-XS.2) crashed renderers with:

KeyError: "Missing required keys in `rope_parameters` for 'rope_type'='default': {'rope_theta'}"

AutoTokenizer.from_pretrained always builds the model config first (to resolve the tokenizer class), even for a plain fast tokenizer. Building Laguna's config runs HF's RoPE validator, which rejects its nested rope_parameters (full_attention/sliding_attention, no top-level rope_theta) when constructed outside vLLM's patch_rope_parameters. The KeyError escaped and killed the tokenizer load — a modeling-layer concern breaking something the tokenizer never needed.

What changed in renderers#72

When AutoTokenizer fails building the config, fall back to loading the repo's self-contained tokenizer.json directly via PreTrainedTokenizerFast (no model config, no RoPE). Modeling-agnostic, runs under the fastokens patch (Laguna keeps the Rust speedup), excludes custom auto_map tokenizers, and re-raises the original error otherwise.

With this bump, Laguna runs no longer need --no-renderers as a workaround.

🤖 Generated with Claude Code


Note

Low Risk
Dependency and lockfile updates with a localized tokenizer-load fallback; no auth, data, or core training-path changes in this diff.

Overview
Updates deps/renderers (renderers#72) so tokenizer loading no longer dies on Laguna-style models when Hugging Face config build hits RoPE validation (rope_theta / nested rope_parameters). Renderers now falls back to tokenizer.json via PreTrainedTokenizerFast when AutoTokenizer fails on config, so Laguna runs can use renderers without --no-renderers.

The visible diff refreshes uv.lock: advances exclude-newer, and relaxes the verifiers dev metadata constraint on openai-harmony from >=0.0.8 to >=0.0.4 (likely from re-locking after the bump).

Reviewed by Cursor Bugbot for commit 2ecd1db. Bugbot is set up for automated code reviews on this repo. Configure here.

Bumps the deps/renderers submodule 2ec28a8 (v0.1.8.dev28) -> 89ab3f0
(v0.1.8.dev35), pulling in PrimeIntellect-ai/renderers#72: when
AutoTokenizer.from_pretrained fails while building the model config
(e.g. HF RoPE validation rejecting nested rope_parameters for
poolside/Laguna-XS.2), fall back to loading the repo's self-contained
tokenizer.json directly. Fixes the tokenizer load crash for the Laguna
model series; loads under the fastokens fast path.

Re-locks uv.lock: renderers now floors openai-harmony at >=0.0.4 (renderers#69).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@hallerite hallerite force-pushed the chore/bump-renderers-tokenizer-rope-fallback branch from 0699f3a to 2ecd1db Compare May 27, 2026 22:26
@hallerite hallerite marked this pull request as ready for review May 27, 2026 22:26
@rasdani rasdani self-requested a review May 27, 2026 22:33
@rasdani rasdani merged commit d372706 into main May 27, 2026
22 checks passed
@hallerite hallerite deleted the chore/bump-renderers-tokenizer-rope-fallback branch May 27, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants