Repository navigation
v1.91.157.0 perf(gen-skill-docs): drop the "When to invoke" body section from Claude SKILL.md files (port of #3129) - #3305
Merged
Conversation
The Claude catalog trim moved each skill's routing prose ("Use when...",
"Proactively invoke...", voice triggers) out of the frontmatter description
into a "## When to invoke this skill" section at the top of the body. Claude
Code reads only name + description to choose a skill and loads the body after
the skill is invoked, so that text cannot route anything and costs tokens on
every invocation. Keep the one-line trimmed description; drop the section.
The routing prose stays in each template's description, which
--catalog-mode=full and the other hosts still render. Tests that pinned the
section now assert its absence, or check the template instead.
(cherry picked from commit 707ccd6)
The Claude catalog trim drops each skill's "Voice triggers (speech-to-text
aliases)" line from its description, and the body section that used to hold
it is gone. {{VOICE_TRIGGER_ROUTING}} renders them in the root router's
routing block instead: one "Voice trigger ... → invoke /<skill>" line per
skill, read from the templates' voice-triggers lists, so disabled-skill
filtering applies to them like any other routing line. Other hosts keep the
line in each description and render nothing here.
extractVoiceTriggers moves to the new resolver so the generator and the
resolver share one parser.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 645e75b)
Output of `bun run gen:skill-docs --host all` after the previous commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ff4c813)
`bun test/helpers/capture-context-budget.ts`. Against a recapture of the base commit, 55 skills that lost the section get a lower ceiling and the root gstack router rises by 532 tokens for the voice-trigger routing lines (5,369 tokens lower in total). devex-review and eval-plan differ from the committed fixture only because it already lagged main. alwaysOnTotal is unchanged: the catalog descriptions did not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d886fce)
…tion Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f1789dc)
…budget ceilings, add #3129 CHANGELOG entry Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…, record the skill-fixture change as non-behavioral Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
Contributor
E2E Evals: ❌ FAIL121 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $32.58 rule/judge cost | reconcile exit: 1 Profile: pr / mode Derived dependents (scripts/pr-dependencies.ts):
Rule and judge shards
Fail-closed reconciliationSliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry. Failures and split verdicts |
…o-invoke Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md
…ion from Claude SKILL.md files (port of #3129) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
github-actions
Bot
changed the title
Contributor
|
flaky: qa-only-no-fix, failed in https://github.com/garrytan/gstack/actions/runs/39332134395/attempts/1 (eval-slices (8)), passed on rerun https://github.com/garrytan/gstack/actions/runs/39332134395/attempts/2, logged for the daily flaky-fix pass. flaky: review-exploratory-small-cli, failed in https://github.com/garrytan/gstack/actions/runs/39332134395/attempts/2 (it passed in attempt 1), passed on rerun https://github.com/garrytan/gstack/actions/runs/39332134395/attempts/3, logged for the daily flaky-fix pass. |
Contributor
|
Merging under Garry's flaky-test rule (2026-10-19). Every case passed after rule-16 reruns of only the failing slices: qa-only-no-fix (failed in attempt 1, passed twice on rerun) and review-exploratory-small-cli (failed once, passed on its next rerun). Both are logged in the flaky ledger and are unrelated to this diff. slices-report stays bound to attempt 1 by design until F1 lands. — GStack Minder |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports #3129 by @thanhnguyentung95 onto a crew branch so it can run the paid gate with secrets. The six commits are cherry-picks with the contributor as author. Two follow-up test commits fix what the port broke.
What changes
Every Claude SKILL.md used to open its body with
## When to invoke this skill, which holds the routing prose ("Use when asked to…", "Proactively invoke…", voice triggers). Claude Code picks a skill from its frontmatternameanddescriptiononly, and loads the body after the Skill tool has already run, so that section routed nothing. It only added tokens to every invocation.applyCatalogTrimno longer emits the section. The routing prose stays in each template'sdescription, so--catalog-mode=fulland the other hosts render as before.- Voice trigger "…" → invoke /<skill>line per skill (21 lines), built by a new{{VOICE_TRIGGER_ROUTING}}resolver.disabled_skillsdrops them like any other routing line.gstack(4289→4821, the voice lines), a net −5,139 tokens.Nothing Claude sees before it invokes a skill changes: the trimmed catalog description, the
## Skill routingblock in CLAUDE.md, and the frontmatter are all as before.Follow-ups on top of the contributor's commits
test/gen-skill-docs.test.ts:4456: the new voice-routing test counted theconnect-chrome→open-gstack-browseralias symlink as a skill. It now skips symlinks the waydiscoverTemplatesdoes (486/1 → 487/0).test/fixtures/context-budget.json: devex-review (20161) and eval-plan (9811) keep their old ceilings. The wholesale recapture had raised them, and this PR doesn't touch either skill.scripts/harness-version.json: records thetest/helpers/skill-fixture.tschange as non-behavioral, so HARNESS_VERSION stays 6. The fixture only drops a heading the generator no longer emits; extraction would throw without the change. The affected cases' new SKILL.md bytes already start their own series.test/helpers/manual-judge-review-fixture.ts: oneLATER_WAVE_EDITSentry puts the dropped section back when the historical cookie-workflow approval is rebuilt, so its hash still matches the approved7f7f76…. The approval file is unchanged. v1.91.150.0 test(evals): re-pin older paid models to the current generation; prosons and cookie-judge repairs (GSTA-49 TM3) #3293 edits the same file about 8 lines lower. Both merge cleanly, but whichever lands second should re-runtest/cookie-workflow-manual-review.test.ts.Gate integrity
Every un-pin follows from the removed output. The full list of gate edits is also in the CHANGELOG entry:
catalog-trim: 3 tests for the deletedbuildWhenToInvokeSectionare removed, and theapplyCatalogTrimassertions are inverted.catalog-mode-full: 1 assertion flipped, 2 dropped.cso-preserved: 1 assertion dropped.gen-skill-docs: the benchmark routing and CSO voice tests now read the template; one new voice-routing test.skill-validation: the trigger and proactive phrase checks now read SKILL.md.tmpl.parity-harness:mustHaveHeadingsdrops the heading.skill-fixtureandskill-e2e-retro: the heading is gone from the section lists.No threshold, trial count or case kind changed, and only the
gstackbudget ceiling rises.Validation
Routing gate (paid, Opus 5.5, one trial each, all PASS, $0.74 total): journey-ideation, journey-plan-eng, journey-debug, journey-qa, journey-code-review, journey-ship, journey-docs, journey-retro, journey-design-system, journey-visual-qa, journey-negatives, context-save-routing. These ran at f5d982a. ce59a32 changes nothing these cases read, so the results carry over as reused evidence.
Focused tests:
All pass, 0 fail.
gen:skill-docs --dry-runreports 0 stale files and typecheck is clean.Pregate: preflight passes 6/6. Remote free tier: 17,226 tests, 0 failed, $0.10. The first remote run caught the 3 failures the follow-up commit fixes.
test/gstack-ff-update.test.tsfailed once and passed on the serial retry; it is unrelated to this diff.Overlaps:
design-review/SKILL.mdwith #3284, which merges cleanly but needs a size-guard re-pin by whichever lands second, andsetup-browser-cookies/SKILL.mdwith #3293, which merges cleanly.CHANGELOG entry is under
[Unreleased]. Unstamped.