Add /dogfood: harvest contribution friction with a team of small-model agents - #876
Open
0-jake-0 wants to merge 3 commits into
Open
Add /dogfood: harvest contribution friction with a team of small-model agents#8760-jake-0 wants to merge 3 commits into
0-jake-0 wants to merge 3 commits into
Conversation
…l agents Runs a team of agents through the repo's own skills — /new-op, /bind-adapter, the examples rules — each implementing a real gap in an isolated worktree, and collects where the repo made the work harder than it needed to be. The code the agents write is the vehicle; the deduplicated friction list is the product. The counter-intuitive part, and the reason the recipe specifies it: run the team on a deliberately weak model. A strong one infers past ambiguity in the docs and reports nothing, so its silence is not evidence. Broken code from a weak agent is an expected outcome, not a failed run. Encodes the operational details that are easy to get wrong: pre-warming a shared CARGO_TARGET_DIR so eight concurrent worktrees fit on a sandbox disk, the build commands to forbid, why agents must not commit (the pre-commit hook runs exactly the --all-targets build the disk rules avoid), and the friction-log template. Carries the lessons of the first run, all of which are about not trusting the reports: verify every claim against the build and the diff, because they are wrong in both directions; treat a weak model's root-cause *diagnosis* as a symptom report; check "X is undocumented" against the docs before believing it; and discount the stale-cache complaints the shared target dir manufactures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YaiuuXWEf3FnSmBMASvqvK
0-jake-0
pushed a commit
that referenced
this pull request
Aug 17, 2026
A team of small-model agents was run through /new-op, /bind-adapter and the examples rules (the /dogfood skill, PR #876). Two of the gaps they found were rules this repo states but nothing checks, so both are now enforced rather than merely written down: * wingfoil-python denies clippy::panic / unwrap_used / todo / unimplemented outside #[cfg(test)]. An agent introduced the crate's first panic! and neither the skill nor clippy noticed. cfg(not(test)) is what makes this precise -- the 300+ unwraps in test modules stay legal, so no per-module opt-outs are needed, and workspace CI already runs the clippy that enforces it. expect is deliberately still allowed. * scripts/check-python-bindings.sh, wired into rust-test.yml beside the example-docs check. A binding needs six separate registrations and a missing one compiles cleanly, silently dropping the adapter from the wheel. The aeron/iceoryx2 wheel exemptions are encoded with their reasons so they stay decisions. It lives in the unfiltered lint job rather than python-test.yml, whose paths filter would miss a registration dropped from elsewhere. Both were checked against an injected violation, not just observed green. The doc fixes: an example whose README pins output must run under HistoricalFrom, because the realtime README rots by design -- the run that produced it does not reproduce. /new-op gains the Default-state-vs-latch trap that made take_while and skip_while silently wrong, the rule that a shared dispatcher's signature is not one caller's to change, and notes on family scope and tuple outputs. /bind-adapter loses a shortcut that is wrong for lines: an unconditional engine module can still hold #[cfg(feature = "async")] items, so the binding needs a feature after all. The prototype patches are discarded; the run's logs and verified outcomes are kept under .claude/friction-runs/, including the reported findings that did not survive checking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YaiuuXWEf3FnSmBMASvqvK
5 tasks
Two changes from running it once. The model choice is no longer offered. Asking implied Sonnet or Opus were reasonable alternatives, and for this skill they are not: a strong model infers past the ambiguity being measured and reports nothing, so its silence looks like a clean bill of health. Haiku is the instrument, and the recipe now says so without hedging. The team is now a fixed eight rather than a suggested range: six examples, one op with Python bindings, one new adapter with Python bindings. The first run spent four agents on ops, which over-sampled one recipe and left the examples rules -- a completely separate rule set, and the only one with a CI checker behind it -- covered by three. Examples now carry the majority and must each touch a different corner of the feature and adapter surface rather than clustering in core/. The adapter task deliberately spans /new-adapter and /bind-adapter. It is the heaviest in the set and the likeliest to come back unfinished, which is the point: it is the only task crossing two skills, and that seam has never been walked end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YaiuuXWEf3FnSmBMASvqvK
The six example slots were previously chosen from a hand-waved list of features. That is the one part of the recipe most likely to waste an agent, because picking a target that already has an example produces no friction data at all -- and picking from memory is exactly how that happens. Checking the tree while writing this found two would-be candidates, demux and fan, already covered; demux has two examples of its own. So the skill now derives the list instead: three commands that read out the declared example targets and their features, the whole feature surface, and the adapter list, then diff. Candidates are confirmed with a grep against examples/ before being assigned. Also records the coverage gaps found while writing this, as a seed rather than as truth -- latency capture (reachable only inside the multi-process showcase examples), the cache and market adapters (no example anywhere), the Signal facade, window/buffer, the profiling and instrument-* features, and the TLS variants -- alongside an explicit do-not-bother list, so a future run does not rediscover that demux, fan and the nested island tier are already done. One selection rule worth keeping: an example needing a live external service is a poor slot, because the agent cannot run it, and an example that cannot be run cannot have real sample output -- which is the rule most worth testing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YaiuuXWEf3FnSmBMASvqvK
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adds one file:
.claude/commands/dogfood.md, a skill that runs a team ofagents through the repo's own skills —
/new-op,/bind-adapter, the examplesrules — each implementing a real gap in an isolated worktree, then collects and
deduplicates the friction they hit into a menu of fixes.
Nothing else in the tree is touched. No code, no CI.
Why
We have three living skills and a
CLAUDE.mdthat encode how to contributehere, and no systematic way to find where they have drifted from what a
contributor actually experiences. Reading them does not reveal the gaps —
following them does.
The design choice worth arguing about is using a deliberately weak model.
It is counter-intuitive and it is the point: a strong model silently infers
past ambiguity and reports nothing, so its silence is not evidence that the
docs are good. A weak model hits every under-specified step head-on. The
corollary, stated explicitly in the skill so agents do not over-claim, is that
broken code is an expected outcome rather than a failed run.
The skill also encodes the operational details that are easy to get wrong and
expensive to rediscover: pre-warming a shared
CARGO_TARGET_DIRso eightconcurrent worktrees fit in a sandbox's disk allowance, which build commands to
forbid, why agents must not commit (the pre-commit hook runs precisely the
--all-targetsbuild the disk rules exist to avoid), and the friction-logtemplate that makes the reports comparable.
How it was verified
cargo fmt --all— no Rust in this diffcargo lint— unaffected, documentation onlycargo test -p wingfoil --all-features— not applicableThe real verification is that the skill has been run end to end once, on eight
agents across ops, examples and Python bindings. That run produced the findings
in #874 and, per the skill's own "feed lessons back" mandate, its lessons are
already folded into the "What the first run taught" section here.
Notes for the reviewer
The most useful section is the one about not trusting the reports, and it
is written from being burned. On the first run: one agent reported a hard
blocker in
#[op(fluent)]that was simply false — the code compiled and itsown op logic was broken instead, which it never noticed, and acting on that
report would have sent someone into
wingfoil-derivefor nothing. Anotherreported clean success while introducing the first
panic!into a crate thathad zero. A third reported "all tests pass" for an op that had silently
deviated from its spec and encoded the deviation as a passing test. Several
"X is undocumented" findings were wrong on checking. Budget review time
accordingly — verification was the bulk of the work, not the agent runs.
One known cost, deliberately kept. Sharing a
CARGO_TARGET_DIRacrossagents is what makes eight of them fit on a sandbox disk, and it manufactures
its own false signal: three of four op agents reported needing
cargo cleanfor "proc macro regeneration", which was fingerprint thrash rather than a repo
defect. The skill says to keep the sharing and discount that class of report.
This was split out of #874 so the tooling can be judged on its own terms
rather than bundled with the fixes it produced. #874 is the findings; this is
the instrument.
Generated by Claude Code