architect: refine #3317 — delete shepherd + daemon brains, keep minimal spawn loop for multi-account

[rjwalters]

Refine #3317: delete shepherd + daemon brains, keep minimal spawn loop for multi-account

TL;DR

#3317 proposed deleting loom-daemon + the shepherd model entirely in favor of /loom:sweep. The framing was approximately correct but over-aggressive on one axis: the process-spawn loop is load-bearing for multi-account load balancing and cannot be deleted, even though the shepherd brain (~16.5k Python LOC) and daemon brain (~4.2k Python LOC) can. This proposal refines #3317 with the right kill list.

Net effect: ~21k LOC deleted, ~150 LOC of minimal spawn-loop added. Same order of magnitude as #3317's "10:1 simplification" claim, with one critical piece preserved.

Why this is the right cut now (not earlier)

A lot of historical investment went into making the Python shepherd robust: checkpoint/resume, judge-retry loops, milestone reporting, stuck detection, heartbeat tracking, force-mode label management, ~10 phase modules totalling 11,500 lines. That investment was correct given the tooling constraints of 2025: Claude Code's native parallel subagent dispatch was either unavailable, unreliable, or untested for orchestration workloads. Loom had to build robustness from scratch in Python because the harness couldn't be trusted to do it.

That constraint has relaxed. Claude Code now reliably dispatches parallel Task subagents at one level of depth (validated empirically up to --builders-per-wave 3 in sweep), handles streaming and tool-use without the pathologies that motivated the shell shepherd, and provides native progress reporting through its own conversation log. The robustness sweep needs is much smaller than the robustness the Python shepherd had to invent, because sweep delegates most of it to a more mature harness.

In short: the Python shepherd is not overengineered. It was correctly engineered for its era. The era has changed.

The constraint that survives: multi-account

Claude Code's Task tool (subagent dispatch) inherits the parent session's OAuth token. There is no in-session mechanism to rotate accounts across subagents. Multi-account load balancing therefore requires per-task process spawn — a fresh claude invocation through spawn-claude.sh that picks a token from .loom/tokens/.ranking at process start.

A single /loom:sweep instance is hard-pinned to one account for its entire run. For users running multi-account batches (Pro/Max with rotation across 3-10 OAuth tokens), this is the deciding architectural constraint.

The implication: a thin spawn loop must survive. Its only job is to pick tokens and launch claude -p '/loom:sweep <N>' per unit of work.

Kill list

Component	LOC	Disposition	Notes
loom-tools/src/loom_tools/shepherd/	~16,500	Delete	Sweep already implements curator/builder/judge/doctor/merge. Builder phase alone is 6,062 LOC; sweep delegates this to the loom-builder subagent.
loom-tools/src/loom_tools/daemon_v2/loop.py, iteration.py, actions/{completions,proposals,retry_blocked,spinning,support_roles}.py	~3,500	Delete	Work generation, pool scaling, support-role triggers — sweep partitions work eagerly; periodic role triggers move to GitHub Actions / cron (#3317 Gap A).
defaults/.claude/commands/loom/shepherd.md + shepherd-lifecycle.md (+ mirrored roles/)	~2,140	Delete	/shepherd is a signal-writer proxy to the daemon; dies with it.
agent_spawn.py	~800	Keep + simplify	Token selection + process launch. Strip shepherd-specific spawn paths; keep the wrapper-script integration.
claude-wrapper.sh, spawn-claude.sh	~600	Keep	Token retry on TOKEN_EXPIRED / TOKEN_EXHAUSTED; allowlist + bad-tokens handling; ranking-based selection.
loom-tools/src/loom_tools/daemon_v2/actions/shepherds.py (spawn path only) + signals.py + command_poller.py	~500	Distill into ~150 LOC spawn loop	New script picks ready issues by label, claims them, spawns claude -p '/loom:sweep <N>' per token slot. Replaces ~700 LOC of pool-management state.
.loom/signals/, .loom/progress/ directories	n/a	Optional keep	Useful for status visibility; sweep can write to .loom/progress/ via existing report-milestone.sh.
loom-daemon/ (Rust crate)	~?	TBD per #3317	Out of scope here; the Rust shell is mostly IPC + terminal management. May survive as the spawn-loop host.

Net delete: ~22k LOC. Net add: ~150 LOC spawn loop. Ratio holds.

Features to port to sweep before deprecation

Four features in the Python shepherd are worth porting; the rest can be lost without harm.

Port (must)

1. --resume from checkpoint after crash — Python shepherd writes a checkpoint per phase and resumes mid-flight on a new spawn. Sweep currently does not. Port estimate: 100-150 LOC in sweep skill (a .loom/sweep-checkpoint/<issue>.json write at each phase boundary; on entry, check for it and skip completed stages).

Port (optional, low cost)

2. Milestone reporting to .loom/progress/ — sweep can call the existing report-milestone.sh from its phase transitions. ~20 lines of additions.

Already in sweep

3. Judge-retry with backoff — sweep's inline Doctor cycle is the architectural equivalent. The Python shepherd's judge_retry milestones are redundant.

Shift to spawn loop

4. Mid-flight token rotation on 429 — when a sweep instance hits its account's rate limit, the spawn loop sees the process exit non-zero, marks the token bad, and relaunches /loom:sweep <remaining-issues> on a fresh token. This is coarser than the shepherd's per-call retry but operationally equivalent at the batch level, and it falls out of claude-wrapper.sh's existing error classification.

Proposed minimal spawn loop (~150 LOC)

# loom-spawn-loop.py — conceptual sketch
def main():
    while not shutdown_requested():
        ready = gh_issues_with_label("loom:issue", limit=50)
        slots_free = MAX_SHEPHERDS - count_running_processes()
        for issue in ready[:slots_free]:
            if claim_issue(issue):
                token = select_token_via_ranking()  # existing logic
                spawn_claude_p(
                    f"/loom:sweep {issue}",
                    token=token,
                    detach=True,
                )
        sleep(POLL_INTERVAL_SECONDS)

That's the whole loop. It owns nothing — no state file, no pipeline-state JSON, no completion harvesting, no warning system. Issue label transitions (loom:issue → loom:building → closed) are the only state, and claim_issue is an atomic label swap via gh issue edit. If the loop dies, the running sweep processes survive and finish their issues. If a sweep process crashes mid-flight, the next loop tick re-claims and --resumes.

Gaps to address

Gap	Status	Resolution
Periodic support roles (champion, auditor, guide on 5-15min intervals)	Open from #3317	GitHub Actions schedules or a separate /schedule cron; not the spawn loop's job.
Cross-session retry history (proposal cooldowns: last_architect_trigger)	Open from #3317	.loom/sweep-history.json (~50 LOC), as in #3317.
Sphere downstream coordination (loom-{daemon,shepherd} subagent files)	Open from #3317	Migration plan ahead of sphere's loom-install rebase. Same as #3317.
Sweep checkpoint/resume	New	Phase 0 port; ~150 LOC into sweep skill.
Daemon-state.json consumers (Tauri app dashboards, MCP tools)	New	Inventory uses; either retire or have spawn loop emit a compatible-but-minimal daemon-state.json for display.

Phased plan

Phase	Description	Status
0	Port --resume checkpoint behavior into sweep skill	Not started
1	Build the ~150-line spawn loop; add LOOM_USE_SPAWN_LOOP=1 opt-in	Not started
2	Soft-deprecate /shepherd skill (warns to user, still works); soft-deprecate daemon_v2 brain (works but warns)	Not started
3	Delete Python shepherd, daemon_v2 brain, and /shepherd skill; spawn loop becomes default	vN.0
4	Sphere coordination + downstream migration	Parallel to Phase 2-3

Open questions for the architect / champion

1. Spawn loop host: does this run as the Rust loom-daemon crate (slimmed dramatically), as a new Python loom-spawn package, or as a shell script? Bias toward shell — the loop is small enough to read in one screen.
2. Backwards compatibility: do existing /shepherd <N> invocations get rewritten to /loom:sweep <N> (alias), or hard-deleted at Phase 3?
3. Tauri app integration: the Loom desktop app reads daemon-state.json and .loom/progress/. Does it need a stable shape from the spawn loop, or do we let those files retire?
4. Should sweep itself gain a --detached or --daemon-mode flag so the spawn loop can launch it cleanly, or is claude -p invocation enough?

Related

• Original proposal: architect: evaluate deprecating loom-daemon + shepherds in favor of /loom:sweep + remote schedule #3317 (closed 2026-05-28)
• Subagent-dispatch hazard: Parallel shepherds stall before any external action (subagents-launching-subagents) #3289 (sweep's "one level deep" rule)
• Token rotation infrastructure: Token rotation: bootstrap pool from .env (loom-tokens bootstrap) #3234, Token rotation: integrate wrapper into loom-daemon and mcp-loom spawn paths #3236
• Sweep extensions merged: feat(sweep): add --builders-per-wave N for in-session parallel waves (daemon-free) #3316 (--builders-per-wave), feat(sweep): natural-language selector parsing for /loom:sweep #3318 (NL selectors), feat(sweep): --dry-run flag for /loom:sweep #3319 (--dry-run)

[rjwalters]

Curation: Acceptance Criteria, Risks, and Phase Dependencies

Curated 2026-06-01. Promoting to loom:epic and spinning out implementable phase children.

Success Metrics

At completion (Phase 3 merged):

1. LOC delta: net deletion of ≥18,000 LOC (target 21k, threshold 18k). Verified by git diff main --stat on the Phase 3 PR.
2. Multi-account preserved: rate-limit telemetry shows token utilization spread across pooled accounts within ±15% of mean over a 24h sweep run (vs. 100/0/0/... pinned to one account in a single sweep session).
3. No regression in throughput: measured PRs/hour over a 48h window post-cutover is within −10% of the pre-cutover baseline (allowing some slack for transitional bugs).
4. No orphaned references: ripgrep across the repo for loom-tools/src/loom_tools/shepherd/, daemon_v2/{loop,iteration}, shepherd.md skill returns zero hits outside of CHANGELOG/migration docs.

Acceptance Criteria (epic-level)

The epic is complete when all of the following hold:

• Phase 0 merged: sweep can resume from checkpoint after process kill mid-flight (manual test: kill claude PID during builder phase, re-launch with same issue, verify it skips curator+pre-flight and resumes at builder).
• Phase 1 merged: spawn loop runs as opt-in (LOOM_USE_SPAWN_LOOP=1); validated with a ≥3-account token pool processing ≥10 issues without manual intervention.
• Phase 2 merged: deprecation warnings emit on /shepherd and daemon_v2 startup; Tauri/MCP consumer audit complete with retire/keep decision per file.
• Phase 3 merged in a tagged vN.0 release: shepherd brain deleted, daemon brain deleted, /shepherd skill deleted, spawn loop is default.
• Sphere downstream coordination complete: sphere's loom-{daemon,shepherd} subagent files migrated or retired ahead of its next loom-install rebase.

Phase Dependencies

Phase 0 (checkpoint/resume) ──┐
                              ├─→ Phase 2 (soft-deprecate) ─→ Phase 3 (delete, breaking)
Phase 1 (spawn loop) ─────────┘                                      ↑
                                                                     │
Phase 2c (consumer audit) ────────────────────────────────────────── ┘

• Phase 0 and Phase 1 are parallelizable — they touch disjoint code (sweep skill vs. new spawn loop).
• Phase 2 cannot start until both Phase 0 and Phase 1 are merged and validated on real workloads.
• Phase 3 is gated on Phase 2c (consumer audit). It is a breaking change and must land in a vN.0 release with a migration guide.
• Sphere coordination runs in parallel to Phase 2 and must complete before Phase 3.

Risks & Mitigations

Risk	Likelihood	Impact	Mitigation
Sweep checkpoint format proves insufficient for some phase (e.g., builder mid-edit recovery)	Medium	Phase 0 reopens	Start with phase-boundary checkpoints only (curator-done, builder-done, judge-done, doctor-done); do not attempt mid-builder recovery. Mid-builder kills get retried from builder start — acceptable cost.
Spawn loop has subtle races with manual /loom:sweep invocations on the same issue	Medium	Phase 1 hardening loop	File-lock around claim_issue (already exists in agent_spawn.py:claim_issue); reuse it.
Mid-flight token exhaustion strands a sweep at the merge step (token dies after PR open but before merge)	Low	One issue requires manual finish	Spawn loop relaunches on next tick with --resume; sweep's checkpoint resumes at merge phase. Documented behavior, not a bug.
Tauri app's UI dashboards break when daemon-state.json shape changes	High	Phase 3 visible regression for desktop users	Phase 2c output decides: either retire the dashboards (acceptable if MCP get_health_metrics covers it) or have spawn loop emit a minimal-compat daemon-state.json.
Sphere installs break when downstream /shepherd references hit deleted skill	High	Downstream pain	Phase 4 (sphere migration) is hard-blocked-by Phase 3 PR; coordinate with sphere maintainers before merging Phase 3.
LOC delete count comes in under target (e.g., dependencies surface that we missed)	Low	Cleanup follow-up	Acceptable — the goal is "delete the brains," not hit a specific number. Filing follow-up issues for residual cleanup is fine.

Out of Scope (defer to follow-up issues)

• Rust loom-daemon/ crate disposition. That crate is the IPC/terminal/activity layer, not the orchestration brain. Whether it survives, gets slimmed, or hosts the spawn loop is decided in a separate proposal after Phase 3.
• Sweep parallelism increase beyond N=3. The current soft cap stands; expanding it is orthogonal to the deprecation.
• New sweep features. Phase 0 is the only sweep change. Other sweep enhancements (e.g., --max-waves, --no-judge) are deferred per the original /loom:sweep skill's "Limitations" section.
• MCP server consolidation. Existing mcp-loom tools stay. Whether they survive Phase 3 is decided in Phase 2c.

Phase Children

Filing six phase issues with loom:epic-phase + loom:curated. They will be linked from this comment as filed.

[rjwalters]

Phase Children Filed (2026-06-01)

The epic has been expanded into six phase children. Cross-references and dependency edges:

Phase	Issue	Blocked by	Blocks
0 — port checkpoint/resume to sweep	#3373	—	#3374, #3376, #3378
1 — minimal spawn loop (multi-account)	#3374	#3373	#3376, #3378
2a — support roles → GitHub Actions	#3375	—	#3376, #3378
2b — soft-deprecate /shepherd + daemon_v2 (warnings)	#3376	#3374, #3375	#3378
2c — consumer audit (daemon-state.json, .loom/progress/)	#3377	—	#3378
3 — delete shepherd + daemon brains + /shepherd skill (vN.0 breaking)	#3378	#3373, #3374, #3375, #3376, #3377	—

Parallelizable starting state: #3373, #3374, #3375, #3377 can all begin immediately (no inter-dependencies among that set). #3376 unblocks once #3374 and #3375 land. #3378 is the terminal phase.

Sphere coordination (the original Phase 4 from the epic body) remains a parallel track and is not filed as a separate issue here — it will be coordinated directly with sphere maintainers when #3376 lands and the deprecation window begins.

All phase children carry loom:epic-phase + loom:curated. Awaiting human promotion to loom:issue (or Champion auto-promotion in --merge mode).

[rjwalters]

Follow-on Phase Children Filed (2026-06-02)

Two additional phase children filed to close gaps identified during phase decomposition:

Phase	Issue	Why filed	Blocked by	Blocks
2d — Architect/Hermit work-generation cadence	#3381	Phase 2a (#3375) handles pure-cadence support roles; threshold-and-cooldown work generation needs its own design. Without this, Phase 3 ships a silent regression (no new proposals).	#3374 (Option A) or #3375 (Option B)	#3378
4 — Sphere downstream coordination	#3382	Tracking issue to ensure sphere maintainers migrate (or pin) before #3378 lands. Was previously in epic body only — easy to lose track of in comments.	None for inventory; #3374 + #3375 for sphere-side migration work.	#3378

Updated dependency picture for #3378 (Phase 3 deletion): now blocked by seven prerequisites — #3373, #3374, #3375, #3376, #3377, #3381, and #3382. The added prerequisites (#3381, #3382) are the right gates; both close real risks.

Decisions explicitly deferred (not filed):

• Rust loom-daemon/ crate disposition — premature; depends on Phase 1's Open Question 1 outcome. Worth a stub loom:architect proposal after Phase 3 (vN.0 breaking): delete Python shepherd + daemon_v2 brain + /shepherd skill #3378 merges.
• Cross-session state file (.loom/sweep-history.json) — folded into Phase 2d: Architect/Hermit work-generation cadence after daemon deprecation #3381 as an implementation detail rather than tracked separately.
• Schedule registry config (.loom/config.json schedule: block) — Phase 2a Open Question 2; defer until Phase 2a: move periodic support roles (Champion, Curator, Judge, Auditor, Guide) to GitHub Actions schedules #3375 implementation needs it.