Skip to content

Latest commit

 

History

History
206 lines (169 loc) · 13.6 KB

scale-envelope.md

File metadata and controls

206 lines (169 loc) · 13.6 KB

Large-spoke scale envelope

How Hive behaves as a single spoke's backlog grows — how many open issues and PRs one spoke can carry before behaviour changes, what changes, and which knobs control it.

This page is about one big spoke. The roadmap already covers many spokes (constellation foundations, #5691) and shared provider capacity (#5698); neither answers "what happens at 300 open PRs in one hive", which is the question adopters actually hit first. #7392 asked for that dimension to be stated rather than discovered.

The design rule

Overflow must paginate, prioritize, or shed load — never fail closed on the work that would shrink the overflow.

A cap that stops the hive from doing the work that drains the backlog is self-reinforcing: the backlog causes the block, and the block prevents the backlog from shrinking. That is not a hypothetical — it is #7385, where a render cap overflowing produced a STAND DOWN, which is a correctness signal ("someone else may be working here"), not a size signal.

Two corollaries, both already true of the shipped caps and worth holding future ones to:

  1. A truncated list must say it is truncated. Every kick list that is cut ends with an explicit "… and N more" line (prListOverflowLine, src/pkg/scheduler/kickmessage.go:416). A silent slice reads to the agent as "this is the complete list".

  2. Truncation must preserve the prioritization signal, not cut an arbitrary slice. This holds today and is easy to assume backwards:

    • Issues are sorted by actionable rank tier, then oldest first within each tier before any cap is applied (RankActionableIssues, src/pkg/github/client.go:4637-4651, descending AgeMinutes).
    • PRs are sorted by review class then oldest-first within class (SortPullRequestsForReview, src/pkg/github/review_priority.go:334; fixes → refactors/docs → tests).

    Render caps then fair-share membership across repos while preserving that ordering within the selected items (fairShareByRepo, src/pkg/scheduler/repofilter.go:126-189). Lowering a cap does not hide one repo's work just because another repo sorted first, and omitted items return on later kicks as the list drains.

What is bounded and what is not

The important structural fact: caps in Hive are render-side, not enumeration-side.

Enumeration is deliberately complete. fetchIssues and fetchPRs page through every open item in every watched repo with no page ceiling (src/pkg/github/client.go:983-1001 and src/pkg/github/client.go:1160-1178, PerPage: 100 looping until resp.NextPage == 0). The governor therefore always sees true queue depth — which is what makes governor-thresholds.md scaling meaningful — and the caps below only decide how much of that is rendered into a kick prompt.

Consequences worth stating plainly:

  • API and wall-clock cost of a poll grows linearly with backlog, at roughly one request per 100 open items per repo per enumeration. On a ~300-PR, 16-repo spoke this is tens of requests per cycle. We have not published a measured rate-limit headroom figure for that shape; treat it as unmeasured rather than as known-safe.
  • Prompt size does not grow with backlog once the render caps apply — that is exactly what #7368 fixed.

Why prompt size is a hard constraint, not a cosmetic one

A kick is typed into a terminal, so prompt size converts directly into delivery time. The measured worst case before the PR cap existed: a spoke with 302 open PRs produced a 69.5 KiB kick (~36 KiB of it the PR list), against the budget documented in src/pkg/dashboard/prompt_history.go:26-59 (the figure is stated at line 44) — ~25 KiB worst case, ~8 KiB typical. Delivery of that prompt took ~3 minutes, and a long delivery window is what made the restart-during-delivery loop of #7363 reachable in practice.

That is the general shape of scale failure here: an unbounded list does not fail by being big, it fails by widening a timing window somewhere else.

Kick layout: static prefix first

Provider prompt caching (Anthropic, OpenAI) only reuses a byte-identical prefix, so the typed kick is laid out static-first (#12125). The order is fixed in src/pkg/scheduler/kick_layout.go and asserted by TestKickLayoutStaticPrefixIsCacheable:

# Block Varies
1 SECURITY CANARY preamble every kick (random, opt-in ioscan.canaries; always first — ADR-0008)
2 [agent:<name>] header never
3 governor work-source rule never
4 independent review section per agent config
5 workflow push ceiling per agent mode
6 formal quality capability per role + config
7 work tracker section per work-source config
8 resolved policy body (${GH_AUTH} is a constant; ${ISSUE_LIST}/${PR_LIST}/… stay where the template puts them) from the first list on
9 REPO-SCOPED CADENCE TARGET per cadence target
10 AUTHORIZED REPOS when repos are paused/scoped
11–14 hold-gated PR coordination, red-PR and review-thread fix-before-new, PR follow-up handoff every kick (data-bearing)
15–17 guide flow line, in-flight note, question answer contract every kick
18 in-progress claim rule whenever the kick lists issues
19 closing instruction (Begin now:) never; always last

So for a hive without canaries the cacheable prefix runs from byte 0 through the header, every static seam and the policy core up to the template's first dynamic list; with canaries it starts after one short random line. What did not change: /clear still precedes every kick, the canary is neither stabilised nor moved, the full contract is typed on every kick (no --append-system-prompt-file, no cached policy state), and every seam is still an unconditional post-resolution injection a customized template cannot omit — only the order moved. The data-bearing preflights (11–14) used to sit directly under the header, ahead of the policy body; they now sit at the tail, directly ahead of the closing instruction, so the body in front of them stays cacheable.

Inventory of cap-triggered behaviours

Every entry below was read in the tree at the cited symbol. "Tunable" means an operator can change it without rebuilding.

Cap Where Default Tunable Behaviour when exceeded
Issue list per kick governor.kick_limits.max_issues → Scheduler.issueCap(), src/pkg/scheduler/kickmessage.go:361 100 Yes (hive.yaml, or Settings → Repos) Fair-share across repos while preserving oldest-first order; list is cut silently in formatIssueListWithPolicy (kickmessage.go:40) and with an explicit note in the scanner/work lists
PR list per kick (actionable, stale drafts, merge-eligible, CI-failing, repair queue) governor.kick_limits.max_prs → Scheduler.prCap(), src/pkg/scheduler/kickmessage.go:371 50 Yes (same block/UI) Fair-share across repos while preserving review-priority order + "… and N more open PRs not listed (cap N per kick; they return on later kicks as this list drains)"
Ceiling on either cap config.MaxKickListCap, src/crux/config/kick_limits.go:18 500 No A configured value above 500 is pinned to 500, not rejected — so a bad hive.yaml degrades to a safe cap instead of failing startup (clampKickListCap, kick_limits.go:39-53). An absent key or a negative value means "use the default"; an explicit 0 is the operator's opt-out of capping (KickListUnlimited, kick_limits.go:27) — the render path then lists every item (repofilter.go:143-149)
Held-PR snapshot, per repo maxHeldPRsPerRepoPerKick, src/pkg/scheduler/policy_overlays.go:507 100 No (hardcoded) Fails closed, by design: the overflowing repo gets "STAND DOWN for <repo> this kick". Scoped per repo since #7391 so one crowded repo no longer blanks the snapshot for the other 15
Red-PR fix-before-new detail redPRFixMaxDetailed, src/pkg/scheduler/policy_overlays.go:175 5 No (hardcoded) Degrade to summary: "… and N more (full list: <path>)"
Review-thread fix-before-new detail same constant, used at src/pkg/scheduler/policy_overlays.go:452 5 No (hardcoded) Degrade to summary: "… and N more PRs (full list: <path>)"
CI-evidence excerpt per PR redPRFixExcerptRunes, src/pkg/scheduler/policy_overlays.go:176 400 runes No Truncate with "…"
Skill text injected per kick maxSkillsInjectionBytes, src/pkg/scheduler/kick_priming.go:138 8192 B No Shed load: skills are dropped whole, never truncated mid-body, and the drop is logged
Issues primed into AGENTS.md context maxIssuesToPrime, src/pkg/scheduler/kick_priming.go:13 5 No Truncate
Auto-merge sweep merges per pass DefaultAutoMergeSweepMaxMerges, src/pkg/github/automerge/automerge_sweep.go:375 3 Yes (auto_merge.max_merges) Shed load: remaining merges wait for the next pass
Task-list sweep closures per tick DefaultTaskListSweepMaxCloses, src/pkg/github/task_list_sweep.go:20 5 Partly (MaxCloses option) Shed load: remaining closures wait for the next tick
Prompt-history on disk promptHistoryMaxSizeMB × backups, src/pkg/dashboard/prompt_history.go:67 96 MiB worst case No Shed oldest: rotate + gzip, oldest prompts age out first
Issue/PR enumeration fetchIssues / fetchPRs, src/pkg/github/client.go:1653, src/pkg/github/client.go:1920 no cap n/a Pages to completion; cost grows with backlog

The inconsistency this inventory exposes

Two caps are operator-tunable (governor.kick_limits.max_issues / max_prs), and the rest are hardcoded constants — including maxHeldPRsPerRepoPerKick, which is the only cap in the table whose overflow behaviour is fail closed. The cap most likely to stop work at scale is the one an operator cannot adjust.

That is not obviously wrong: a fail-closed disjointness guarantee is a poor thing to let an operator weaken, and #7391's per-repo scoping removed most of the practical pressure to raise it. But the split is currently accidental rather than decided. A reasonable policy to adopt:

  • Truncating / load-shedding caps (prompt size, evidence detail, sweep rate) are presentation or pacing choices and should be tunable, with a ceiling — the MaxKickListCap = 500 pattern, where the ceiling exists specifically so the control cannot be used to recreate the failure it prevents.
  • Fail-closed caps (safety/correctness gates) stay hardcoded, and instead carry the burden of being scoped as narrowly as the guarantee allows — which is exactly the change #7391 made.

Deciding that explicitly is tracked on #7392; this page records the current state, not an accepted policy.

The envelope today

Stated honestly, with the evidence we have:

Backlog per spoke Status
Up to ~100 open PRs / ~100 open issues Inside every default cap. No list truncates, no degraded mode.
~100–300 open PRs Exercised in production (projectbluefin, ~304 open PRs across 16 repos, 2026-09-17). Kick lists truncate at the caps and say so; the governor still sees full depth. This is where the #7368 / #7363 / #7385 / #7386 cluster was found and fixed.
>300 open PRs, or >100 held PRs in a single repo Untested. The per-repo held-PR stand-down (maxHeldPRsPerRepoPerKick) is the first thing expected to bite, and it is not operator-tunable.
Any size, per-repo enumeration cost Unmeasured against GitHub rate limits at this shape.

No number here is a support commitment; it is a record of what has been run and what has not.

For contributors adding a cap

If your PR introduces a cap, threshold, or list limit, state in the PR body:

  1. What happens when it is exceeded — truncate, paginate, shed load, or fail closed.
  2. If it truncates, that the ordering puts the least important items in the dropped tail, and that the output says the list is partial.
  3. If it fails closed, why the guarantee needs it, and the narrowest scope it can hold (per repo, per agent — not per hive, if per repo will do).
  4. Whether an operator can change it, and if so what the ceiling is.

Related