Repository navigation
feat(multibranch): multibranch-dev integration branch (draft, not for merge) - #1869
Draft
xiaofei-zheng wants to merge 232 commits into
Draft
xiaofei-zheng wants to merge 232 commits into
xiaofei-zheng wants to merge 232 commits into
Conversation
xiaofei-zheng
commented
Oct 14, 2026
xiaofei-zheng
commented
Collaborator
- Description: Integration branch for the multi-branch architecture. Feature branches are merged into multibranch-dev directly; this draft PR exists only so CI runs on every push to that branch. It will not be merged as one PR: the work lands on main in reviewable batches.
- Linked issue(s): none
- Tests: CI runs on every push. The skeleton adds src/hyperloom_multibranch/tests/test_cli.py.
- Size/complexity triggers crossed: the branch diff will exceed the diff budget over time; it is split into batches before anything merges to main.
- If this simplifies or refactors: n/a
- Observable effect: none on main. The hyperloom-multibranch command exists only on this branch.
- Breaking changes: no
- PR addresses single concern: no, it accumulates the multi-branch work and is split before merge.
- Root cause is upstream: n/a
Code metrics gate: PASSEDJudged: the 339 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
A9 Tuple return (advisory, never fails): 3
A10 Raw vocabulary comparison (advisory, never fails): 1
A12 Single-implementation abstraction (advisory, never fails): 14
Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 1
Already over at the base (backlog, not this change's): 1
91 findings in files this change does not touchOutside this change (informational): 91
...and 66 more; the job log lists every row.
Thresholds and sources: |
…ply and differ A Recipe is what a multi-branch commit stores and what hosted measurement compiles an environment from: the launch config (server arguments and extra environment), a per-component patch stack against a fixed upstream commit, the kernel artifacts it loads, the base image digest and an optional runtime snapshot of an enablement build. Its parts have the shape of the Recipe KB config, patch and kernel columns; its digest names its content. A RecipeDelta is one change and the action a test request carries; its config part keeps the single-lineage recipe_delta shape, and apply composes server arguments with compose_server_args and dedupe_extra_server_args, the functions the launcher already uses. diff reports per-flag and per-variable changes (config conflicts are values both sides set differently), moved upstream refs, and patch and kernel sets compared by content hash. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…e protocol per interface The protocols are the interfaces of multi-branch design section 5.1 as code, so the toolbox, rule gate, scheduler, integrator and specialist runtime are written against them in parallel: test record and tool-result writes, node health, judgements and merge attempts, baseline and regression records, lineage, claims with same-state deduplication, branch logs, the inbox, progress and lineage, committed-commit queries, startup briefs, record search, session records and one snapshot of every view. Every method is a coroutine so callers keep their code when the notebook moves behind a server; errors are typed (ClaimHeld and AlreadyTested carry the deduplication answer). They live in hyperloom_multibranch.contracts.notebook, the one place the other modules import notebook types from. The records include the session profile the scheduler writes at PRELUDE (the seven serving dimensions of a Recipe KB canonical id, TP, concurrency, sequence lengths, the kernel optimizer and the roofline ceiling per metric) and a manifest fingerprint per activity (socket implementations, specialist prompt and scheduling policy versions); test records carry when their request, result and judgement arrived, commits when they were made, and branches when they opened and stopped. The intent, test request and result, judgement and branch log entry are stand-ins with only the fields the notebook reads, until their owners' types land; the fields they add beyond section 5.2 are proposals to those owners. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…s on as its fake LocalNotebook implements every notebook protocol on a directory that works the same on a local disk and on a shared NFS folder: segments/ holds one append-only JSONL segment per handle, so no write takes a lock, and claims/ holds claim generations, each created exclusively by linking a complete file into place, so of two writers racing on one claim exactly one wins and the other decides again. A small codec stores the frozen dataclasses as JSON, the recipe delta in its own document form. Each write checks that the writer's role owns the state: test requests and results the toolbox, specialist-branch judgements the rule gate, integration-branch judgements and merge attempts the integrator, branch starts and stops, the session profile and manifest fingerprints the scheduler, splits the rule gate, claims and branch logs the specialist runtime of that branch. Requests and results are idempotent per request_id and refuse conflicting rewrites; judgements append and the latest is in force. Claims are keyed by (parent commit, intent fingerprint) and held by a branch: the same action measured on the state is refused, the same intent measured through another action needs a stated reason, a held key or the same action in progress is refused as held, and the same intent on other states comes back for reference. The toolbox attaches its request and renews from then on; the result completes the claim; an unrenewed claim reads as expired. Every claim decision is logged for the intent-deduplication statistics. Lineage, progress, committed commits with their merge state, recorded values, regression cases by ancestry, startup briefs, the per-branch inbox, record search and profile records are derived from those entries, each fact written once. snapshot() returns every view at once, so the RSI, the dashboard and the exports read one consistent picture instead of the notebook's internals. The tests run the whole flow (submit a test, judge it, commit, merge, split with a regression case) and pin the refusals; a scripted session feeds the tests of what is built on the notebook. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The config column is the launch document; each component becomes one overlay stack whose refs follow the patch/overlays/<stack>/<member>-<name>.patch form the Recipe KB orders replay by, with a provenance row holding the component, its upstream repository and commit as base_sha, and every patch's own path and hash; kernel artifacts land in the rows list of their kind. The round trip is exact, and the KB's own patch-column check and warm-recipe reader accept the result. It lives in hyperloom_multibranch.notebook.knowledge, which the notebook's CLOSE write-back and PRELUDE warm start build on. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
… and exports the session breakdown publish_final_recipe files the record under the canonical id and scope the session profile gives, with the columns the single-lineage CLOSE writes: optimized_throughput and validated_e2e_gain, the workload shape, the recipe's value columns, and the experience lists. The patch and kernel files are copied out of the final commit's checkout under the refs value names, each checked against the hash the recipe pins; write_if_better promotes it only past the partition's champion. The single-lineage PRELUDE reader accepts the record. AgentX sessions are skipped with a reason until the gate records the interactivity objective their champion compares. lessons.py turns a snapshot into those lists: what worked best first, what failed worst first, pitfalls, the hypotheses settled, and the roofline gaps. breakdown.py writes session_breakdown.json with its own schema version, multibranch-1: the session, baseline, lineage, branches, tests, usage per measurement environment, and the write-back outcome. The tests read one scripted session written through the local notebook. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…t summaries A reader that has to know whether an Experience was measured under its own conditions (same GPU, framework, bottleneck, sequence lengths) had to parse the records back out of prompt_block, a rendering meant for a prompt. Each summary in /v1/read and /v1/list now carries the Experience's identity map, which prompt_block already shows in full, so nothing new leaves the service. The field is additive: existing clients read the summaries by key and ignore it. The HTTP reference and the optimizer skill list it. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…ecall them by condition experience.py exports every specialist test the rule gate judged, or that failed by the candidate's own fault, and the packaged multibranch-notebook-v1 mapping turns each into one inference-recipe-v1 Experience: the session profile plus the intent's target, mechanism and layer as conditions, the recipe delta as the action, the parent commit's and the candidate's recorded values with the judged gain plus or minus the threshold as the consequence, the request ids as evidence. The reasoning and reflection come from the branch log entries citing the request; the mapping refuses a candidate without reasoning, a recorded parent value or an outcome. A config change gets the fingerprint the single-lineage producer gives it. Judgements waiting on their split, and merges, are not published. recall.py reads once per branch with the context the Framework phase sends; an Experience measured under every queried condition is a prior, any other a lead to re-verify, with the differing conditions named. The mapping is package data. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…top threshold and a branch cap archive.py turns a finished notebook snapshot into a search tree on the session clock: each branch with when it opened and stopped, its commits' cumulative gains over the baseline, when its tests were requested and the GPU time they spent, plus the session's task conditions, its baseline's dispersion and the measurement backends it used. A shared archive keeps each session's tree once; the deployment names it with HYPERLOOM_RSI_ARCHIVE. similarity.py scores task similarity (labels match, sizes fade with their log ratio, bottleneck profiles compare by cosine) and counts effective sessions. replay.py only cuts what a session recorded, never adds a branch it did not open, and has two parameters: the stop threshold and the branch cap. It evaluates a policy over similar sessions; choosing one stays with the scheduler. backfill.py turns single-lineage sessions (journal, manifest, breakdown noise band and baseline measurements) into one-branch trees. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…hesis ledger analysis.py reads a session's own record three ways. A failed test (a regression, a flat result, a candidate that failed, one that failed the accuracy check) is same_state when tried on the branch head, ancestor when on a commit the head builds on, elsewhere otherwise; a later commit of the same intent on the same state redeems it, and failures_at tiers them against any commit. Direction occupancy shows, per target, mechanism and layer, the states it was tried on and how, the running branches whose head carries one of its commits, and who holds a live claim on it. The hypothesis ledger follows supersedes to each hypothesis's latest version. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…session's task as leads warm_start.py takes the single-lineage PRELUDE's two steps: search_identities for the canonical ids matching six of the seven serving dimensions (labelled as canonical_labels labels them, the framework version left free), then get_view for each one's champion in exactly the session's scope. Every record goes through knowledge_to_warm_recipe, so records either flow's CLOSE wrote read the same way; a WarmLead carries the gain, launch arguments and environment, lessons, pitfalls and what failed, and the dimensions where it differs from the session. A record that fails validation is logged and skipped. Only exact scopes are read: the KB Store does not document whether get_rollup without a scope lists a record's other scopes, so the nearest-scope fallback waits until it does. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…at other sessions left CrossSessionBriefing is the second StartupBriefProvider; the local notebook's brief stays the fake. It keeps the notebook's list of what was tried on the start commit and the states it builds on, and adds lessons and leads: this session's repeated failures in one direction on those states, Experiences recalled for the specialist's focus (priors measured under this session's conditions apart from those to re-verify), the warm-start champions, and their pitfalls and failures. Lessons and leads stay under a character cap, 4000 by default, the most specific first; at the root before anything was tried the brief is cross-session only. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…prototype FrontierScoring is a ProgressQuery over the notebook's: the same rows, each running specialist branch's frontier filled with an expected gain and its components: the gain trend over the last window, the tests and GPU hours spent, the headroom to the roofline bound PRELUDE recorded, the gain similar archived branches still made after spending as many tests (weighted by task similarity, left out under 1.5 effective sessions), and whether a kernel-layer direction it tested lately is still under observation. The expected gain is the prior where there is one, else the trend; observation keeps a floor and the headroom caps it. The module says the formula is a prototype until the scheduler's owner settles what the score means. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…the same task dispersion_prior answers Q1 before PRELUDE measures its own: for a model, TP and GPU it takes the archived sessions that ran them and returns the dispersion to expect, the measurement environment it describes, how many baseline measurements and which sessions back it. Sessions that measured in the requested environment give the median of their baseline dispersions; single-host measurements, which single-lineage sessions all are, only raise a lower value as a floor; without either, the noise bands old sessions graded under stand in; with nothing, the global median (0.38%) or p90 (3.30%), which the caller chooses. Pulse is not read yet. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
notebook_tools binds eight tools to one specialist's notebook handle and branch, in the shape of the context tools' specs (name, description, input schema, handler) as a dataclass: lookup, claim, release, search, record, branch_log, append_branch_log and brief, all prefixed notebook_. They are the notebook forwarding the tool layer registers for the specialist (design Q13). An intent is named by its target, mechanism and layer and a change by its recipe delta document, whose fingerprint the tool computes; the caller is the handle's writer and never an argument. Handlers return compact text: a line is at most 160 characters, a list at most 20 lines with a count of the rest, a reply at most 4000 characters. A refused claim comes back with who holds it and what was tried, and any failed call as text starting with error:. Renewing a claim, and everything only a program writes, has no tool. The brief comes from whichever StartupBriefProvider the tool layer passes, the notebook's own by default. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…page of it build_dashboard turns one notebook snapshot into JSON-ready data: each branch's progress, its head's gain over the baseline and its frontier score when the rows carry one; the tests waiting for a result and the claims held now (the toolbox's own queue, once it reports one, passed in as it is); every commit and branch, and the branches split off for a fix; per measurement environment the tests and GPU time; the nodes out of service. render_html makes one page of it with inline CSS, no script and nothing fetched, every value escaped. write_dashboard writes dashboard.json and dashboard.html atomically; when to rebuild them is the run entry point's call. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…t can carry its recipe digest The notebook keeps no recipe; the experiment repository stores one per commit. The notebook meets recipes through the commit ids it records and the recipe delta every test request carries, whose fingerprint is the action same-state deduplication compares. A tried direction in a startup brief now says what its test changed, not only its target, mechanism and layer: the arguments and environment set or removed, the components moved or patched, the kernels dropped or added, in one line (notebook.changes.describe_change). A fill specialist sees the change that was tried, and the brief tool renders it after the outcome. A judgement that makes a commit, and the root lineage event, may name the digest of the commit's recipe (Recipe.digest()). The commit node carries it, the session breakdown exports it with the lineage, and a commit keeps the digest it was first recorded with: naming another one is a conflicting write. CLOSE skips a final recipe whose digest differs from the one recorded for its commit (recipe_digest_mismatch). The parameter is optional, so a writer that does not know the digest yet writes as before; same-state deduplication still compares parent commits. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
siliangchen-amd
added
the
size-exception
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…ret scan does not read as a key Gitleaks' generic-api-key rule took `claim.key, ttl_seconds=10 * TTL` for a key followed by a 14-character value of entropy 3.52, just past its 3.5 threshold and ended by a space, and failed the secret scan of the push. Naming the value `long_ttl` ends the argument at the closing parenthesis, as the other renew calls in the test do, which the rule does not match. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…mmaries again main has since rebuilt the KB's read summaries around each declaration's field roles, so the one-line change no longer applies there and kept multibranch-dev from merging with main, which stops the pull request CI from running. The line returns as a small pull request against main. Until it lands, recall finds no identity in a summary, so every recalled Experience is one to re-verify and none is taken as a prior; a test now pins that. The end-to-end Experience KB test reads the stored identities from the export, which carries complete records. This reverts commit 5eb2335. Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
main moved hyperloom_kb to collect mapping format v2: the declaration names each category's fields, the rationale holds the preconditions and the reasoning, and the change family and fingerprint sit directly in the change. The multi-branch mapping follows, and the tests read an Experience's categories as the field maps they now are. Co-authored-by: Cursor <cursoragent@cursor.com>
The multi-branch rule gate grades on the same bars and cannot import a private executors module. _accuracy_gate re-exports the names for its current callers in the optimizer; they move over when multibranch-dev merges back into main.
…pair request contracts
…ontracts The scheduler and the integrator build against these before the git-backed repository and the specialist runtime exist; the in-memory fakes stand in for them in tests.
…rom their owner's contracts peers.py re-exports them for the notebook modules that still import from it; the re-export goes once those import the owner modules directly.
…m-retest comparison
…n the profiled commit ToolResultPointer gains commit_id, profile_request_id and op_shares (percent_of_total per op, at most MAX_OP_SHARES), all optional, so the Amdahl ceiling of a claim can be read on the parent commit. Older rows load with the defaults. Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/49 Co-authored-by: Cursor <cursoragent@cursor.com>
…mit's op shares ToolOutput can now carry commit_id, profile_request_id and op_shares, and the runner copies them into the ToolResultPointer. The tracelens and roofline tools fill them: percent_of_total per op from the analysis.json beside the analysis report, read by the main flow's own loader (exported as trace_analysis.analysis_json), the 50 largest only. The commit and the PROFILE request id come from the request's commit_id / profile_request_id params; with no commit or no table the shares stay empty. Every other tool keeps the defaults. Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/49 Co-Authored-By: Claude <noreply@anthropic.com>
…n the parent commit Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/49
…starts at, the first one when that fails Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/42
…ranch's head commit Adds the start_analysis specialist MCP tool. RealToolbox.submit_tool had no caller besides tests, so no specialist could profile the commit it works on. The route allows only tracelens and roofline, refuses any other tool with a named reason, and reads commit_id from the head of the caller's branch in the experiment repository, never from the caller. profile_request_id is kept only when the notebook holds a PROFILE test of that id, asked by the same branch, on that commit; otherwise the call is refused. A branch whose head cannot be read, and a toolbox without a tool executor, are refused by name. Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/49 Co-Authored-By: Claude <noreply@anthropic.com>
…text for the manifest fingerprint Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/29
…n its gain lands, as the role prompt says
…n its gain lands, as the role prompt says
… its tests are submitted with The role prompt has the specialist pass the claim's intent fingerprint to submit_test; the answer did not carry it, so a test went in without one and its claim never settled. It is on its own line so the line cap cannot cut it. Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/50 Co-authored-by: Cursor <cursoragent@cursor.com>
… it cannot reach the gate's bar notebook_claim takes target_share_pct, the summed share of GPU time of what the change speeds up. acquire raises CeilingTooLow when 100*w/(100-w) is below the smallest threshold_pct of the baseline; before a baseline there is no bar. The claim decision row keeps the ceiling and the bar, so refusals stay visible. The notebook keeps the latest TraceLens op shares a tool reported per commit, read with op_shares(commit_id), and the claim answer names the parent's largest ops, or says the share is unverified when the parent has none. Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/49 Co-authored-by: Cursor <cursoragent@cursor.com>
…ahl ceiling of its profiled head's largest op NotebookSnapshot gains op_shares by commit; FrontierBasis gains AMDAHL. score_at takes one Readings (observing, headroom_pct, amdahl_pct) instead of two keyword arguments, so the calibration passes the same bundle. Interface-Change: https://global.primus-safe.amd.com/board/siliang.chen/hyperloom-multibranch/issues/49 Co-authored-by: Cursor <cursoragent@cursor.com>
…s with their Amdahl ceilings StartupBrief gains op_shares, the latest a tool reported on the start commit, filled by the local notebook so the cross-session brief carries it too. render_brief opens with a Ceilings section, up to CEILING_OPS ops each with its share of GPU time and the most speeding it up can gain, and points at target_share_pct; an unprofiled commit has no section. Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/49 Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Oct 17, 2026
…y, for the repeated-literal gate Co-authored-by: Cursor <cursoragent@cursor.com>
…tart_analysis and claim with its target's share of GPU time Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/49
…nd through the session's real MCP tools
…led key and bundle ResultArchive.bundle_path names a stored bundle. Acceptance now journals the signing key the platform reported for the sandbox and the sheet fingerprint beside the entry, so HostedMeasurementBackend.reverify verifies the archived bundle again without a live runner (reverify_record). HostedExecutor.difference derives the rounds and accuracy from the verified bundle and names the first recorded field that differs, so an edited value with untouched hashes is caught. Interface-Change: https://global.primus-safe.amd.com/board/boards/hyperloom-multibranch/issues/53 Co-Authored-By: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.