feat(experience-kb): add Distills, synced from Hyperloom-KB by zili-amd · Pull Request #1921 · AMD-AGI/Hyperloom

zili-amd · 2026-10-15T20:17:29Z

Description: what and why

Adds Distills to the Experience KB. A Distill is a pattern a user draws from a group of one schema's Experiences: a scope that selects the group, a claim written in a registered form, and the evidence, which is the group's members, each classified as supporting, contradicting, or irrelevant to the claim.
- Each version is immutable and freezes its evidence.
- Every listing and read checks that evidence against today's Experiences, and a changes inbox names the Distills whose evidence moved in a way their author should look at.
- Labels, restores, and exclusions carry Distills along with Experiences.
- distill sync writes every visible Distill as a markdown page into a directory a Curated Wiki searches. Neither wiki enters a prompt.
The PR has three commits:
1. src/hyperloom_kb exactly as AMD-BRAIN-Internal/Hyperloom-KB holds it at ad4647220. That is its PR Update README.md #8, already reviewed and merged there, and this commit matches it file for file.
2. The changes the code-metrics gate needs, which Hyperloom-KB does not run:
  - the Distill CLI is registered in two groups;
  - push, pull, and rebind are served from _dispatch_sync;
  - GlobalSync.pull reads its page through _next_page;
  - repeated literals are named once;
  - duplicated test setup is removed;
  - add_commands(..., distill_dir=...) gives distill sync a tool's default directory;
  - the Distill skill no longer names a global KB.
  These go back to Hyperloom-KB in a follow-up PR, so the two trees stay identical.
3. The Hyperloom integration:
  - workspace commands (distill sync defaults to distilled/ under the service's data home);
  - a Distill count in the workspace push and pull line;
  - the hyperloom-kb-distill skill installed beside hyperloom-kb;
  - pointers in the optimizer and setup skills;
  - docs/reference/experience-kb-distill.md and the Experience KB reference, which name a global KB only by its settings, as these docs already do.
Linked issue(s): none. The source is AMD-BRAIN-Internal/Hyperloom-KB PR Update README.md #8.
Tests: added and updated.
- New contract tests: src/hyperloom_kb/tests/test_distill_store.py, test_distill_api.py, test_distill_sync.py, and test_distill_cli.py.
- Updated: test_cli.py (embedding, distill_dir, the skills naming every command), test_http_service.py (the /health shape), and test_experience_kb_service.py (the workspace sync directory, the push and pull line, the skill pointers).
- Run locally:
  - pytest src/hyperloom_kb/tests src/hyperloom/inference_optimizer/tests/test_experience_kb_service.py src/hyperloom/inference_optimizer/tests/test_packaging_lint.py: 347 passed, 1 skipped.
  - ruff check and ruff format --check: clean.
  - diff-cover against origin/main: 96% of the changed production lines are covered.
  - In Hyperloom-KB, the same suite also passes on PostgreSQL 16.
Size/complexity triggers crossed: split, or why the shape is right:
- size-exception: 3487 production Python lines. Commit 1 is code already reviewed and merged in Hyperloom-KB, and splitting it here would only replay that review in pieces. Commits 2 and 3 are the parts new to this repository, and each reviews on its own.
- baseline-raise: src/hyperloom_kb/http_service.py goes from 1539 to 1729 lines (limit 1200). The Distill host methods and routes join the service's single request handler. Splitting http_service.py would move existing code unrelated to this feature, so it belongs in a change of its own.
- Every other gated limit is met for the files this PR touches: cyclomatic complexity 20, cognitive complexity 30, function length 80, nesting depth 5, duplication, dead code, and repeated literals. GlobalSync.pull goes from 86 to 82 lines, and RequestHandler._dispatch from 91 to under 80.
- Two advisory A12 findings remain. DistillHost and SyncedDistills are Protocols, each with one implementation. Each is declared by the lower module, the Distill package and sync, for the higher one, the HTTP service, to implement, which is the exemption that rule names.
If this simplifies or refactors: commit 2 moves code without changing any contract.
- The push, pull, and rebind routes move from _dispatch into _dispatch_sync, keeping their bodies and refusals.
- GlobalSync.pull reads its page through _next_page, which keeps the restart-from-the-start rule and the push reset after a lost cursor.
- The existing sync, global identity, and CLI tests pass unchanged, apart from the shared fixtures they now use.
Observable effect: what an operator will now see.
- CLI: hyperloom-kb (and python -m hyperloom.inference_optimizer.experience_kb_service) gains schema show, experience get, and distill explore | forms | add-form | preview | create | update | get | draft | list | changes | delete | restore-entry | sync. A preview the KB would refuse exits 1, as does a draft file that cannot be read. In a workspace, distill sync writes to distilled/ under the service's data home unless --dir names another directory.
- HTTP API: new routes GET /v1/schemas/{schema_ref}, GET /v1/experiences/{id}, and /v1/distill/....
  - GET /health gains distills: per schema, the visible and deleted Distills, how many needed attention at the last check, and when that check ran. Health never recomputes evidence, so it is as cheap as before.
  - Push and pull reports gain distills counts, and the workspace push and pull line shows them, unless the service still running predates Distills.
- Metrics: hyperloom_kb_distill_entries, hyperloom_kb_distill_writes_total, hyperloom_kb_distill_evidence_seconds, hyperloom_kb_distill_changes, and hyperloom_kb_distill_checked_timestamp_seconds.
- Logs: new audit actions register_distill_form, distill_create, distill_update, and distill_receive. The writes table records a record_kind.
- Skill: hyperloom-kb-distill installs under .agents, .claude, and .cursor beside hyperloom-kb.
- What it costs to be without it: today a pattern seen across Experiences lives only in someone's notes. It has no evidence attached, nothing says when new runs contradict it, and a Curated Wiki has nothing from the KB to search.
Breaking changes: yes, for rollback only. On start, a service migrates its database to schema version 2. The migration only adds tables and a column, but code from before this PR refuses a version 2 database, so rolling a workspace back to an older Hyperloom means restoring database.sqlite3 from before the upgrade. A global KB that predates Distills keeps syncing every Experience: a push counts the Distills it holds back under held_back and sends them once that KB is upgraded.
PR addresses single concern: yes, Distills. The three commits are the reviewed code, the changes this repository's gate needs, and the Hyperloom integration.
Root cause is upstream: not applicable.

src/hyperloom_kb as AMD-BRAIN-Internal/Hyperloom-KB holds it at ad4647220 (its PR #8), unchanged: Distills, the patterns a user draws from groups of one schema's Experiences, kept as versioned entries with frozen evidence, checked against new evidence on every listing and read, carried by labels, restores, exclusions, push, and pull, and written into a directory a Curated Wiki searches; their HTTP API, RemoteClient methods, the hyperloom-kb schema, experience, and distill commands, the hyperloom-kb-distill skill, and their tests. Co-authored-by: Cursor <cursoragent@cursor.com>

…e and give distill sync a tool's directory The Experience KB code taken from Hyperloom-KB meets every limit the code-metrics gate holds a change to: - cli registers the Distill commands in two groups, authoring and upkeep, each under 80 lines; - the request handler serves push, pull, and rebind from _dispatch_sync, and GlobalSync.pull reads its page through _next_page, so neither grows past what main holds; - the record fields, Draft fields, and query values written three times or more in one module are named once; - the CLI tests take their schema from the shared Distill fixtures, and the sync tests start their three KBs in one helper, so no test repeats another's lines. add_commands takes distill_dir: distill sync writes there unless it names --dir, which stays required without one. The hyperloom-kb-distill skill names a global KB nowhere, as the hyperloom-kb skill does, and says a tool may give --dir a default. Behavior is otherwise unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>

…r skill experience_kb_service runs every hyperloom-kb distill command on the workspace's service, defaulting to its schema; its distill sync writes the pages into distilled/ under the service's data home unless --dir names another directory. A push or pull line also counts the Distills it moved, unless the service still running predates them. The hyperloom-kb-distill skill installs beside hyperloom-kb, and the optimizer and setup skills point to it. The Experience KB reference gains the Distills page and their commands, routes, health counts, metrics, audit events, and data home entries, naming a global KB only by its settings as the rest of these docs do. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-10-15T20:19:03Z

Code metrics gate: PASSED with waived findings (`baseline-raise`)

Judged: the 31 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension	Limit	New	Baseline entries	vs base
Cyclomatic complexity	fails > 20	0	72	+0
Cognitive complexity	fails > 30	0	284	+0
Function length (lines)	fails > 80	0	449	+0
Nested block depth	fails > 5	0	27	+0
Module length (lines)	warns > 800, fails > 1200	1	53	+0
Duplicated lines per file	fails > 0	0	118	+0
Dead code (vulture)	fails > 0	0	2	+0

Check	Findings	Effect
Comments	0
English only	0
Production code imports test code	0
Repeated literals	0
A1 Private name of another package	0
A8 Flag argument	0
A9 Tuple return	0
A10 Raw vocabulary comparison	0
A11 Hidden global write	0
A12 Pure forwarder	0
A12 Single-implementation abstraction	2	advisory, never fails

New violations (not in the baseline) (waived by the `baseline-raise` label): 1

Unit	Dimension	Value	Limit
`src/hyperloom_kb/http_service.py:1`	Module length (lines)	1729	fails > 1200

A12 Single-implementation abstraction (advisory, never fails): 2

Where	Problem
`src/hyperloom_kb/distill/service.py:95`	A12 single implementation: `DistillHost` is abstract and only `ExperienceHTTPService` (src/hyperloom_kb/http_service.py) implements it; use the implementation until a second one exists
`src/hyperloom_kb/sync.py:284`	A12 single implementation: `SyncedDistills` is abstract and only `DistillService` (src/hyperloom_kb/distill/service.py) implements it; use the implementation until a second one exists

How to fix

New or worse: split the unit (extract a function, a class, a module) until it is back under the limit, or no worse than its recorded value. The baseline cannot absorb a new violation: CI refuses any baseline entry added or raised relative to the base branch.
Baseline out of date: something improved. Run python scripts/code_metrics.py --update-baseline and commit scripts/code_metrics_baseline.txt; the command only lowers or removes entries.
A12 Single-implementation abstraction: use the one implementation directly until a second one exists; a lower package declaring an interface for a higher one to implement is exempt.
Override: the baseline-raise PR label waives new, worse and baseline-growth findings and the failing design checks (they stay listed here). Use it only with the reason in the PR description; it waives nothing else.
Tools for a local run: pip install ruff==0.16.2 complexipy==8.0.1 vulture==2.16 and npm install -g jscpd@5.4.1 (or point CODE_METRICS_JSCPD at another jscpd command, such as npx --yes jscpd@<version>).

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 1

Module	Lines	Limit
`src/hyperloom_kb/remote.py`	929	warns > 800

Already over at the base (backlog, not this change's): 1

Unit	Dimension	Base	Now
`src/hyperloom_kb/sync.py:536` `GlobalSync.pull`	Function length (lines)	86	82

93 findings in files this change does not touch

Outside this change (informational): 93

Unit	Dimension	Baseline	Now
`src/hyperloom/orchestrator/enablement/recipe/projections.py` `project_launch_evidence`	Cognitive complexity	absent	36
`src/hyperloom/orchestrator/state/shared_state.py` `resolve_graded_comparison`	Cognitive complexity	absent	34
`src/hyperloom_kb/collect/mapping.py` `_experience`	Cognitive complexity	absent	31
`src/hyperloom_kb/schema.py` `FieldDeclaration.__post_init__`	Cognitive complexity	absent	32
`src/kernelforge/cli/forge_loop_command.py` `_prepare_measurement_driver`	Cognitive complexity	absent	36
`src/kernelforge/cli/forge_loop_command.py` `_read_kb_warm_start`	Cognitive complexity	absent	40
`src/kernelforge/loop/runner.py` `IterationLoop._collect_session_candidate`	Cognitive complexity	absent	44
`src/kernelforge/loop/runner.py` `IterationLoop._rule_on_attempt`	Cognitive complexity	absent	38
`src/hyperloom_kb/tests/test_global_identity_and_pull.py`	Duplicated lines per file	absent	13
`src/hyperloom_kb/tests/test_local_state.py`	Duplicated lines per file	absent	13
`src/hyperloom/inference_optimizer/cli/__init__.py` `_reexport_session_environment`	Function length (lines)	absent	112
`src/hyperloom/inference_optimizer/cli/__init__.py` `_resolve_agent_lineup`	Function length (lines)	absent	89
`src/hyperloom/inference_optimizer/cli/__init__.py` `_resolve_run_topology`	Function length (lines)	absent	92
`src/hyperloom/inference_optimizer/cli/__init__.py` `_resume_session`	Function length (lines)	absent	171
`src/hyperloom/inference_optimizer/cli/__init__.py` `_seed_fresh_session`	Function length (lines)	absent	183
`src/hyperloom/inference_optimizer/multi_node/cli.py` `_restart_multinode`	Function length (lines)	absent	107
`src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py` `_build_parser`	Function length (lines)	absent	84
`src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py` `_spawn_rank_actors`	Function length (lines)	absent	90
`src/kernelforge/cli/forge_loop_command.py` `_RunReporter.build_result`	Function length (lines)	absent	91
`src/kernelforge/cli/forge_loop_command.py` `_prepare_measurement_driver`	Function length (lines)	absent	140
`src/kernelforge/cli/forge_loop_command.py` `_read_kb_warm_start`	Function length (lines)	absent	154
`src/kernelforge/loop/knowledge_integration.py` `_measure_warmstart_field`	Function length (lines)	absent	83
`src/kernelforge/loop/runner.py` `IterationLoop._close_iteration`	Function length (lines)	absent	135
`src/kernelforge/loop/runner.py` `IterationLoop._collect_session_candidate`	Function length (lines)	absent	187
`src/kernelforge/loop/runner.py` `IterationLoop._implementer_history`	Function length (lines)	absent	112

...and 68 more; the job log lists every row.

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

_{Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.}

+    metrics: Metrics
+
+    @property
+    def code_digest(self) -> str: ...


+    def code_digest(self) -> str: ...
+
+    @property
+    def schema_refs(self) -> tuple[str, ...]: ...


+    @property
+    def schema_refs(self) -> tuple[str, ...]: ...
+
+    def declaration_for(self, schema_ref: str) -> ExperienceDeclaration: ...


+
+    def declaration_for(self, schema_ref: str) -> ExperienceDeclaration: ...
+
+    def visible_experiences(self, schema_ref: str) -> tuple[Experience, ...]: ...


+
+    def visible_experiences(self, schema_ref: str) -> tuple[Experience, ...]: ...
+
+    def held(self, experience_id: str) -> Experience: ...


+
+    def held(self, experience_id: str) -> Experience: ...
+
+    def is_visible(self, experience: Experience) -> bool: ...


+
+    def is_visible(self, experience: Experience) -> bool: ...
+
+    def missing_files(self, refs: Iterable[FileRef]) -> tuple[str, ...]: ...


+
+    def missing_files(self, refs: Iterable[FileRef]) -> tuple[str, ...]: ...
+
+    def file_path(self, ref: FileRef) -> Path: ...


+
+    def file_path(self, ref: FileRef) -> Path: ...
+
+    def kb_state(self, schema_ref: str) -> str: ...


+
+    def kb_state(self, schema_ref: str) -> str: ...
+
+    def lock_schema(self, connection: Connection, schema_ref: str) -> None: ...


+    metrics: Metrics
+
+    @property
+    def code_digest(self) -> str: ...


+    def code_digest(self) -> str: ...
+
+    @property
+    def schema_refs(self) -> tuple[str, ...]: ...


+    @property
+    def schema_refs(self) -> tuple[str, ...]: ...
+
+    def declaration_for(self, schema_ref: str) -> ExperienceDeclaration: ...


+
+    def declaration_for(self, schema_ref: str) -> ExperienceDeclaration: ...
+
+    def visible_experiences(self, schema_ref: str) -> tuple[Experience, ...]: ...


+
+    def visible_experiences(self, schema_ref: str) -> tuple[Experience, ...]: ...
+
+    def held(self, experience_id: str) -> Experience: ...


+
+    def held(self, experience_id: str) -> Experience: ...
+
+    def is_visible(self, experience: Experience) -> bool: ...


+
+    def is_visible(self, experience: Experience) -> bool: ...
+
+    def missing_files(self, refs: Iterable[FileRef]) -> tuple[str, ...]: ...


+
+    def missing_files(self, refs: Iterable[FileRef]) -> tuple[str, ...]: ...
+
+    def file_path(self, ref: FileRef) -> Path: ...


+
+    def file_path(self, ref: FileRef) -> Path: ...
+
+    def kb_state(self, schema_ref: str) -> str: ...


+
+    def kb_state(self, schema_ref: str) -> str: ...
+
+    def lock_schema(self, connection: Connection, schema_ref: str) -> None: ...


zili-amd and others added 3 commits October 15, 2026 13:10

zili-amd requested a review from a team as a code owner October 15, 2026 20:17

zili-amd added baseline-raise Waives new, worse and baseline-growth code-metrics findings; reason required in the PR description size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description labels Oct 15, 2026

github-advanced-security Bot found potential problems Oct 15, 2026

View reviewed changes

github-code-quality Bot found potential problems Oct 15, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(experience-kb): add Distills, synced from Hyperloom-KB - #1921

feat(experience-kb): add Distills, synced from Hyperloom-KB#1921
zili-amd wants to merge 3 commits into
mainfrom
feat/experience-kb-distills

zili-amd commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026

Outside this change (informational): 93

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		def declaration_for(self, schema_ref: str) -> ExperienceDeclaration: ...

		def visible_experiences(self, schema_ref: str) -> tuple[Experience, ...]: ...


		def visible_experiences(self, schema_ref: str) -> tuple[Experience, ...]: ...

		def held(self, experience_id: str) -> Experience: ...


		def held(self, experience_id: str) -> Experience: ...

		def is_visible(self, experience: Experience) -> bool: ...


		def is_visible(self, experience: Experience) -> bool: ...

		def missing_files(self, refs: Iterable[FileRef]) -> tuple[str, ...]: ...


		def missing_files(self, refs: Iterable[FileRef]) -> tuple[str, ...]: ...

		def file_path(self, ref: FileRef) -> Path: ...


		def file_path(self, ref: FileRef) -> Path: ...

		def kb_state(self, schema_ref: str) -> str: ...


		def kb_state(self, schema_ref: str) -> str: ...

		def lock_schema(self, connection: Connection, schema_ref: str) -> None: ...

Conversation

zili-amd commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026

Code metrics gate: PASSED with waived findings (baseline-raise)

New violations (not in the baseline) (waived by the baseline-raise label): 1

A12 Single-implementation abstraction (advisory, never fails): 2

How to fix

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 1

Already over at the base (backlog, not this change's): 1

Outside this change (informational): 93

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Code metrics gate: PASSED with waived findings (`baseline-raise`)

New violations (not in the baseline) (waived by the `baseline-raise` label): 1