manifest grammar: durable comments, ordered sets, reference recognition - #9
Open
jmpaz wants to merge 6 commits into
Open
manifest grammar: durable comments, ordered sets, reference recognition#9jmpaz wants to merge 6 commits into
jmpaz wants to merge 6 commits into
Conversation
Adds an outline pass over the authored manifest text: components, groups, and files/repos/manifests members are walked in authored order with source line spans, block/end-of-line comments, and commented-out list items recognized as disabled members. Disabled members produce no payload and are never resolved (they were already invisible to the YAML parser); what's new is that they're now surfaced in index.json instead of silently discarded. Verbatim source preservation (58e0ee3) is extended to keep disabled members and their comments in hydrated `manifest.yaml` copies rather than stripping them, since they're deliberate authored curation.
`- set: NAME` sits alongside `name:`/`group:` in component position: an ordered run of `files:` members that resolves to ONE hydrated file instead of one-per-source, addressable by name like any other component. Parts are concatenated in authored order behind a part boundary header carrying the source key, title (explicit `comment:` or the member's trailing `#` comment), and timestamps where known; the boundaries are also recorded structurally in index.json. Groups may contain sets; sets may not contain groups, repos, or manifests (fusion only makes sense over resolvable file content).
…edges `manifests:` keeps meaning full inclusion; a `files:` entry that happens to be a manifest is now recognized (registry membership, `ctx/manifest` frontmatter tag, or a parseable fenced/whole-document manifest, checked cheapest-first since registry membership can shell out to zk) and recorded as a followable edge without changing its payload -- the authored note still hydrates as an ordinary file copy. Both forms land in index.json under `references.out`, each entry stating its form, detection tier, and whether the payload is a live pointer (full inclusion, symlinked to the linked context) or a resolved copy (recognized reference, the note's own file content). Relation naming and the inbound half of the edge (which needs a registry-wide join across multiple contexts' indexes) are left to the serving-surface round; each index already carries what that join needs (source_path/source_ref per member, target_path per edge).
YAML permits a list to sit at the same indentation as its parent key (`key:\n- item`), a style used throughout the real note corpus. The block-end scan for `components:`/`files:`/`repos:`/`manifests:` keys was stopping on the first such item instead of the next sibling key, producing an empty outline (and, for the same reason, an empty group slice) for any manifest written this way. Item-boundary scans (finding where one component/member ends and its sibling begins) are unaffected and still terminate on a same-indent dash -- only key-block-end scans (finding where a key's value block ends) now tolerate a same-indent dash as a continuation. Verified against the real note corpus: outline/normalize_components component counts matched on all 95 parseable manifests (0 mismatches, down from 53 before this fix).
A component with no name/group/set key (`- files:\n - a.md`) puts the `files:` key directly on the item's dash line rather than on its own line; the member-list scan only looked from the line after the dash onward and silently found no members for such components.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A manifest's authored structure — order, comments, exclusions, nesting — is signal, and until now the bounce destroyed it: hydration re-serialized the YAML, so every comment died and every deliberate exclusion vanished.
Four grammar moves, all recognition of what authors already write (no existing manifest becomes invalid):
manifest.yamlcopies are now verbatim slices of the authored source.# - notes/x.mdparses as a member with state off: produces no payload, never resolves, but appears in the index (with its raw text) as deliberate exclusion — curation you can see.- set: NAMEin component position is an ordered run whose members concatenate into one unit with marked part boundaries (source key, title, timestamps). Hydrates as a single file, addressable like any component.files:(detected via frontmatter tag, fenced-block parse, or registry membership) includes the authored note as its payload and records a followable edge — bidirectional data inindex.json, with the live-pointer vs resolved-copy distinction always explicit.index.jsongrows anoutline(authored order, line spans, comments, disabled members, set parts) andreferences— everything a serving surface needs to render a manifest as authored.Verification: 227 tests passing (+14 this branch; the 2 failures are pre-existing environment-dependent plugin-auth tests, identical on main). The outline parser was swept against all 99 real manifests in the working corpus, which surfaced and fixed two YAML-shape bugs (same-indent block sequences; bare components with keys on the dash line). A live scratch hydration exercised all four moves end-to-end.
Follow-up (next round, deliberately not here): the serving surface — one query core rendered as CLI verbs and MCP tools per
.plans/2026-07-08-manifest-grammar-and-serving-surface.md.