Skip to content

Commit 25e0cf8

Browse files
committed
Read sealed text generations through verified native postings
Verified strict Debug compilation, scoped whitespace, governance, native posting corruption tests and ten process recovery cases.
1 parent 258855d commit 25e0cf8

38 files changed

Lines changed: 735 additions & 243 deletions

File tree

‎docs/ADR/ADR-078-full-text-projection.md‎

Lines changed: 27 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,11 @@ do not copy its implementation or use its unbounded/partial-cancellation search.
1717
Canonical records, policy, exact BM25 statistics/ranks, vector/RRF accumulation,
1818
selected payload projection, synchronous WAL and acknowledgement from every configured copy remain unchanged.
1919

20-
This first provider generation is a correctness integration: canonical authorized
21-
corpus visitation remains the exact scoring and completeness oracle. Execution
22-
postings contribute bounded candidate verification. No acceleration follows from
23-
this wiring. Removing the canonical pass, global ranking, CDC incremental replay,
24-
managed ANN, async rebuild barriers and performance claims need their own accepted
25-
contracts and evidence; this stage does not close those distinct acceptance gates.
20+
Bootstrap publication visits the complete authorized corpus and verifies the
21+
persisted native postings against the tuples observed during tokenization.
22+
Searches on that sealed cut use verified postings to select canonical documents
23+
for exact scoring. Global ranking, CDC incremental replay, managed ANN and async
24+
rebuild barriers have separate contracts and qualification gates.
2625

2726
## Closed generation contract
2827

@@ -37,11 +36,11 @@ verified generation becomes current. Never move storage into an Orleans grain.
3736
```mermaid
3837
flowchart LR
3938
Request[Authorized request grain] --> Cut[Node-local committed read cut]
40-
Cut --> Canonical[Exact authorized corpus and ranks]
39+
Cut --> Canonical[Canonical authorized records]
4140
Cut --> Lease[Bounded generation lease]
42-
Canonical --> Verify[Complete candidate verification]
43-
Lease --> Verify
44-
Verify --> Result[Projected exact result]
41+
Lease --> Verify[Verify complete requested postings]
42+
Verify --> Canonical
43+
Canonical --> Result[Exact rank and projection]
4544
```
4645

4746
The generation scope binds source physical NodeId, Incarnation, current storage identity,
@@ -66,11 +65,17 @@ The Query-owned internal cross-assembly contract is `ITextProjection.Acquire(sco
6665
budget)` -> IDisposable `ITextProjectionLease`. It observes each canonical record
6766
with `BeginRecord(reference,revision)`, each enumerated token with `ObserveToken`,
6867
and validates the completed exact positive set with `VerifyCandidates(terms,
69-
references,budget)`. Cached generations verify the complete canonical identity /
70-
revision visitation and full token union; any unmapped record, missing
71-
canonical positive, source-scope mismatch or malformed generation fails closed.
72-
Known hash-collision false positives remain internal and never alter rank/output.
73-
Neither Limit nor provider paging can truncate branch ranks before RRF.
68+
references,budget)`. At seal, a single ordered native pass checks completeness
69+
and records each token's posting count and ordered digest with exact corpus
70+
totals. Sealed bootstrap reads verify every requested token, including absence,
71+
and point-read the union at the original authorized cut. The ranker derives
72+
exact term frequencies and document frequencies from that union before applying
73+
the match mode. The captured source of a text-only read retains those point
74+
records and the ranking metadata. Hybrid reads retain the vector branch's
75+
dependencies as well. Source-scope mismatches, malformed postings and
76+
inconsistent candidate revisions fail closed. Hash collisions can add candidates
77+
and exact token comparisons remove false matches. Neither Limit nor provider
78+
paging can truncate branch ranks before RRF.
7479

7580
Use the actual primary iterator with `NoRefresh`, no block-cache
7681
contribution, per-token lower-bound seeks, and a checked unique record-ID union.
@@ -113,19 +118,19 @@ even plausible filenames, never become owned through directory scanning.
113118
Complete current files remain
114119
recognized after disposal so restart can validate owners and manifests before
115120
discard/rebuild; malformed manifests fail closed and stay preserved.
116-
The completed
117-
manifest binds the exact scope and ordered identity/revision metadata, format1,
118-
tokenizer/hash versions and completed source cut. Its generated contracts
119-
use stable aliases/IDs and an SHA256 payload checksum; no internal JSON fallback.
120-
Manifest Id5 adds a bounded ordered file inventory. Each generated
121+
The completed manifest binds the exact scope, digest of the ordered canonical
122+
identity and revision map, corpus and posting proof, tokenizer and hash
123+
versions, and completed source cut. The owner retains the ordinal map in RAM
124+
while the generation is live. The generated contracts use stable aliases and
125+
IDs and a SHA256 payload checksum. The manifest contains a bounded ordered
126+
file inventory. Each generated
121127
`keyload.server.text.file` record fixes Id0 normalized relative path and
122128
Id1 byte length at publication. ZoneTree owns the content of these files and
123129
rewrites its own metadata, so KeyLoad records and checks the owned file set,
124130
never file content. Capture actual files after synchronous flush and handle
125131
settlement before complete publication; no static guessed filename allowlist
126132
is used. Restart validates the exact owned path inventory before cleanup.
127-
Searches do not re-enumerate or re-read generation files. Reopening a completed generation
128-
must preserve or atomically refresh its recognized inventory at settlement.
133+
Reader admission and settlement verify the compact manifest and owned file set.
129134
An owner receipt alone cannot authorize arbitrary recursive deletion. A completed
130135
generation requires both an exact tracked layout and a verified closed inventory.
131136
An unpublished generation without a complete manifest may be discarded only

‎docs/Features/Search/TextProviderAudit.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ The selected dependency pair is the published ZoneTree and ZoneTree.FullTextSear
1919
| Requirement | Acceptance |
2020
|---|---|
2121
| REQ-FTS-AUDIT-001: selected postings remain derived state under canonical authority | AC-FTS-AUDIT-001: reuse actual TextProtectedRebuildColdOwner/RuntimeTextProjection, real ZoneTree and original SearchEngine; literal English/Ukrainian/control documents, exact full documents/revisions/redaction/scores and update/delete receipt oracles pass before and after replacement. Existing Unicode/NFKC/supplementary/digits/short/repeated/collision/parity suites remain required. |
22-
| REQ-FTS-AUDIT-002: checksum validity does not replace manifest semantic validation | AC-FTS-AUDIT-002: each of eight genuine same-root cold lifecycles recomputes the existing envelope checksum after changing exactly one actual invariant: tokenizer, hash, format, scope position, contiguous record ID, unique reference, record partition or file digest. The real lazy owner constructor must refuse Corruption with its current safe detail, without removing/rewriting the corrupt manifest, any other file, canonical row or original history. No partial successful query is accepted. |
22+
| REQ-FTS-AUDIT-002: checksum validity does not replace manifest semantic validation | AC-FTS-AUDIT-002: each of eight genuine same-root cold lifecycles recomputes the existing envelope checksum after changing the tokenizer, hash, source incarnation, scope position, ordinal digest shape, certified document count, scope partition or owned file path. The real lazy owner constructor must refuse Corruption with its current safe detail, preserving the corrupt manifest, all other files, canonical rows and original history. |
2323
| REQ-FTS-AUDIT-003: exact repair preserves immutable canonical history | AC-FTS-AUDIT-003: restore only the exact test-owned original manifest bytes, then reconstruction returns literal updated Ukrainian/control records and absence of the deleted English record. Original full results and ordered private CompositionReferences plus first genuine outcome/locator bytes survive actual Apply/Resolve replay, a healthy new producer and two canonical/same-root cold opens. |
2424
| REQ-FTS-AUDIT-004: ownership, resource and qualification boundaries stay real | AC-FTS-AUDIT-004: canonical absolute/existing/nonempty root is checked before existing cold owner reopen; original complete nine-field identity, raw canonical bytes and commit position are retained. file inventory is bounded, ordered and byte exact; no symlink following or broad cleanup. Existing owner/observer joins every original operation and cleanup failure. Source review and prepared preflight are not compiler, process-kill, cluster, signature, endurance, performance or Linux evidence. |
2525

0 commit comments

Comments
 (0)