feat(chunking): semantic chunk type labeling (#600)#623
Merged
Conversation
kh3rld
force-pushed
the
feat/semantic-chunk-labeling
branch
from
330373a to
ea4d5e8
Compare
Goldziher
requested changes
Apr 1, 2026
Goldziher
left a comment
Goldziher
left a comment
Member
There was a problem hiding this comment.
This PR is partial - missing multiple bindings. Missing documentation.
Must pass for linting for all languages.
Must have full e2e tests for all languages.
Must have updated context tests.
kh3rld
added a commit
that referenced
this pull request
Apr 2, 2026
… and e2e coverage - Add chunk_type field to all language package bindings: C# (ChunkType property), Elixir (chunk_type field), Go (ChunkType string), Java (chunkType + getChunkType()), TypeScript (chunkType + ChunkType type) - Add ChunkType documentation type to TypeScript core types - Add each_has_chunk_type assertion to all e2e helpers: TypeScript, wasm-deno, wasm-workers, Java, Ruby, R, PHP, C# (was done), Elixir (was done), Go (was done), Python (was done) - Update generator source files (wasm_deno.rs, wasm_workers.rs, r/helpers.rs, python.rs) to emit chunk_type in tests and helpers - Sync all generator snapshot templates (tools/e2e-generator/e2e/) to match the embedded generator templates - Add fixture schema entries for each_has_chunk_type and content_starts_with_heading - Add test_chunk_type_populated context test to config_features.rs - Remove wrongly-placed e2e/e2e/php/ duplicate files (were generated from wrong working directory) closes reviewer feedback on #623
Extend Chunk with a chunk_type field classifying each chunk's structural role. ChunkType has 13 variants: heading, party_list, definitions, operative_clause, signature_block, schedule, table_like, formula, code_block, image, org_chart, diagram, unknown. Field is serde(default) so existing JSON without it deserializes cleanly.
Add classifier.rs with 9 ordered rules (heading, code_block, table_like, formula, schedule, definitions, signature_block, operative_clause, party_list) covering all proposed ChunkType variants. First match wins; falls back to Unknown. Includes 20 unit tests. Wire classify_chunk() into build_chunks() so every produced chunk gets a semantic label automatically.
… API Update FFI (C/C#/PHP/Ruby/R), Python, Node.js, PHP type definitions, CLI embed command, HTTP API handlers, MCP server, and Ruby gem result type to include the new chunk_type field. Defaults to unknown where not explicitly classified.
Regenerated e2e tests across C, C#, Elixir, Go, Java, PHP, Python, R, Ruby, Rust, TypeScript (HTTP/WASM/Deno/Workers) to assert the chunk_type field is present in contract, embeddings, and structured test responses.
… bugs - Pass original content (pre-trim) to is_code_block so leading 4-space indentation is not stripped from the first line before the check - Remove over-broad heading rule that classified short body text as a Heading when it appeared under a single-level heading context Both bugs were caught by the classifier's own unit tests.
… and e2e coverage - Add chunk_type field to all language package bindings: C# (ChunkType property), Elixir (chunk_type field), Go (ChunkType string), Java (chunkType + getChunkType()), TypeScript (chunkType + ChunkType type) - Add ChunkType documentation type to TypeScript core types - Add each_has_chunk_type assertion to all e2e helpers: TypeScript, wasm-deno, wasm-workers, Java, Ruby, R, PHP, C# (was done), Elixir (was done), Go (was done), Python (was done) - Update generator source files (wasm_deno.rs, wasm_workers.rs, r/helpers.rs, python.rs) to emit chunk_type in tests and helpers - Sync all generator snapshot templates (tools/e2e-generator/e2e/) to match the embedded generator templates - Add fixture schema entries for each_has_chunk_type and content_starts_with_heading - Add test_chunk_type_populated context test to config_features.rs - Remove wrongly-placed e2e/e2e/php/ duplicate files (were generated from wrong working directory) closes reviewer feedback on #623
- Add missing chunk_type field in benchmark and manifest Chunk literals - Fix TypeScript double-cast for Chunk → PlainRecord - Reformat Elixir defstruct to pass mix format --check-formatted - Refactor assert_chunks helpers in Python e2e snapshot (complexity lint) - Fix trailing whitespace and missing EOF newlines in generated e2e files
kh3rld
force-pushed
the
feat/semantic-chunk-labeling
branch
from
720ffcb to
f66e7ee
Compare
Goldziher
pushed a commit
that referenced
this pull request
Apr 2, 2026
* feat(types): add ChunkType enum and chunk_type field to Chunk (#600) Extend Chunk with a chunk_type field classifying each chunk's structural role. ChunkType has 13 variants: heading, party_list, definitions, operative_clause, signature_block, schedule, table_like, formula, code_block, image, org_chart, diagram, unknown. Field is serde(default) so existing JSON without it deserializes cleanly. * feat(chunking): add heuristic classifier and wire into chunk builder Add classifier.rs with 9 ordered rules (heading, code_block, table_like, formula, schedule, definitions, signature_block, operative_clause, party_list) covering all proposed ChunkType variants. First match wins; falls back to Unknown. Includes 20 unit tests. Wire classify_chunk() into build_chunks() so every produced chunk gets a semantic label automatically. * feat(bindings): propagate chunk_type across all language bindings and API Update FFI (C/C#/PHP/Ruby/R), Python, Node.js, PHP type definitions, CLI embed command, HTTP API handlers, MCP server, and Ruby gem result type to include the new chunk_type field. Defaults to unknown where not explicitly classified. * feat(e2e-generator): emit chunk_type field in generated test fixtures * test(e2e): update all language e2e tests to include chunk_type field Regenerated e2e tests across C, C#, Elixir, Go, Java, PHP, Python, R, Ruby, Rust, TypeScript (HTTP/WASM/Deno/Workers) to assert the chunk_type field is present in contract, embeddings, and structured test responses. * docs(changelog): add entry for semantic chunk labeling (#600) * fix(chunking): fix indented code-block and heading-context classifier bugs - Pass original content (pre-trim) to is_code_block so leading 4-space indentation is not stripped from the first line before the check - Remove over-broad heading rule that classified short body text as a Heading when it appeared under a single-level heading context Both bugs were caught by the classifier's own unit tests. * fix: resolve the failing doctest (wrong example text for OperativeClause) * feat: create chunk_type: Default::default() field (needed for Chunk struct) * fix(chunking): address review feedback - complete chunk_type bindings and e2e coverage - Add chunk_type field to all language package bindings: C# (ChunkType property), Elixir (chunk_type field), Go (ChunkType string), Java (chunkType + getChunkType()), TypeScript (chunkType + ChunkType type) - Add ChunkType documentation type to TypeScript core types - Add each_has_chunk_type assertion to all e2e helpers: TypeScript, wasm-deno, wasm-workers, Java, Ruby, R, PHP, C# (was done), Elixir (was done), Go (was done), Python (was done) - Update generator source files (wasm_deno.rs, wasm_workers.rs, r/helpers.rs, python.rs) to emit chunk_type in tests and helpers - Sync all generator snapshot templates (tools/e2e-generator/e2e/) to match the embedded generator templates - Add fixture schema entries for each_has_chunk_type and content_starts_with_heading - Add test_chunk_type_populated context test to config_features.rs - Remove wrongly-placed e2e/e2e/php/ duplicate files (were generated from wrong working directory) closes reviewer feedback on #623 * fix(ci): resolve chunk_type CI failures - Add missing chunk_type field in benchmark and manifest Chunk literals - Fix TypeScript double-cast for Chunk → PlainRecord - Reformat Elixir defstruct to pass mix format --check-formatted - Refactor assert_chunks helpers in Python e2e snapshot (complexity lint) - Fix trailing whitespace and missing EOF newlines in generated e2e files * style: apply biome formatting to TypeScript e2e files
Goldziher
pushed a commit
that referenced
this pull request
May 17, 2026
* feat(types): add ChunkType enum and chunk_type field to Chunk (#600) Extend Chunk with a chunk_type field classifying each chunk's structural role. ChunkType has 13 variants: heading, party_list, definitions, operative_clause, signature_block, schedule, table_like, formula, code_block, image, org_chart, diagram, unknown. Field is serde(default) so existing JSON without it deserializes cleanly. * feat(chunking): add heuristic classifier and wire into chunk builder Add classifier.rs with 9 ordered rules (heading, code_block, table_like, formula, schedule, definitions, signature_block, operative_clause, party_list) covering all proposed ChunkType variants. First match wins; falls back to Unknown. Includes 20 unit tests. Wire classify_chunk() into build_chunks() so every produced chunk gets a semantic label automatically. * feat(bindings): propagate chunk_type across all language bindings and API Update FFI (C/C#/PHP/Ruby/R), Python, Node.js, PHP type definitions, CLI embed command, HTTP API handlers, MCP server, and Ruby gem result type to include the new chunk_type field. Defaults to unknown where not explicitly classified. * feat(e2e-generator): emit chunk_type field in generated test fixtures * test(e2e): update all language e2e tests to include chunk_type field Regenerated e2e tests across C, C#, Elixir, Go, Java, PHP, Python, R, Ruby, Rust, TypeScript (HTTP/WASM/Deno/Workers) to assert the chunk_type field is present in contract, embeddings, and structured test responses. * docs(changelog): add entry for semantic chunk labeling (#600) * fix(chunking): fix indented code-block and heading-context classifier bugs - Pass original content (pre-trim) to is_code_block so leading 4-space indentation is not stripped from the first line before the check - Remove over-broad heading rule that classified short body text as a Heading when it appeared under a single-level heading context Both bugs were caught by the classifier's own unit tests. * fix: resolve the failing doctest (wrong example text for OperativeClause) * feat: create chunk_type: Default::default() field (needed for Chunk struct) * fix(chunking): address review feedback - complete chunk_type bindings and e2e coverage - Add chunk_type field to all language package bindings: C# (ChunkType property), Elixir (chunk_type field), Go (ChunkType string), Java (chunkType + getChunkType()), TypeScript (chunkType + ChunkType type) - Add ChunkType documentation type to TypeScript core types - Add each_has_chunk_type assertion to all e2e helpers: TypeScript, wasm-deno, wasm-workers, Java, Ruby, R, PHP, C# (was done), Elixir (was done), Go (was done), Python (was done) - Update generator source files (wasm_deno.rs, wasm_workers.rs, r/helpers.rs, python.rs) to emit chunk_type in tests and helpers - Sync all generator snapshot templates (tools/e2e-generator/e2e/) to match the embedded generator templates - Add fixture schema entries for each_has_chunk_type and content_starts_with_heading - Add test_chunk_type_populated context test to config_features.rs - Remove wrongly-placed e2e/e2e/php/ duplicate files (were generated from wrong working directory) closes reviewer feedback on #623 * fix(ci): resolve chunk_type CI failures - Add missing chunk_type field in benchmark and manifest Chunk literals - Fix TypeScript double-cast for Chunk → PlainRecord - Reformat Elixir defstruct to pass mix format --check-formatted - Refactor assert_chunks helpers in Python e2e snapshot (complexity lint) - Fix trailing whitespace and missing EOF newlines in generated e2e files * style: apply biome formatting to TypeScript e2e files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
create a
chunk_typefield toChunkthat classifies each chunk into a semantic category (Heading, CodeBlock, OperativeClause, Definitions, SignatureBlock, Schedule, TableLike, Formula, PartyList, Unknown) using a deterministic, zero-latency heuristic classifier.changes
types/extraction.rs: AddChunkTypeenum andchunk_typefield toChunk,HeadingContext/HeadingLeveltypeschunking/classifier.rs: Heuristic rule engine — ordered by precision, first match wins; includesheading_contextsignal from Markdown chunkerchunking/builder.rs: Wire classifier into chunk constructionchunk_typefieldchunk_typein all output pathschunk_typein fixturesembed_benchmark.rs: addchunk_type: Default::default()toChunkconstructionnotes
chunk_typedefaults toUnknownand is#[serde(default)]ChunkTypevariants and heading-context pathcloses #600