Skip to content

Commit db1db8f

Browse files
committed
feat: support csharp regions, html-query raw content, and java outline signatures
1 parent d23e319 commit db1db8f

20 files changed

Lines changed: 431 additions & 110 deletions

‎.github/copilot-instructions.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,11 @@ Exemptions (gate passes, read directly): file under ~200 lines (tokens <file>) a
5252

5353
Failure shapes to catch yourself in, and the command that replaces each:
5454
- not knowing which command answers a question you can already state in words → answer "<question>", which runs the right command and refuses rather than guess
55-
- a shell text search with context flags to find a function body → read "file::symbol"
56-
- paging a function/file or receiving a [tg] range denial → stop range reads; run its exact command, read "file::symbol" for a definition, read "file@start-end" for a window, skeleton file for structure, or grep "<pattern>" file -C 15 --symbol for an excerpt
55+
- a shell text search with context flags to find a function body → read "file" "symbol"
56+
- paging a function/file or receiving a [tg] range denial → stop range reads; run its exact command, read "file" "symbol" for a definition, read "file" "start-end" for a window, skeleton file for structure, or grep "<pattern>" file -C 15 --symbol for an excerpt
5757
- reading a symbol plus chasing its callers and containing doc section as separate reads → brief "file::symbol"
58-
- reading one heading of a large doc → section "file::Heading"
59-
- searching for a symbol's callers → refs file::symbol --callers
58+
- reading one heading of a large doc → section "file" "Heading"
59+
- searching for a symbol's callers → refs "file::symbol" --callers
6060
- searching across symbols, headings, text, and concepts concurrently → search "<query>"
6161
- searching for a *concept* rather than a literal string → semantic "description"
6262
- re-reading output you already captured → bash-output/web-output/mcp-output by ID

‎CLAUDE.arch.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ The table below is generated by `scripts/sync-arch-docs.mjs` from the tracked mo
4343
| `src/db_integrity.ts` | Storage and Database | Integrity probe and quarantine for the global index database | QuickCheckResult, QuarantineResult, quickCheckDb, quarantineMalformedDb, recreateDb |
4444
| `src/index_reader.ts` | Storage and Database | Read side of the symbol index | DEFAULT_QUERY_LIMIT, querySymbols, distinctSymbolKinds, countSymbols, queryRefs |
4545
| `src/section_reader.ts` | Storage and Database | Named-section extraction from text files | SectionResult, normalizeHeading, findMarkdownHeaders, extractShellBannerHeading, findShellBannerHeaders |
46+
| `src/section_regions.ts` | Storage and Database | C# and Visual Basic region directive extraction | RegionSectionHeader, findCsharpRegionHeaders, findVbRegionHeaders |
4647
| `src/stats.ts` | Storage and Database | Token-savings telemetry: record and aggregate stats in the stats table | StatsSummary, HARNESS_UNRECORDED, PRICING_VERSION_UNRECORDED, hasMixedPricingEras, SOURCE_IMAGE |
4748
| `src/stats_report.ts` | Storage and Database | Storage and Database module: stats_report.ts | _useRichStats, _buildStatsDataForTest, renderShortStats, renderStats |
4849
| `src/embed_backfill.ts` | Embeddings and Semantic Search | One-time-per-release sweep that deletes chunk rows an already-indexed file would no longer be allowed to contribute | BACKFILL_META_KEY, BackfillResult, selectUnembeddableChunkFiles, EmbedSkipMarkers, pruneUnembeddableChunks |

‎config/arch-docs.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@
4242
["src/db_integrity.ts","Storage and Database"],
4343
["src/index_reader.ts","Storage and Database"],
4444
["src/section_reader.ts","Storage and Database"],
45+
["src/section_regions.ts","Storage and Database"],
4546
["src/stats.ts","Storage and Database"],
4647
["src/stats_report.ts","Storage and Database"],
4748
["src/embed_backfill.ts","Embeddings and Semantic Search"],

‎docs/cli.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
4747
| `token-goat xml-outline <file>` | Structural summary of an XML document (element tag hierarchy, attribute keys, child counts) instead of a raw Read. Supports `--depth <n>` (alias for `--max-depth`) and `--json`. |
4848
| `token-goat xml-query <file> [path]` | Extract one value, element text/XML, or a projected/filtered subset from an XML document by dot-path or `--xpath <expr>` instead of a raw Read (dot-path grammar matches `json-query`: element tags, `@attr`, `[n]`, `[*]`, `[attr=value]`). Supports `--xpath <expression>` with namespace and attribute predicate support, `--with-lines` to output exact source line ranges (`(lines start-end)`), `--decode-embedded-xml` to pretty-print and bound entity-encoded nested AML/XML payloads, `--head <n>`, and `--json`. |
4949
| `token-goat html-outline <file>` | Structural outline of an HTML document (DOM hierarchy, tags, IDs, classes, element counts, depth, landmarks, tables, forms) instead of reading thousands of lines of markup. Supports `--json`. |
50-
| `token-goat html-query <file> <selector>` | Extract matching HTML elements or text using standard CSS selectors (tags, `#id`, `.class`, attribute operators `[attr]`, `[attr=val]`, `[attr*=val]`, `[attr^=val]`, `[attr$=val]`, child `>` and descendant combinators) without a headless browser or heavy DOM dependencies. Supports `--text` to strip tags, `--head <n>`, and `--json`. |
50+
| `token-goat html-query <file> <selector>` | Extract matching HTML elements or text using standard CSS selectors (tags, `#id`, `.class`, attribute operators `[attr]`, `[attr=val]`, `[attr*=val]`, `[attr^=val]`, `[attr$=val]`, child `>` and descendant combinators) without a headless browser or heavy DOM dependencies. Supports `--text` to extract visible inner text, `--content` to emit raw source from matching `<script>` or `<style>` elements (e.g. `html-query page.html script --content`; JSON items use a `content` field), `--head <n>`, and `--json`. |
5151
| `token-goat html-lint <file>` | Fast, zero-dependency HTML5 structural linter checking for unclosed tags, void element violations, duplicate IDs, missing viewport/charset, missing alt attributes, and inline script bloat. Supports `--json`. |
5252
| `token-goat php-lint [paths...]` | Runs PHP's `-l` syntax check on PHP files in one or more directories; with no paths, uses the current directory, and does not index files. `--php <executable>` selects the runtime (default `php`). `--jobs <count>` controls parallel checks (1-8, default 4). Invalid paths, a missing runtime, or syntax errors exit 1; no PHP files exit 0. Example: `token-goat php-lint ./src ./plugins --jobs 4`. |
5353
| `token-goat json-outline <file> [--filter TEXT]` | Structural summary of a JSON document (array shape / object key types) instead of a raw Read. `--filter TEXT` keeps only the top-level keys containing the text, for a registry with hundreds of them: `json-outline registry.json --filter 11862` prints ten keys and `(10 of 93 keys contain "11862")`. |

‎scripts/parser-fingerprint.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ const DECISION_SOURCES = [
2929
path.join(ROOT, 'src', 'markdown_lines.ts'),
3030
path.join(ROOT, 'src', 'markdown_frontmatter.ts'),
3131
path.join(ROOT, 'src', 'section_reader.ts'),
32+
path.join(ROOT, 'src', 'section_regions.ts'),
3233
path.join(ROOT, 'src', 'line_matchers.ts'),
3334
path.join(ROOT, 'src', 'encoding.ts'),
3435
path.join(ROOT, 'src', 'source_text.ts'),

‎src/cli_cmd_formats.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -551,6 +551,7 @@ export function registerFormatCommands(program: Command, guard: GuardFn): void {
551551
.option('--json', 'emit the result as JSON instead of text')
552552
.option('--text', 'extract only the inner text of matching elements')
553553
.option('--attr <name>', 'extract the value of the specified attribute')
554+
.option('--content', 'extract raw source from matching script or style elements')
554555
.action(guard(cmdHtmlQuery))
555556

556557
program

‎src/cli_cmd_inspect.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -265,7 +265,7 @@ export function registerInspectCommands(program: Command, guard: GuardFn, handle
265265
.description(description)
266266
.option('-j, --json', 'output as JSON')
267267
.option('--min-lines <n>', 'only show symbols at least N lines long')
268-
.option('--grep <pattern>', 'only show symbols whose name matches this regex (literal substring if it is not valid regex)')
268+
.option('--grep <pattern>', 'match symbol names and Java declaration signatures, including annotations (literal substring if it is not valid regex)')
269269
.option('--force-refresh', 'reparse file from disk before querying (ignore stale index)')
270270
.option('--stats', 'add per-symbol reference count and doc-coverage flag')
271271
.action(

‎src/cli_structured.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ export function cmdHtmlOutline(file: string, opts: { json?: boolean }): void {
105105
export function cmdHtmlQuery(
106106
file: string,
107107
selector: string,
108-
opts: { head?: string; json?: boolean; text?: boolean; attr?: string },
108+
opts: { head?: string; json?: boolean; text?: boolean; attr?: string; content?: boolean },
109109
): void {
110110
process.exitCode = runHtmlQuery({ file, selector, ...opts })
111111
}

‎src/hints/file_type_handler.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -77,10 +77,16 @@ export function handleHtml(filePath: string, content: string, contentLengthHint?
7777
const length = contentLengthHint ?? content.length
7878
if (length < FILE_TYPE_THRESHOLDS.html) return { shouldBlock: false, message: '' }
7979

80+
const quotedPath = quotedArg(displaySafePath(filePath))
81+
const htmlQueryHints =
82+
`Use ${fencedCommand(`token-goat html-outline ${quotedPath}`)} for structure, or extract inline JavaScript ` +
83+
`with ${fencedCommand(`token-goat html-query ${quotedPath} script --content`)}, or CSS with ` +
84+
`${fencedCommand(`token-goat html-query ${quotedPath} style --content`)}`
85+
8086
if (previewUnavailable(content, length)) {
8187
return {
8288
shouldBlock: true,
83-
message: `Large HTML file (${formatBytes(length)}) — too large to preview (exceeds the in-hook scan cap). Use token-goat section to extract a section by heading, or convert to text: pandoc ${quotedArg(displaySafePath(filePath))} -t plain`,
89+
message: `Large HTML file (${formatBytes(length)}) — too large to preview (exceeds the in-hook scan cap). ${htmlQueryHints}, or use token-goat section to extract a section by heading, or convert to text: pandoc ${quotedPath} -t plain`,
8490
}
8591
}
8692

@@ -99,7 +105,7 @@ export function handleHtml(filePath: string, content: string, contentLengthHint?
99105
if (isMinified) {
100106
return {
101107
shouldBlock: true,
102-
message: `HTML file appears minified (${formatBytes(length)}). Consider fetching the source or converting with: pandoc ${quotedArg(displaySafePath(filePath))} -t plain`,
108+
message: `HTML file appears minified (${formatBytes(length)}). ${htmlQueryHints}. Consider fetching the source or converting with: pandoc ${quotedPath} -t plain`,
103109
}
104110
}
105111

@@ -115,7 +121,7 @@ export function handleHtml(filePath: string, content: string, contentLengthHint?
115121
.join('\n'),
116122
)
117123
: '',
118-
`Use token-goat section to extract a section by heading, or convert to text: pandoc ${quotedArg(displaySafePath(filePath))} -t plain`,
124+
`${htmlQueryHints}, or use token-goat section to extract a section by heading, or convert to text: pandoc ${quotedPath} -t plain`,
119125
].filter(Boolean).join('\n'),
120126
}
121127
}

‎src/parser_fingerprint.ts‎

Lines changed: 53 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,60 +1,60 @@
11
// GENERATED FILE -- do not edit by hand. Run `npm run parser:fingerprint` to regenerate.
22
//
33
// A digest of the shared extraction-decision sources returned by sharedExtractionSources() in scripts/parser-fingerprint.mjs -- the driver and every module that decides extraction for all languages at once. It is the stamp files.parser_sha carries for a file whose language has no adapter module of its own (the tree-sitter languages, the structured-document formats parser.ts extracts inline, and 'unknown'), and the fallback for any language missing from LANGUAGE_PARSER_FINGERPRINTS below. The freshness gates treat a mismatch as changed, so an extraction-logic change invalidates already-indexed files whose content never moved. Before this existed those files kept their old symbols indefinitely, because content was the only key.
4-
export const PARSER_FINGERPRINT = '3ce75644bef57b1b'
4+
export const PARSER_FINGERPRINT = 'eabb5d79c7b275a6'
55

66
// Per-language digests, each over the shared sources above plus that language's own adapter modules under src/languages/. Keyed by the id stored in files.language, which is what the gates look a stamp up by -- see parserFingerprintForLanguage() in src/parser_stamp.ts. A fix to one adapter moves one entry here, so only that language's already-indexed files are reparsed; before this was per-language, a Dart adapter fix reparsed every file in every project, including projects holding no Dart at all.
77
export const LANGUAGE_PARSER_FINGERPRINTS: ReadonlyMap<string, string> = new Map([
8-
['abap', '168a6592ad44dec6'],
9-
['abl', 'eaec05d2dd45c05e'],
10-
['apache', '4200100137787f28'],
11-
['apex', '8929e6eefdfc6720'],
12-
['asm', '23851ccefdf8a89a'],
13-
['bash', 'e3d0c6cb265abf95'],
14-
['batch', '9760c9c7a3e3d321'],
15-
['caddy', 'c5ccefff2585bfc6'],
16-
['clojure', '001ff94540affd3a'],
17-
['cmake', 'f4d820a3bfcfe4cd'],
18-
['common_lisp', 'f05e368831751cce'],
19-
['csharp', '7b217530691451fe'],
20-
['dart', '510e149e69bddcd9'],
21-
['elixir', 'b535ee225050ca2f'],
22-
['emacs_lisp', '9e015b16bbf84800'],
23-
['erlang', 'd18a4fe01f8ae6c4'],
24-
['fortran', '0620f4278ecbbaa8'],
25-
['fsharp', '881639ea16dd7ac1'],
26-
['graphql', 'd22cc559bc2924fc'],
27-
['groovy', '1d2df3d36d3bb051'],
28-
['haskell', 'c896fa8ab5b96117'],
29-
['jcl', '63ba9e8e1bd5ea0c'],
30-
['kotlin', '54ab44a3e63b4b66'],
31-
['liquid', '6e734023add5076d'],
32-
['lua', '532e2eabba157761'],
33-
['makefile', '8cf61982f7b10f69'],
34-
['matlab', 'cba98ac7e05a3d86'],
35-
['nginx', '65e31f2b15051c82'],
36-
['nix', '030b72623f5c2ea6'],
37-
['objc', '158636e04c0167cf'],
38-
['ocaml', '7a3cdc83e6eacfd2'],
39-
['pascal', '63a58c645e6b75bd'],
40-
['perl', 'ed3558da1e976fed'],
41-
['php', '855b076045139372'],
42-
['pli', '7b2ea684701e360b'],
43-
['powershell', '6ad1dd550e24ae02'],
44-
['proto', 'd5fdc22ea35f32a7'],
45-
['r', 'a89b8d1d85a1e00c'],
46-
['racket', 'a4420e9f2b1bf2b7'],
47-
['rpg', '2e42661cae98a85e'],
48-
['salesforce_metadata', 'fbcec6e4f45d4446'],
49-
['sas', '75b4def08569c651'],
50-
['scala', '2f384ebb04d9c527'],
51-
['scheme', 'e98cbf58d7894eef'],
52-
['solidity', '8a1a319c45416505'],
53-
['sql', '6ea0134cea2e6131'],
54-
['swift', 'f91ebe97fc7eeb01'],
55-
['terraform', 'b485d5bd9e791f69'],
56-
['thrift', 'e3aa8cc3ae42b7c7'],
57-
['vb', '3682588d1848928c'],
58-
['vhdl', '56354614b6521cb7'],
59-
['zig', 'b61c4c5695bcb547'],
8+
['abap', 'eae2d536f41d515d'],
9+
['abl', '480a116b808fdde9'],
10+
['apache', '87ba8be24963103b'],
11+
['apex', 'bd8a80d31bd31e65'],
12+
['asm', 'f7619665269dbdfc'],
13+
['bash', '2cc190b7887bff34'],
14+
['batch', '09aa0a15b71e176a'],
15+
['caddy', 'ccafcb9e6a2bb891'],
16+
['clojure', 'cd64e4ca22ed219a'],
17+
['cmake', '38721a933bfad612'],
18+
['common_lisp', '6ac0d683adf143cf'],
19+
['csharp', '57b2c5a2a79dea7e'],
20+
['dart', '3d0eddcc7483051d'],
21+
['elixir', '754dcbfecb692c47'],
22+
['emacs_lisp', '5997e5e6c3f0a3f3'],
23+
['erlang', '5c5c7df657f871cd'],
24+
['fortran', 'f52a65355ae5eeca'],
25+
['fsharp', '8982f2740e12d5c1'],
26+
['graphql', '42a5c908345b57f5'],
27+
['groovy', '12df6a64a0d7a44c'],
28+
['haskell', 'c6a6093fd1eb0a16'],
29+
['jcl', '1393271e08a9080e'],
30+
['kotlin', '8c2f2c7607da0375'],
31+
['liquid', 'd08facb332483be8'],
32+
['lua', '279c1c0f176f945a'],
33+
['makefile', 'e6757eb6c0471aef'],
34+
['matlab', '8f855174e73b9258'],
35+
['nginx', 'ffe077544af835ca'],
36+
['nix', '74d060b53a9c793e'],
37+
['objc', 'bf0565b7f9f86c9a'],
38+
['ocaml', '57cc986e5e6b2dab'],
39+
['pascal', '7c73d6510c794eff'],
40+
['perl', '507ea58bd578bc70'],
41+
['php', '41f06caa8e852404'],
42+
['pli', 'bd98ccb6036625ed'],
43+
['powershell', '29315f71b3543712'],
44+
['proto', '2dbc70bf8440c171'],
45+
['r', 'f7618bdb9d430c0b'],
46+
['racket', 'c1cf9b6bcb9186c3'],
47+
['rpg', 'e6dc166125902be4'],
48+
['salesforce_metadata', '4d5aa6bb6983c616'],
49+
['sas', '615184a5aeb65e13'],
50+
['scala', 'e9c320ea5b3c3294'],
51+
['scheme', 'a51f0bb7e0ba3981'],
52+
['solidity', 'f793f07747d85f7c'],
53+
['sql', 'd3a026e0c768709d'],
54+
['swift', '0b4497922fff2fbb'],
55+
['terraform', 'd789b8650042ed65'],
56+
['thrift', '28688dd7aec15309'],
57+
['vb', 'f42e5eb8b0f7f91e'],
58+
['vhdl', 'a39f12ceddb262af'],
59+
['zig', '63f9d6e478881849'],
6060
])

0 commit comments

Comments
 (0)