Skip to content

Commit 73da20d

Browse files
committed
merge: resolve conflicts with origin/main (bash output compression)
Three textual conflicts plus one semantic one between this branch's feature work and the bash-output-compression feature that landed on ``main`` while this PR was open. Resolutions: * **install.py** PreToolUse matcher: ``main`` widened the matcher to ``Read|Bash`` so the bash-compress wrapper can intercept noisy commands; this branch widened it to ``Read|Grep|Bash`` for the Grep dedup hint. Final value: ``Read|Grep|Bash`` with a comment explaining both reasons. * **CHANGELOG.md**: kept both ``[Unreleased]`` blocks — compression first, then this branch's eleven surfaces, then the shared ``Changed`` / ``Fixed`` sections. * **README.md** "What changes" table: kept both sets of rows (compression filters first, then the dedup / recovery / auto-redirect / structured-config rows from this branch). * **hooks_read.py / cli.py**: auto-merged. The resulting ``pre_read`` dispatch is bash-dedup → bash-read-equivalent → bash-compress → CONTINUE for Bash, grep-dedup → CONTINUE for Grep, and the existing Read / Glob branches untouched. This ordering is intentional: a session-cache hit is the cheapest signal and short-circuits before we ever look at the command shape. * **test_bash_dedup_hint.py**: the negative-assertion cases used ``pytest -v src/`` and ``make build`` which are now compressible commands — the bash-compress wrapper fires before pre_read's fall-through, so the ``hookSpecificOutput not in result`` assertion broke. Switched to ``find`` / ``echo`` / ``head`` which are deliberately not in main's filter list, so the only reason for ``pre_read`` to emit a hookSpecificOutput is a real dedup hit. Comment in the test module docstring records the invariant. 636 tests pass; ruff clean; mypy adds zero new errors over baseline.
2 parents 3835b77 + 1e96258 commit 73da20d

15 files changed

Lines changed: 3792 additions & 40 deletions

‎CHANGELOG.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,30 +6,34 @@ All notable changes to Token-Goat are documented in this file. Format follows Ke
66

77
### Added
88

9+
- **Bash output compression.** PreToolUse hook on Bash detects compressible commands and rewrites them to flow through `token-goat compress`, which runs the original through the system shell, captures stdout + stderr, applies a per-tool filter, and prints a compressed view that surfaces failures first. Twelve filters cover the noisiest dev commands: `pytest`, `jest` / `vitest`, `cargo`, `npm` / `pnpm` / `yarn` / `bun`, `docker` / `buildah` / `podman`, `kubectl` / `helm`, `aws`, `ruff` / `eslint` / `mypy` / `pyright` / `pylint` / `stylelint` / `biome` / `tsc`, `git`, `make` / `ninja` / `gradle` / `mvn` / `bazel` / `go`, `terraform` / `tofu`, `pip` / `pipx`. Typical savings: pytest 80-97%, npm 88%, docker 75%, linters 80%. Each filter strips ANSI, collapses `\r` progress bars, dedupes consecutive lines, groups linter issues by rule (3 examples per code), keeps every error and warning block verbatim, and caps total output at 1000 lines / 64 KiB. The wrapper preserves the original exit code, kills the process group on timeout (SIGTERM then SIGKILL after a grace period on POSIX), and caps each stream capture at 32 MiB. Configurable via `[bash_compress]` in config.toml (`enabled`, `disabled_filters`, `max_lines`, `max_bytes`, `timeout_seconds`) or disabled with `TOKEN_GOAT_BASH_COMPRESS=0`. Savings are recorded per filter as `bash_compress:<name>`. New CLI subcommand `token-goat compress` for previewing compression on any command.
910
- **Post-compaction recovery hint.** ``SessionStart`` now detects ``source == "compact"`` and emits a one-shot ``additionalContext`` block listing the most recently-read files, cached Bash outputs (``token-goat bash-output <id>``), and cached WebFetch responses (``token-goat web-output <id>``) from the *pre*-compaction session. The cache is intentionally preserved across the compact so the recovery hint has data to draw from; the cache reset still fires on every other source value (startup / resume / clear / unknown). When the prior session was empty, no hint is emitted — the recovery path is silent until it has something worth surfacing.
1011
- **Grep dedup hint.** A repeat ``Grep`` invocation with the same ``(pattern, path)`` pair within the staleness window now produces a ``"this ran ~Ns ago and matched N lines"`` advisory. Same mechanism as the bash and web dedup hints but pointed at the existing ``session.greps`` history — no new disk store is involved. Suppressed when the prior result was below 50 matches (the hint preamble would approach the saving).
1112
- **WebFetch result cache.** A new ``PostToolUse(WebFetch)`` hook persists non-image response bodies to ``data_dir() / "web_outputs"`` and records the ``(url_sha → output_id)`` mapping in the session cache. On a repeat fetch of the same URL the pre-fetch hook emits a dedup hint pointing at ``token-goat web-output <id>``, mirroring the bash-cache pattern. Two new CLI commands surface the cache: ``token-goat web-output`` (with the same ``--head`` / ``--tail`` / ``--grep`` slicers as ``bash-output``, plus ``numbered_lines`` in JSON mode) and ``token-goat web-history``. Disk store is byte-capped (32 MB default) with oldest-first eviction + paired sidecar cleanup.
1213
- **Dockerfile section extractor.** ``Dockerfile``, ``Containerfile``, and ``*.dockerfile`` now produce one ``Section`` per ``FROM`` build stage, so ``token-goat section Dockerfile::builder`` extracts a single stage instead of forcing a full-file read. Multi-stage builds resolve by ``AS <name>`` alias when present; unnamed stages fall back to the image reference so they remain addressable.
13-
- **Pre-Grep matcher + pre-Bash matcher in install.** ``PreToolUse`` now fires on ``Read|Grep|Bash`` (was ``Read``-only) so the new Grep dedup hint actually runs. Pre-existing Bash dedup hint logic in ``hooks_read`` now reaches the wire too — under the old matcher it only fired under Codex where the matcher was already permissive.
14+
- **Pre-Grep matcher + pre-Bash matcher in install.** ``PreToolUse`` now fires on ``Read|Grep|Bash`` (matcher widened from the prior ``Read|Bash``) so the new Grep dedup hint actually runs alongside the Bash compression rewriter from the prior entry.
1415
- **``token-goat doctor`` cache visibility.** A new ``Caches`` section reports the size, file count, and oldest-entry age for ``bash_outputs/``, ``web_outputs/``, and ``session_snapshots/``. Each row warns when the directory has grown more than 10% over its byte cap, surfacing potential eviction gaps without needing to grep the data directory by hand.
1516
- **Close-match auto-redirect on ``token-goat symbol``.** When a symbol query returns zero results and the project has exactly one close-match candidate at high confidence (difflib ratio ≥ 0.85), the lookup is automatically re-run against that candidate. The redirected response carries a ``redirected_from`` field in JSON output and a ``(redirected from: …)`` marker in plain-text output so the substitution is auditable. Pass ``--strict`` to disable the redirect and get the previous "Did you mean: …?" suggestion list behaviour.
1617
- **``bash`` and ``web`` source buckets in stats.** ``token-goat stats`` now attributes ``bash_*`` kinds to a visible ``bash`` bucket (orange in the fancy renderer) and ``web_*`` kinds to a new ``web`` bucket (yellow), so the new mechanisms get first-class lines in the by-source panel instead of falling into the ``other`` catch-all. ``grep_dedup_hint`` lands in the existing ``hint`` bucket because it prevents a Read-equivalent burst (consistent with ``diff_hint``).
17-
1818
- **Bash output interception.** A new `PostToolUse(Bash)` hook persists large stdout/stderr to disk under `data_dir() / "bash_outputs"` and records the command in the session cache. When the same command is about to run again in the same session, the pre-Bash hint suggests `token-goat bash-output <id>` (optionally with `--head N`, `--tail N`, or `--grep PATTERN`) instead of re-executing — avoiding both runtime cost and duplicated tokens. The store is byte-capped (16 MB default) with oldest-first eviction; outputs above 2 MB are tail-preserved with a truncation marker. Two new CLI commands surface the cache: `token-goat bash-output` retrieves a sliced view, `token-goat bash-history` lists cached entries newest-first.
1919
- **Diff-aware re-read.** `post_read` now writes a per-session content snapshot (under `data_dir() / "session_snapshots"`, capped at 256 KB per file and 150 snapshots per session) so a follow-up `Read` after a `Write`/`Edit`/`MultiEdit` can be answered with a unified diff hint instead of a `pre_read` blocking message that silently allowed the full re-read. The diff is bounded to 4 KB and only fires when the realised saving exceeds ~250 tokens; below that the existing session-cache hint path runs unchanged. Stats record both the realised saving (`diff_hint`) and the hint's injection cost (`diff_hint_overhead`) for honest accounting.
2020
- **TOML, YAML, JSON, INI, CFG, and dotenv section extraction.** `token-goat section pyproject.toml::tool.ruff` (and equivalents for `.yaml`, `.yml`, `.json`, `.ini`, `.cfg`, `.env`, and `.envrc`) now extract a single table/key block instead of forcing a full-file read. The TOML scanner emits one `Section` per `[table]` and `[[array]]` header; the YAML scanner emits top-level keys plus one nested layer (`spec.replicas`-style) computed from the file's detected indent; JSON gains depth-1 section detection on pretty-printed files; INI/CFG indexes one section per `[name]` header; `.env`/`.envrc` index each `KEY=value` assignment as a symbol. None of the six pulls in an extra dependency — all use line-scanners and the existing stdlib parsers. The parser dispatcher gained a basename-keyed table (alongside the existing suffix table) so dotfiles with empty extensions (`.env`, `.envrc`) resolve correctly.
2121
- **Stale-data sweeps in the background worker.** `cleanup_on_startup` now also drops snapshot directories older than 24 hours and enforces the bash-output byte cap, so a long-lived install does not accumulate per-session debris.
2222
- **Compaction manifest gained a "Commands Run" section.** The PreCompact manifest now surfaces the most recent meaningful Bash invocations (cmd preview, exit code, byte size, cache ID) so the test/build context that drives the next agent turn survives compaction. Each entry includes the `token-goat bash-output <id>` cache key for surgical recall. `event_count` includes `bash_history` so a session whose only activity is a cached test run still clears the `min_events` threshold.
2323
- **`token-goat bash-output --json` now surfaces line numbers.** The JSON shape adds `numbered_lines` (a 1-based, original-body-anchored `[{lineno, text}]` list) and `total_lines`, mirroring the surgical-read response shape elsewhere in the codebase. Agents can now `--head` / `--tail` / `--grep` filter and still map back to positions in the original output.
2424
- **Hardened PostToolUse Bash payload extraction.** `_extract_bash_response` now tolerates every documented Bash result shape: dict-with-named-fields (Claude Code), MCP `CallToolResult` content arrays, bare-string blobs, top-level flattening (no `tool_response` wrapper), `tool_result`/`response` aliases, `returncode` and string-typed `exit_code` variants. Each shape is covered by a dedicated regression test in `test_post_bash_payloads.py`.
25-
- **New stats bucket `bash` (orange) and diff-hint accounting.** `token-goat stats` now attributes `diff_hint` / `diff_hint_overhead` to the `hint` bucket and `bash_dedup_hint*` / `bash_output_cached` to a new visible `bash` bucket, so the new mechanisms appear in the by-source panel instead of falling into the `other` catch-all.
2625

2726
### Changed
2827

2928
- **`reset_session`** now also removes per-session content snapshots, matching the existing JSON-cache reset semantics.
3029
- **Codex Bash matcher in `~/.codex/config.toml`** now points at the new `post-bash` hook instead of `post-read`; under Codex, `post-read` previously did nothing for `Bash` calls (no branch in the handler), so this is a strict gain.
3130
- **`bash_cache.evict_old_entries`** removes body + sidecar pairs together, and runs a second pass to sweep any orphan sidecars left over from out-of-band deletion. Previously, manual `rm` of a body file or a write race could leave a `.json` sidecar with no matching body that lived forever.
3231

32+
### Fixed
33+
34+
- **`paths.open_log_file` returned a `StreamHandler` instead of a `FileHandler` on POSIX.** The type hint and docstring claimed `FileHandler`, but the implementation wrapped `os.fdopen()` in a bare `StreamHandler` to apply 0o600 permissions, breaking `isinstance(handler, FileHandler)` checks (such as the `test_setup_logging_skips_console_handler_when_not_tty` worker test). Replaced with a private `FileHandler` subclass that overrides `_open` to apply the tighter mode at open time, preserving the type identity callers depend on.
35+
- **`test_canonicalize_drive_case_collapsed` and `test_canonicalize_cross_shell_paths_produce_same_hash` failed on POSIX.** Both assert Windows-shell drive-letter normalisation invariants that only fire when `Path.resolve()` returns an absolute Windows path; on POSIX `Path("C:/Projects/foo").resolve()` becomes `cwd + "/C:/Projects/foo"` and the assertions test against synthesised POSIX paths. Now skipped on non-Windows with an explanatory message.
36+
3337
## [0.5.2] - 2026-05-17
3438

3539
### Fixed

‎README.md‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,20 +36,24 @@
3636

3737
Claude reads `auth.py`. Then reads it again. Then a third time after compaction wipes the session. You pay for every token.
3838

39-
Long sessions accumulate waste three ways. Screenshots cross the model at full resolution. A single PNG can land at 3.3 MB. The agent re-reads files it already parsed earlier in the same conversation. And when a session compacts, the summary LLM doesn't know which files were edited or which symbols mattered, so it preserves the wrong things.
39+
Long sessions accumulate waste four ways. Screenshots cross the model at full resolution. A single PNG can land at 3.3 MB. The agent re-reads files it already parsed earlier in the same conversation. When a session compacts, the summary LLM doesn't know which files were edited or which symbols mattered, so it preserves the wrong things. And every `pytest`, `npm install`, `docker build`, or `git log` dumps thousands of lines of progress bars, deprecation warnings, and passing-test names that bury the one line that actually matters.
4040

41-
Each one is preventable. Token-Goat intercepts all three, automatically.
41+
Each one is preventable. Token-Goat intercepts all four, automatically.
4242

4343
## What changes
4444

4545
| Without Token-Goat | With Token-Goat |
4646
|--------------------|-----------------|
47-
| 3.3 MB screenshot lands in model context | 84 KB compressed copy — 97.4% smaller |
47+
| 3.3 MB screenshot lands in model context | 84 KB compressed copy, 97.4% smaller |
4848
| Agent re-reads files from earlier in the session | "Already read this" reminder with narrow slice suggestion |
4949
| Agent re-reads a file edited mid-session | Unified diff injected as a hint — full Read avoided when the diff covers the change |
5050
| Compaction forgets which files were edited | Structured session manifest injected before compact |
5151
| Same files re-read from scratch after `/compact` | Recovery hint at SessionStart lists cached snapshot + bash + WebFetch IDs |
52-
| Full file read for one function or section | `token-goat read file::symbol` — about 85% smaller |
52+
| Full file read for one function or section | `token-goat read file::symbol`, about 85% smaller |
53+
| `pytest` dumps 150 PASSED lines + dots + tracebacks | Failures-first view, 80 to 97% smaller |
54+
| `npm install` floods deprecation warnings + spinner | Errors kept; warnings collapsed by package, ~90% smaller |
55+
| `docker build` emits sha256 digests + transfer progress | Step headers + errors kept; noise dropped, ~75% smaller |
56+
| `ruff` / `eslint` / `mypy` repeat the same rule 50 times | Grouped by rule with first 3 examples, ~80% smaller |
5357
| Same `pytest` / `cargo` / `git log` re-run mid-session | Pre-Bash dedup hint points at the cached output (`token-goat bash-output <id>`) |
5458
| Same `Grep` pattern re-run with hundreds of matches | Pre-Grep dedup hint quotes the prior match count |
5559
| Same docs URL fetched twice | Pre-WebFetch dedup hint points at the cached body (`token-goat web-output <id>`) |
@@ -140,6 +144,28 @@ out: ~4 KB # top-ranked files + key symbols (92% smaller)
140144

141145
`--budget` is a hard cap. Below 6 KB the output automatically switches to short-label mode (`f:` files, `s:` symbols, `c:` calls) to fit more signal per byte. `token-goat map --compact` is a shortcut for a 300-token budget when you only need the high-rank cluster.
142146

147+
**5. Bash output compression**
148+
149+
```
150+
# Without token-goat: pytest dumps every PASSED line + dots + tracebacks.
151+
$ pytest -v tests/
152+
... (3 KB of output, 150 PASSED lines, 1 FAILED at the bottom)
153+
154+
# With token-goat: the PreToolUse hook rewrites the command to
155+
# `token-goat compress --filter pytest`. The wrapper runs pytest, captures
156+
# stdout+stderr, applies the per-tool filter, and prints failures first.
157+
$ token-goat compress --filter pytest --cmd "pytest -v tests/"
158+
= test session starts =
159+
collected 150 items
160+
FAILED tests/test_x.py::test_one
161+
= 1 failed, 149 passed in 2.3s =
162+
163+
[token-goat: collapsed 149 PASSED lines]
164+
[token-goat: pytest filter compressed 4.8 KiB to 0.1 KiB (97% saved)]
165+
```
166+
167+
Twelve built-in filters cover the noisiest dev commands: `pytest`, `jest` / `vitest`, `cargo`, `npm` / `pnpm` / `yarn`, `docker`, `kubectl` / `helm`, `aws`, `ruff` / `eslint` / `mypy`, `git`, `make` / `gradle` / `mvn` / `go`, `terraform`, and `pip`. Each one strips ANSI escapes, collapses `\r` progress bars, dedupes repeated lines, groups linter issues by rule, keeps every error block verbatim, and caps total output at 1000 lines / 64 KiB. Disable globally with `TOKEN_GOAT_BASH_COMPRESS=0`, per-filter via `[bash_compress] disabled_filters = ["docker"]` in config.toml, or preview the output of any command with `token-goat compress --cmd '<your command>'`.
168+
143169
## Install
144170

145171
**Windows requirements:** Windows 10 or 11 · Python 3.11, 3.12, or 3.13 · [uv](https://docs.astral.sh/uv/) (`winget install astral-sh.uv`)

0 commit comments

Comments
 (0)