You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 73da20d
Browse filesBrowse the repository at this point in the historyBrowse files
merge: resolve conflicts with origin/main (bash output compression)
Three textual conflicts plus one semantic one between this branch's
feature work and the bash-output-compression feature that landed on
``main`` while this PR was open.
Resolutions:
* **install.py** PreToolUse matcher: ``main`` widened the matcher to
``Read|Bash`` so the bash-compress wrapper can intercept noisy
commands; this branch widened it to ``Read|Grep|Bash`` for the
Grep dedup hint. Final value: ``Read|Grep|Bash`` with a comment
explaining both reasons.
* **CHANGELOG.md**: kept both ``[Unreleased]`` blocks — compression
first, then this branch's eleven surfaces, then the shared
``Changed`` / ``Fixed`` sections.
* **README.md** "What changes" table: kept both sets of rows
(compression filters first, then the dedup / recovery /
auto-redirect / structured-config rows from this branch).
* **hooks_read.py / cli.py**: auto-merged. The resulting
``pre_read`` dispatch is bash-dedup → bash-read-equivalent →
bash-compress → CONTINUE for Bash, grep-dedup → CONTINUE for Grep,
and the existing Read / Glob branches untouched. This ordering
is intentional: a session-cache hit is the cheapest signal and
short-circuits before we ever look at the command shape.
* **test_bash_dedup_hint.py**: the negative-assertion cases used
``pytest -v src/`` and ``make build`` which are now compressible
commands — the bash-compress wrapper fires before pre_read's
fall-through, so the ``hookSpecificOutput not in result``
assertion broke. Switched to ``find`` / ``echo`` / ``head``
which are deliberately not in main's filter list, so the only
reason for ``pre_read`` to emit a hookSpecificOutput is a real
dedup hit. Comment in the test module docstring records the
invariant.
636 tests pass; ruff clean; mypy adds zero new errors over baseline.
Copy file name to clipboardExpand all lines: CHANGELOG.md
+7-3Lines changed: 7 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,30 +6,34 @@ All notable changes to Token-Goat are documented in this file. Format follows Ke
6
6
7
7
### Added
8
8
9
+
- **Bash output compression.** PreToolUse hook on Bash detects compressible commands and rewrites them to flow through `token-goat compress`, which runs the original through the system shell, captures stdout + stderr, applies a per-tool filter, and prints a compressed view that surfaces failures first. Twelve filters cover the noisiest dev commands: `pytest`, `jest` / `vitest`, `cargo`, `npm` / `pnpm` / `yarn` / `bun`, `docker` / `buildah` / `podman`, `kubectl` / `helm`, `aws`, `ruff` / `eslint` / `mypy` / `pyright` / `pylint` / `stylelint` / `biome` / `tsc`, `git`, `make` / `ninja` / `gradle` / `mvn` / `bazel` / `go`, `terraform` / `tofu`, `pip` / `pipx`. Typical savings: pytest 80-97%, npm 88%, docker 75%, linters 80%. Each filter strips ANSI, collapses `\r` progress bars, dedupes consecutive lines, groups linter issues by rule (3 examples per code), keeps every error and warning block verbatim, and caps total output at 1000 lines / 64 KiB. The wrapper preserves the original exit code, kills the process group on timeout (SIGTERM then SIGKILL after a grace period on POSIX), and caps each stream capture at 32 MiB. Configurable via `[bash_compress]` in config.toml (`enabled`, `disabled_filters`, `max_lines`, `max_bytes`, `timeout_seconds`) or disabled with `TOKEN_GOAT_BASH_COMPRESS=0`. Savings are recorded per filter as `bash_compress:<name>`. New CLI subcommand `token-goat compress` for previewing compression on any command.
9
10
-**Post-compaction recovery hint.**``SessionStart`` now detects ``source == "compact"`` and emits a one-shot ``additionalContext`` block listing the most recently-read files, cached Bash outputs (``token-goat bash-output <id>``), and cached WebFetch responses (``token-goat web-output <id>``) from the *pre*-compaction session. The cache is intentionally preserved across the compact so the recovery hint has data to draw from; the cache reset still fires on every other source value (startup / resume / clear / unknown). When the prior session was empty, no hint is emitted — the recovery path is silent until it has something worth surfacing.
10
11
-**Grep dedup hint.** A repeat ``Grep`` invocation with the same ``(pattern, path)`` pair within the staleness window now produces a ``"this ran ~Ns ago and matched N lines"`` advisory. Same mechanism as the bash and web dedup hints but pointed at the existing ``session.greps`` history — no new disk store is involved. Suppressed when the prior result was below 50 matches (the hint preamble would approach the saving).
11
12
-**WebFetch result cache.** A new ``PostToolUse(WebFetch)`` hook persists non-image response bodies to ``data_dir() / "web_outputs"`` and records the ``(url_sha → output_id)`` mapping in the session cache. On a repeat fetch of the same URL the pre-fetch hook emits a dedup hint pointing at ``token-goat web-output <id>``, mirroring the bash-cache pattern. Two new CLI commands surface the cache: ``token-goat web-output`` (with the same ``--head`` / ``--tail`` / ``--grep`` slicers as ``bash-output``, plus ``numbered_lines`` in JSON mode) and ``token-goat web-history``. Disk store is byte-capped (32 MB default) with oldest-first eviction + paired sidecar cleanup.
12
13
-**Dockerfile section extractor.**``Dockerfile``, ``Containerfile``, and ``*.dockerfile`` now produce one ``Section`` per ``FROM`` build stage, so ``token-goat section Dockerfile::builder`` extracts a single stage instead of forcing a full-file read. Multi-stage builds resolve by ``AS <name>`` alias when present; unnamed stages fall back to the image reference so they remain addressable.
13
-
-**Pre-Grep matcher + pre-Bash matcher in install.**``PreToolUse`` now fires on ``Read|Grep|Bash`` (was ``Read``-only) so the new Grep dedup hint actually runs. Pre-existing Bash dedup hint logic in ``hooks_read`` now reaches the wire too — under the old matcher it only fired under Codex where the matcher was already permissive.
14
+
-**Pre-Grep matcher + pre-Bash matcher in install.**``PreToolUse`` now fires on ``Read|Grep|Bash`` (matcher widened from the prior ``Read|Bash``) so the new Grep dedup hint actually runs alongside the Bash compression rewriter from the prior entry.
14
15
-**``token-goat doctor`` cache visibility.** A new ``Caches`` section reports the size, file count, and oldest-entry age for ``bash_outputs/``, ``web_outputs/``, and ``session_snapshots/``. Each row warns when the directory has grown more than 10% over its byte cap, surfacing potential eviction gaps without needing to grep the data directory by hand.
15
16
-**Close-match auto-redirect on ``token-goat symbol``.** When a symbol query returns zero results and the project has exactly one close-match candidate at high confidence (difflib ratio ≥ 0.85), the lookup is automatically re-run against that candidate. The redirected response carries a ``redirected_from`` field in JSON output and a ``(redirected from: …)`` marker in plain-text output so the substitution is auditable. Pass ``--strict`` to disable the redirect and get the previous "Did you mean: …?" suggestion list behaviour.
16
17
-**``bash`` and ``web`` source buckets in stats.**``token-goat stats`` now attributes ``bash_*`` kinds to a visible ``bash`` bucket (orange in the fancy renderer) and ``web_*`` kinds to a new ``web`` bucket (yellow), so the new mechanisms get first-class lines in the by-source panel instead of falling into the ``other`` catch-all. ``grep_dedup_hint`` lands in the existing ``hint`` bucket because it prevents a Read-equivalent burst (consistent with ``diff_hint``).
17
-
18
18
-**Bash output interception.** A new `PostToolUse(Bash)` hook persists large stdout/stderr to disk under `data_dir() / "bash_outputs"` and records the command in the session cache. When the same command is about to run again in the same session, the pre-Bash hint suggests `token-goat bash-output <id>` (optionally with `--head N`, `--tail N`, or `--grep PATTERN`) instead of re-executing — avoiding both runtime cost and duplicated tokens. The store is byte-capped (16 MB default) with oldest-first eviction; outputs above 2 MB are tail-preserved with a truncation marker. Two new CLI commands surface the cache: `token-goat bash-output` retrieves a sliced view, `token-goat bash-history` lists cached entries newest-first.
19
19
-**Diff-aware re-read.**`post_read` now writes a per-session content snapshot (under `data_dir() / "session_snapshots"`, capped at 256 KB per file and 150 snapshots per session) so a follow-up `Read` after a `Write`/`Edit`/`MultiEdit` can be answered with a unified diff hint instead of a `pre_read` blocking message that silently allowed the full re-read. The diff is bounded to 4 KB and only fires when the realised saving exceeds ~250 tokens; below that the existing session-cache hint path runs unchanged. Stats record both the realised saving (`diff_hint`) and the hint's injection cost (`diff_hint_overhead`) for honest accounting.
20
20
-**TOML, YAML, JSON, INI, CFG, and dotenv section extraction.**`token-goat section pyproject.toml::tool.ruff` (and equivalents for `.yaml`, `.yml`, `.json`, `.ini`, `.cfg`, `.env`, and `.envrc`) now extract a single table/key block instead of forcing a full-file read. The TOML scanner emits one `Section` per `[table]` and `[[array]]` header; the YAML scanner emits top-level keys plus one nested layer (`spec.replicas`-style) computed from the file's detected indent; JSON gains depth-1 section detection on pretty-printed files; INI/CFG indexes one section per `[name]` header; `.env`/`.envrc` index each `KEY=value` assignment as a symbol. None of the six pulls in an extra dependency — all use line-scanners and the existing stdlib parsers. The parser dispatcher gained a basename-keyed table (alongside the existing suffix table) so dotfiles with empty extensions (`.env`, `.envrc`) resolve correctly.
21
21
-**Stale-data sweeps in the background worker.**`cleanup_on_startup` now also drops snapshot directories older than 24 hours and enforces the bash-output byte cap, so a long-lived install does not accumulate per-session debris.
22
22
-**Compaction manifest gained a "Commands Run" section.** The PreCompact manifest now surfaces the most recent meaningful Bash invocations (cmd preview, exit code, byte size, cache ID) so the test/build context that drives the next agent turn survives compaction. Each entry includes the `token-goat bash-output <id>` cache key for surgical recall. `event_count` includes `bash_history` so a session whose only activity is a cached test run still clears the `min_events` threshold.
23
23
-**`token-goat bash-output --json` now surfaces line numbers.** The JSON shape adds `numbered_lines` (a 1-based, original-body-anchored `[{lineno, text}]` list) and `total_lines`, mirroring the surgical-read response shape elsewhere in the codebase. Agents can now `--head` / `--tail` / `--grep` filter and still map back to positions in the original output.
24
24
-**Hardened PostToolUse Bash payload extraction.**`_extract_bash_response` now tolerates every documented Bash result shape: dict-with-named-fields (Claude Code), MCP `CallToolResult` content arrays, bare-string blobs, top-level flattening (no `tool_response` wrapper), `tool_result`/`response` aliases, `returncode` and string-typed `exit_code` variants. Each shape is covered by a dedicated regression test in `test_post_bash_payloads.py`.
25
-
-**New stats bucket `bash` (orange) and diff-hint accounting.**`token-goat stats` now attributes `diff_hint` / `diff_hint_overhead` to the `hint` bucket and `bash_dedup_hint*` / `bash_output_cached` to a new visible `bash` bucket, so the new mechanisms appear in the by-source panel instead of falling into the `other` catch-all.
26
25
27
26
### Changed
28
27
29
28
-**`reset_session`** now also removes per-session content snapshots, matching the existing JSON-cache reset semantics.
30
29
-**Codex Bash matcher in `~/.codex/config.toml`** now points at the new `post-bash` hook instead of `post-read`; under Codex, `post-read` previously did nothing for `Bash` calls (no branch in the handler), so this is a strict gain.
31
30
-**`bash_cache.evict_old_entries`** removes body + sidecar pairs together, and runs a second pass to sweep any orphan sidecars left over from out-of-band deletion. Previously, manual `rm` of a body file or a write race could leave a `.json` sidecar with no matching body that lived forever.
32
31
32
+
### Fixed
33
+
34
+
-**`paths.open_log_file` returned a `StreamHandler` instead of a `FileHandler` on POSIX.** The type hint and docstring claimed `FileHandler`, but the implementation wrapped `os.fdopen()` in a bare `StreamHandler` to apply 0o600 permissions, breaking `isinstance(handler, FileHandler)` checks (such as the `test_setup_logging_skips_console_handler_when_not_tty` worker test). Replaced with a private `FileHandler` subclass that overrides `_open` to apply the tighter mode at open time, preserving the type identity callers depend on.
35
+
-**`test_canonicalize_drive_case_collapsed` and `test_canonicalize_cross_shell_paths_produce_same_hash` failed on POSIX.** Both assert Windows-shell drive-letter normalisation invariants that only fire when `Path.resolve()` returns an absolute Windows path; on POSIX `Path("C:/Projects/foo").resolve()` becomes `cwd + "/C:/Projects/foo"` and the assertions test against synthesised POSIX paths. Now skipped on non-Windows with an explanatory message.
Copy file name to clipboardExpand all lines: README.md
+30-4Lines changed: 30 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -36,20 +36,24 @@
36
36
37
37
Claude reads `auth.py`. Then reads it again. Then a third time after compaction wipes the session. You pay for every token.
38
38
39
-
Long sessions accumulate waste three ways. Screenshots cross the model at full resolution. A single PNG can land at 3.3 MB. The agent re-reads files it already parsed earlier in the same conversation. And when a session compacts, the summary LLM doesn't know which files were edited or which symbols mattered, so it preserves the wrong things.
39
+
Long sessions accumulate waste four ways. Screenshots cross the model at full resolution. A single PNG can land at 3.3 MB. The agent re-reads files it already parsed earlier in the same conversation. When a session compacts, the summary LLM doesn't know which files were edited or which symbols mattered, so it preserves the wrong things. And every `pytest`, `npm install`, `docker build`, or `git log` dumps thousands of lines of progress bars, deprecation warnings, and passing-test names that bury the one line that actually matters.
40
40
41
-
Each one is preventable. Token-Goat intercepts all three, automatically.
41
+
Each one is preventable. Token-Goat intercepts all four, automatically.
42
42
43
43
## What changes
44
44
45
45
| Without Token-Goat | With Token-Goat |
46
46
|--------------------|-----------------|
47
-
| 3.3 MB screenshot lands in model context | 84 KB compressed copy — 97.4% smaller |
47
+
| 3.3 MB screenshot lands in model context | 84 KB compressed copy, 97.4% smaller |
48
48
| Agent re-reads files from earlier in the session | "Already read this" reminder with narrow slice suggestion |
49
49
| Agent re-reads a file edited mid-session | Unified diff injected as a hint — full Read avoided when the diff covers the change |
50
50
| Compaction forgets which files were edited | Structured session manifest injected before compact |
51
51
| Same files re-read from scratch after `/compact`| Recovery hint at SessionStart lists cached snapshot + bash + WebFetch IDs |
52
-
| Full file read for one function or section |`token-goat read file::symbol` — about 85% smaller |
52
+
| Full file read for one function or section |`token-goat read file::symbol`, about 85% smaller |
`--budget` is a hard cap. Below 6 KB the output automatically switches to short-label mode (`f:` files, `s:` symbols, `c:` calls) to fit more signal per byte. `token-goat map --compact` is a shortcut for a 300-token budget when you only need the high-rank cluster.
142
146
147
+
**5. Bash output compression**
148
+
149
+
```
150
+
# Without token-goat: pytest dumps every PASSED line + dots + tracebacks.
151
+
$ pytest -v tests/
152
+
... (3 KB of output, 150 PASSED lines, 1 FAILED at the bottom)
153
+
154
+
# With token-goat: the PreToolUse hook rewrites the command to
155
+
# `token-goat compress --filter pytest`. The wrapper runs pytest, captures
156
+
# stdout+stderr, applies the per-tool filter, and prints failures first.
0 commit comments