Skip to content

Commit 1e96258

Browse files
authored
Merge pull request #3 from DFKHelper/claude/compress-command-output-p30fI
bash output compression: 12 per-tool filters, failures-first, ~80-97% smaller
2 parents 471b8e4 + d5e8a98 commit 1e96258

14 files changed

Lines changed: 3772 additions & 26 deletions

‎CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,15 @@ All notable changes to Token-Goat are documented in this file. Format follows Ke
44

55
## [Unreleased]
66

7+
### Added
8+
9+
- **Bash output compression.** PreToolUse hook on Bash detects compressible commands and rewrites them to flow through `token-goat compress`, which runs the original through the system shell, captures stdout + stderr, applies a per-tool filter, and prints a compressed view that surfaces failures first. Twelve filters cover the noisiest dev commands: `pytest`, `jest` / `vitest`, `cargo`, `npm` / `pnpm` / `yarn` / `bun`, `docker` / `buildah` / `podman`, `kubectl` / `helm`, `aws`, `ruff` / `eslint` / `mypy` / `pyright` / `pylint` / `stylelint` / `biome` / `tsc`, `git`, `make` / `ninja` / `gradle` / `mvn` / `bazel` / `go`, `terraform` / `tofu`, `pip` / `pipx`. Typical savings: pytest 80-97%, npm 88%, docker 75%, linters 80%. Each filter strips ANSI, collapses `\r` progress bars, dedupes consecutive lines, groups linter issues by rule (3 examples per code), keeps every error and warning block verbatim, and caps total output at 1000 lines / 64 KiB. The wrapper preserves the original exit code, kills the process group on timeout (SIGTERM then SIGKILL after a grace period on POSIX), and caps each stream capture at 32 MiB. Configurable via `[bash_compress]` in config.toml (`enabled`, `disabled_filters`, `max_lines`, `max_bytes`, `timeout_seconds`) or disabled with `TOKEN_GOAT_BASH_COMPRESS=0`. Savings are recorded per filter as `bash_compress:<name>`. New CLI subcommand `token-goat compress` for previewing compression on any command.
10+
11+
### Fixed
12+
13+
- **`paths.open_log_file` returned a `StreamHandler` instead of a `FileHandler` on POSIX.** The type hint and docstring claimed `FileHandler`, but the implementation wrapped `os.fdopen()` in a bare `StreamHandler` to apply 0o600 permissions, breaking `isinstance(handler, FileHandler)` checks (such as the `test_setup_logging_skips_console_handler_when_not_tty` worker test). Replaced with a private `FileHandler` subclass that overrides `_open` to apply the tighter mode at open time, preserving the type identity callers depend on.
14+
- **`test_canonicalize_drive_case_collapsed` and `test_canonicalize_cross_shell_paths_produce_same_hash` failed on POSIX.** Both assert Windows-shell drive-letter normalisation invariants that only fire when `Path.resolve()` returns an absolute Windows path; on POSIX `Path("C:/Projects/foo").resolve()` becomes `cwd + "/C:/Projects/foo"` and the assertions test against synthesised POSIX paths. Now skipped on non-Windows with an explanatory message.
15+
716
## [0.5.2] - 2026-05-17
817

918
### Fixed

‎README.md‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,18 +36,22 @@
3636

3737
Claude reads `auth.py`. Then reads it again. Then a third time after compaction wipes the session. You pay for every token.
3838

39-
Long sessions accumulate waste three ways. Screenshots cross the model at full resolution. A single PNG can land at 3.3 MB. The agent re-reads files it already parsed earlier in the same conversation. And when a session compacts, the summary LLM doesn't know which files were edited or which symbols mattered, so it preserves the wrong things.
39+
Long sessions accumulate waste four ways. Screenshots cross the model at full resolution. A single PNG can land at 3.3 MB. The agent re-reads files it already parsed earlier in the same conversation. When a session compacts, the summary LLM doesn't know which files were edited or which symbols mattered, so it preserves the wrong things. And every `pytest`, `npm install`, `docker build`, or `git log` dumps thousands of lines of progress bars, deprecation warnings, and passing-test names that bury the one line that actually matters.
4040

41-
Each one is preventable. Token-Goat intercepts all three, automatically.
41+
Each one is preventable. Token-Goat intercepts all four, automatically.
4242

4343
## What changes
4444

4545
| Without Token-Goat | With Token-Goat |
4646
|--------------------|-----------------|
47-
| 3.3 MB screenshot lands in model context | 84 KB compressed copy — 97.4% smaller |
47+
| 3.3 MB screenshot lands in model context | 84 KB compressed copy, 97.4% smaller |
4848
| Agent re-reads files from earlier in the session | "Already read this" reminder with narrow slice suggestion |
4949
| Compaction forgets which files were edited | Structured session manifest injected before compact |
50-
| Full file read for one function or section | `token-goat read file::symbol` — about 85% smaller |
50+
| Full file read for one function or section | `token-goat read file::symbol`, about 85% smaller |
51+
| `pytest` dumps 150 PASSED lines + dots + tracebacks | Failures-first view, 80 to 97% smaller |
52+
| `npm install` floods deprecation warnings + spinner | Errors kept; warnings collapsed by package, ~90% smaller |
53+
| `docker build` emits sha256 digests + transfer progress | Step headers + errors kept; noise dropped, ~75% smaller |
54+
| `ruff` / `eslint` / `mypy` repeat the same rule 50 times | Grouped by rule with first 3 examples, ~80% smaller |
5155

5256
> Four hours of use on the author's machine: **59.7 MB** of data that never hit the model, with an estimated **11.5 million tokens** avoided.
5357
@@ -133,6 +137,28 @@ out: ~4 KB # top-ranked files + key symbols (92% smaller)
133137

134138
`--budget` is a hard cap. Below 6 KB the output automatically switches to short-label mode (`f:` files, `s:` symbols, `c:` calls) to fit more signal per byte. `token-goat map --compact` is a shortcut for a 300-token budget when you only need the high-rank cluster.
135139

140+
**5. Bash output compression**
141+
142+
```
143+
# Without token-goat: pytest dumps every PASSED line + dots + tracebacks.
144+
$ pytest -v tests/
145+
... (3 KB of output, 150 PASSED lines, 1 FAILED at the bottom)
146+
147+
# With token-goat: the PreToolUse hook rewrites the command to
148+
# `token-goat compress --filter pytest`. The wrapper runs pytest, captures
149+
# stdout+stderr, applies the per-tool filter, and prints failures first.
150+
$ token-goat compress --filter pytest --cmd "pytest -v tests/"
151+
= test session starts =
152+
collected 150 items
153+
FAILED tests/test_x.py::test_one
154+
= 1 failed, 149 passed in 2.3s =
155+
156+
[token-goat: collapsed 149 PASSED lines]
157+
[token-goat: pytest filter compressed 4.8 KiB to 0.1 KiB (97% saved)]
158+
```
159+
160+
Twelve built-in filters cover the noisiest dev commands: `pytest`, `jest` / `vitest`, `cargo`, `npm` / `pnpm` / `yarn`, `docker`, `kubectl` / `helm`, `aws`, `ruff` / `eslint` / `mypy`, `git`, `make` / `gradle` / `mvn` / `go`, `terraform`, and `pip`. Each one strips ANSI escapes, collapses `\r` progress bars, dedupes repeated lines, groups linter issues by rule, keeps every error block verbatim, and caps total output at 1000 lines / 64 KiB. Disable globally with `TOKEN_GOAT_BASH_COMPRESS=0`, per-filter via `[bash_compress] disabled_filters = ["docker"]` in config.toml, or preview the output of any command with `token-goat compress --cmd '<your command>'`.
161+
136162
## Install
137163

138164
**Windows requirements:** Windows 10 or 11 · Python 3.11, 3.12, or 3.13 · [uv](https://docs.astral.sh/uv/) (`winget install astral-sh.uv`)

0 commit comments

Comments
 (0)