harness: opt-in token-budget gate for MicrocompactMiddleware (cache-prefix stability)

[M3gA-Mind]

Problem

MicrocompactMiddleware blanks older tool-result bodies (keeping the newest keep_recent verbatim) to bound a long, tool-heavy transcript. But it fires purely on count:

if tool_idxs.len() <= self.keep_recent { return Ok(()); }

Because "keep the newest keep_recent" is a moving boundary, once a run has more than keep_recent tool results the middleware rewrites one more tool result full → placeholder on essentially every model call. Blanking a body that was sent verbatim on an earlier iteration mutates an already-transmitted prefix position, which invalidates the provider KV-cache from that point onward — on every iteration — even when the whole transcript still fits the model's context window.

The result: the trimming churns the prompt cache to reclaim tokens the model still had room for, which is frequently a net cost loss. Downstream (tinyhumansai/openhuman#4755) this shows up as large input-token bloat on tool/delegated turns with only ~38% of input cached.

Fix (minimal, opt-in, backward-compatible)

Add a token-budget gate to MicrocompactMiddleware:

• new field token_budget: Option<u64> (default None);
• builder .with_token_budget(budget) (0 disables) + a token_budget() getter;
• in before_model, when a budget is set and the pre-blank transcript estimate is <= budget, return early without blanking.

Below the budget the request stays append-only and fully cache-eligible; compaction only kicks in once the transcript would exceed the window — the only time the cache-invalidation cost of blanking an already-sent body actually pays for itself. The gate reads the un-blanked estimate (which grows monotonically as the run appends), so it can never oscillate between blanked and full.

token_budget = None (the default) preserves the existing count-only behaviour byte-for-byte — every existing test is unchanged.

Tests

• microcompact_with_token_budget_is_a_noop_below_budget — >keep_recent tool results but under budget → nothing blanked (the fix; contrast microcompact_clears_older_tool_bodies_and_keeps_recent).
• microcompact_with_token_budget_blanks_once_over_budget — still reclaims when genuinely over budget.
• microcompact_with_token_budget_zero_disables_the_gate — 0 opt-out reverts to legacy blanking.

Activation in openhuman (follow-up on the vendored bump)

This lands the harness capability; it is inert until the host opts in. In openhuman/src/openhuman/tinyagents/middleware.rs (where MicrocompactMiddleware::new(...) is constructed, ~L340), add:

MicrocompactMiddleware::new(self.microcompact_keep_recent, CLEARED_PLACEHOLDER)
    .with_token_budget((context_window as f64 * 0.7) as u64)

context_window is already threaded into the turn seam, so a run that fits the window is never compacted (append-only, cacheable) while long runs still compact to stay under it.

Closes tinyhumansai/openhuman#4755

Summary by CodeRabbit

• New Features

  • Added an optional token budget that controls when older tool-result content is compacted.
  • Content remains intact while the transcript fits within the configured budget, helping preserve prompt-cache stability.
  • Added configuration and inspection support for the token-budget setting.

• Bug Fixes

  • Avoids unnecessary transcript changes when compaction is not needed.
  • Preserves existing compaction behavior when no budget is configured or the budget is exceeded.

[coderabbitai]

📝 Walkthrough

Walkthrough

MicrocompactMiddleware gains an optional token-budget configuration. before_model skips tool-result blanking when the estimated transcript fits within that budget, while zero or unset budgets preserve legacy behavior. Tests cover below-budget, over-budget, and disabled-gate cases.

Changes

Microcompaction token-budget gating

Layer / File(s)	Summary
Budget configuration contract src/harness/middleware/types.rs, src/harness/middleware/library/context.rs	Adds the optional token_budget field, initializes it as disabled, and exposes builder and getter methods.
Before-model budget gate src/harness/middleware/library/context.rs, src/harness/middleware/test.rs	Uses estimated tokens to skip blanking within budget, retains legacy blanking when over budget or disabled, and tests all three behaviors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• [agent][efficiency] Input-token bloat on tool/delegated turns — context re-sent per iteration without trimming (~38% cached) openhuman#4755 — Adds harness-level control for context trimming and prompt-cache stability during repeated tool turns.

Possibly related PRs

• tinyhumansai/tinyagents#35 — Changes transcript token estimation used by this token-budget gate.

Poem

I’m a rabbit with tokens to spare,
Keeping cached prompts smooth as air.
Below the budget, tools stay whole,
Above it, old results lose their soul.
Zero means the old rules run—
Hop, hop, compacting done!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately describes the token-budget gate for MicrocompactMiddleware.
Linked Issues check	✅ Passed	The changes address the issue's prompt-cache stability and tool-result trimming concerns via an opt-in token-budget gate and tests.
Out of Scope Changes check	✅ Passed	The described changes stay focused on MicrocompactMiddleware budget gating and tests, with no clear unrelated additions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

src/harness/middleware/library/context.rs (1)

298-322: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Simplify the duplicated token-estimate branching.

The gate is correct (verified total_message_tokens is computed at most once per call across all emit_events/token_budget combinations), but computing from_tokens and then re-deriving tokens via the same if self.emit_events conditional is easy to desynchronize in a future edit. Consider computing the estimate once, gated by whether either consumer needs it.

♻️ Proposed simplification

-        let from_tokens = if self.emit_events {
-            total_message_tokens(&request.messages)
-        } else {
-            0
-        };
-        if let Some(budget) = self.token_budget {
-            let tokens = if self.emit_events {
-                from_tokens
-            } else {
-                total_message_tokens(&request.messages)
-            };
-            if tokens <= budget {
-                return Ok(());
-            }
-        }
+        let needs_estimate = self.emit_events || self.token_budget.is_some();
+        let from_tokens = if needs_estimate {
+            total_message_tokens(&request.messages)
+        } else {
+            0
+        };
+        if let Some(budget) = self.token_budget {
+            if from_tokens <= budget {
+                return Ok(());
+            }
+        }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/harness/middleware/library/context.rs` around lines 298 - 322, Compute
total_message_tokens once in a local value when either emit_events is enabled or
token_budget is present, then reuse that value for both from_tokens and the
budget check. Remove the duplicated conditional branching while preserving the
existing early return behavior and the zero-value path when neither consumer
needs the estimate.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/harness/middleware/library/context.rs`:
- Around line 298-322: Compute total_message_tokens once in a local value when
either emit_events is enabled or token_budget is present, then reuse that value
for both from_tokens and the budget check. Remove the duplicated conditional
branching while preserving the existing early return behavior and the zero-value
path when neither consumer needs the estimate.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 81656b5c-b7aa-4d4e-b21e-eb2a19154525

📥 Commits

Reviewing files that changed from the base of the PR and between 088ad7c and ab85523.

📒 Files selected for processing (3)

• src/harness/middleware/library/context.rs
• src/harness/middleware/test.rs
• src/harness/middleware/types.rs