TinyAgents migration (v0.58.7 → main) parity & bug-fix tracker — harness audit findings

[senamakel]

TinyAgents migration (v0.58.7 → HEAD): parity & bug-fix tracker

A full audit was run comparing production release v0.58.7 against current main over src/openhuman/agent/ (especially harness/) — the TinyAgents migration (#4249, landed via #4261 → #4372 → #4399) plus follow-ups: Langfuse tracing (#4434), memory protocol (#4444), tool-failure classification (#4445). ~4,900 insertions / ~15,200 deletions across 98 files; the in-house engine (engine/, tool_loop.rs, token_budget.rs, …) was replaced by the published tinyagents = "1.3" crate via the seam in src/openhuman/tinyagents/.

Audit verdict

The migration is thoughtfully executed — approval gate, tool policy, iteration caps, cap checkpoints, streaming, payload summarizer, cache-align, handoff cache internals, spawn-depth/tier/sandbox enforcement, and checkpoint/resume were all verified faithfully re-ported. Genuine improvements landed too (real streaming deltas, durable event journal, max_tool_calls + wall-clock backstops, #4093 empty-final re-prompt, orphaned tool_call_id elimination).

But it is not at parity. The recurring failure pattern: behaviors that lived inline in the old loop were re-expressed as middleware/events/spawned tasks whose semantics differ subtly — and in several places a comment asserts parity the code doesn't deliver (handoff ordering, empty-allowlist convention, capture-middleware ordering). Separately, two follow-up PRs shipped with broken end-to-end assumptions (memory protocol pivots on an unregistered tool; the failure classifier was tuned against strings our own approval gate never emits).

Test accounting: net −67 test functions (1,029 → 962, +79 seam tests) despite −6.7k test lines — most coverage was rewritten or moved into the crate (~711 test attrs). The behaviors that lost all coverage line up with the dropped features tracked below. Spec Phase 11 (parity matrix, behavior-cluster ports) remains unchecked in docs/tinyagents-migration-spec.md.

Plan

Phase 1 — Critical correctness & security (release blockers)

• Harness: fatal tool-arg schema validation aborts the whole turn (legacy loop self-corrected) #4451 Fatal tool-arg schema validation aborts the whole turn
• Subagent empty tool allowlist is fail-open — child inherits parent's full tool surface (shell/spawn) #4452 Subagent empty tool allowlist is fail-open (prompt-injection escalation)
• Credential scrubbing dropped from tool outputs on the tinyagents path #4453 Credential scrubbing dropped from tool outputs
• Trace exporter ignores capture_content — prompts/replies written to NDJSON/app log; Langfuse cached-token double-count #4454 Trace exporter ignores capture_content (PII into files/log) + Langfuse usage double-count

Phase 2 — Durable-state corruption & leaks

• Mid-turn steering corrupts persisted history — pre-steer tool rounds and the steer text are dropped #4455 Mid-turn steering corrupts persisted history
• Steering forwarder task leaks on cancelled turns; late steer messages silently lost #4456 Steering forwarder leak on cancelled turns; late steers lost
• Turn finalization: empty assistant row on EmptyProviderResponse, stale error slot, duplicate TurnCompleted #4457 Turn finalization: empty assistant row / stale error slot / duplicate TurnCompleted
• Streamed model calls run in detached spawn: thread_id/task-locals lost, cancellation doesn't stop billed provider call #4460 Detached spawn for streamed model calls: task-locals lost, cancellation doesn't stop billing

Phase 3 — Follow-up PR repairs

• Memory protocol mandates unregistered update_memory_md tool — permanent index-drift nag loop; non-atomic MEMORY.md writes #4458 Memory protocol: register update_memory_md; atomic MEMORY.md writes
• Tool-failure classification: approval denial → 'try again', TTL expiry → auto-retry Timeout, failure missing from snapshot #4459 Failure classification: [policy-denied] handling, TTL expiry, snapshot persistence

Phase 4 — Parity restoration

• Context compaction: summarizer failure aborts the turn; re-summarizes on every model call once over threshold #4461 Compaction: summarizer failure aborts turn; per-call re-summarization
• Token-budget regressions: image marker cost dropped, fixed trim reserve guts small local models, silent trim #4462 Token budget: image marker cost, trim reserve, silent trim
• Restore loop guards: repeat-call/output breakers, recoverable-failure headroom, terminal-inference halt, autonomous iter-cap lift #4463 Loop guards: repeat breakers, recoverable headroom, terminal-inference halt, autonomous iter-cap lift
• after_tool middleware order inverted: progressive-disclosure handoff is dead code, outcome capture sees pre-cap content #4464 after_tool middleware ordering: handoff dead, capture pre-cap
• PFormat tool-call syntax dropped from parser while ~10 builtin prompts still teach it #4465 PFormat parser/prompt contract
• Subagent persistence/observability: failed runs persist nothing, degraded mirror, lossy deltas, tokenjuice off, breaker halt reports Completed #4466 Subagent persistence & observability
• Cost & telemetry parity: charged USD lost, cap-summary/failed-run usage unrecorded, elapsed_ms=0, tool DomainEvents gone #4467 Cost & telemetry parity

Phase 5 — Tests & cleanup

• Test parity matrix: port behavior clusters that lost all coverage in the tinyagents migration (spec Phase 11) #4468 Test parity matrix (spec Phase 11)
• Misc parity cleanups & doc drift from the tinyagents migration #4469 Misc parity cleanups & doc drift

Working notes

• Each child issue carries: problem, evidence (file:line at current HEAD), concrete failure scenario, code-level fix plan, acceptance criteria.
• Several fixes belong in the seam (src/openhuman/tinyagents/); a few may warrant upstream tinyagents crate changes (validation policy, trim estimator hook, working-transcript compression) — noted inside the child issues.
• Suggested sequencing within a phase: Harness: fatal tool-arg schema validation aborts the whole turn (legacy loop self-corrected) #4451+Subagent empty tool allowlist is fail-open — child inherits parent's full tool surface (shell/spawn) #4452 first (user-visible breakage + security), after_tool middleware order inverted: progressive-disclosure handoff is dead code, outcome capture sees pre-cap content #4464 before Credential scrubbing dropped from tool outputs on the tinyagents path #4453 (scrub placement depends on the corrected middleware order).