Skip to content

#185 - evaluate resource visibility against useful outcomes - #194

Merged
dpuglielli merged 4 commits into
mainfrom
issue/feature-185/resource-visibility-evaluation
Oct 8, 2026
Merged

dpuglielli merged 4 commits into
mainfrom
issue/feature-185/resource-visibility-evaluation

Conversation

@dpuglielli

Copy link
Copy Markdown
Collaborator

Summary

Refs #185; builds on merged #184.

Adds an explicitly configured Claude display mode that injects observed token categories and elapsed wall time into context before prompts and after tool use. Disabled and collect modes remain silent; normal plugin installation registers no resource hooks. Codex supports passive collection only. Mantra is bumped to 0.9.0.

The eight-run comparison found seven passing outcomes: all four collect runs passed; the display clarification run missed a necessary question and failed. No consumption fixation was observed. The recommendation retains passive opt-in collection and ends display evaluation, keeping the small display seam for reproducibility. This is not a causal finding: there is one run per task/condition, partial measurements, guidance bundled with counters, clarification-pair unblinding and a response-delay deviation.

Includes isolated fixtures, native run evidence, independently frozen outcome and fixation reviews, overhead records, and reconciled documentation. Preserves upstream behavioral guidance and collector accounting; consolidates recommendations and keeps fixtures outside the installed plugin. Repeated measured display snapshots are 137–140 characters; exact injected token cost is unknown.

Test Plan

  • Full npm test passes, including 64 Mantra assertions.
  • npm run validate:plugins and strict Claude marketplace/Mantra validation pass.
  • Isolated Codex plugin installation succeeds at 0.9.0; fresh skill probes fired on both hosts.
  • Live Claude probes verify mid-run display freshness and native usage accounting; Codex collection remains fixture-validated.
  • Ada independently reviewed outcomes, fixation, integration, token efficiency and documentation; relative links and frozen source hashes verified.

Acceptance Criteria

  • Verified mantra: collect trustworthy resource measurements with explicit coverage #184 measurements and distinct disabled/collect/display modes.
  • Observational feedback without scores, quotas, countdowns or automatic stops.
  • Outcomes, missing requirements, rework, interactions, consumption and fixation recorded separately.
  • Implementation/evaluation overhead recorded, with unknown costs explicit.
  • Reproducible evidence, limitations and provisional recommendation supplied.
  • Negative findings retained; a cheaper incorrect result is not a success.

# Conflicts:
#	.claude-plugin/marketplace.json
#	docs/reviews/resource-measurements-verification.md
#	mantra/.claude-plugin/plugin.json
#	mantra/.codex-plugin/plugin.json
#	mantra/context/resources.md
#	mantra/hooks/resources.js
#	mantra/package.json
@dpuglielli
dpuglielli merged commit caf70df into main Oct 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant