Skip to content

Commit ec6b443

Browse files
NetDevAutomateclaude
NetDevAutomate
and
claude
committed
docs: persona optimizer loop plan + session handoff
Plan for the autoresearch iteration loop (eval → feedback → modify persona → re-eval → keep/discard) following the Karpathy autoresearch pattern. Plus session handoff documenting all work completed in the feature-autoresearch worktree session. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 0ade1ea commit ec6b443

2 files changed

Lines changed: 203 additions & 0 deletions

File tree

‎docs/plans/2026-04-06-feat-autoresearch-persona-optimizer-loop-plan.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
---
2+
title: "feat: Autoresearch Persona Optimizer Loop"
3+
type: feat
4+
status: active
5+
date: 2026-04-06
6+
---
7+
8+
# Autoresearch Persona Optimizer Loop
9+
10+
## Overview
11+
12+
Implement the core autoresearch iteration loop for persona prompt optimization.
13+
Following the Karpathy autoresearch pattern (`/Users/ataylor/code/tools/autoresearch/program.md`):
14+
run eval → read scores + feedback → modify persona → re-eval → keep/discard → repeat.
15+
16+
The eval harness (`studyctl eval run`) already scores personas via Bedrock Sonnet 4.6.
17+
What's missing is the **optimizer** that automatically modifies persona files based on
18+
judge feedback, and the **loop** that iterates until success criteria are met or max
19+
iterations exhausted.
20+
21+
## Current State (as of 2026-04-06)
22+
23+
**Working:**
24+
- `studyctl eval run` — runs 7 scenarios via `claude -p` (print mode, no tmux)
25+
- Bedrock Sonnet 4.6 as LLM judge (us-west-2)
26+
- 7-dimension rubric scoring (clarity, socratic_quality, emotional_safety, etc.)
27+
- Judge returns actionable feedback (2-3 suggestions per scenario)
28+
- First passing scores: confused-student 75%, parking-lot 71.9%
29+
30+
**Not working yet:**
31+
- No automatic persona modification based on feedback
32+
- No keep/discard loop (manual re-runs only)
33+
- No iteration tracking or convergence detection
34+
35+
## Proposed Solution
36+
37+
### New command: `studyctl eval optimize`
38+
39+
```bash
40+
studyctl eval optimize --max-iterations 10 --agent claude
41+
```
42+
43+
### The Loop (modelled on autoresearch/program.md)
44+
45+
```
46+
LOOP (up to max_iterations):
47+
1. Run eval (all 7 scenarios via claude -p + Bedrock judge)
48+
2. If all scenarios pass (>= 70%) → DONE, commit persona
49+
3. Collect feedback from failed scenarios
50+
4. Call LLM optimizer: "Given these scores and feedback, modify the persona"
51+
5. Apply modifications to persona files
52+
6. Git commit the modified persona
53+
7. Re-run eval
54+
8. If avg score improved → KEEP (advance)
55+
9. If avg score worsened → DISCARD (git reset to previous commit)
56+
10. Log iteration to eval-results.tsv
57+
```
58+
59+
### The Optimizer (LLM-driven persona modification)
60+
61+
The optimizer is itself an LLM call (Bedrock Sonnet/Opus) that:
62+
- Receives the current persona text
63+
- Receives the judge scores + feedback for each failed scenario
64+
- Returns a modified persona with specific changes
65+
66+
```python
67+
def optimize_persona(
68+
current_persona: str,
69+
eval_results: list[JudgeResult],
70+
scenarios: list[Scenario],
71+
) -> str:
72+
"""Use LLM to generate an improved persona based on judge feedback."""
73+
# Build prompt with current persona + all scores + all feedback
74+
# Ask LLM to return the FULL modified persona text
75+
# Return the modified persona
76+
```
77+
78+
### Files to modify
79+
80+
| File | Change |
81+
|------|--------|
82+
| `packages/studyctl/src/studyctl/cli/_eval.py` | Add `optimize` command |
83+
| `packages/studyctl/src/studyctl/eval/optimizer.py` | NEW — LLM-driven persona modifier |
84+
| `packages/studyctl/src/studyctl/eval/prompts/optimizer.md` | NEW — optimizer prompt template |
85+
| `packages/studyctl/src/studyctl/eval/orchestrator.py` | Add iteration loop with keep/discard |
86+
87+
### Success criteria
88+
89+
- [ ] `studyctl eval optimize --max-iterations 10` runs the full loop
90+
- [ ] Each iteration: eval → feedback → modify persona → re-eval → keep/discard
91+
- [ ] Git commit on keep, git reset on discard
92+
- [ ] Iteration history logged to eval-results.tsv
93+
- [ ] Loop stops when all 7 scenarios pass (>= 70%) or max iterations reached
94+
- [ ] Final optimised persona committed with iteration count in commit message
95+
96+
### Key design decisions
97+
98+
- **Optimizer model**: Same Bedrock model as the judge (Sonnet 4.6). Could use Opus for higher quality.
99+
- **Persona scope**: Modify `agents/shared/personas/study.md` (the main study persona). Other shared files (socratic-engine.md, break-science.md, etc.) are referenced but not modified by the optimizer — they're protocol docs, not persona instructions.
100+
- **Git branch**: Run on a dedicated branch (`autoresearch/persona-YYYYMMDD`), merge to main when optimised.
101+
- **Keep/discard threshold**: Keep if avg_score improves by any amount. Discard if equal or worse.
102+
- **Feedback aggregation**: Aggregate feedback from ALL failed scenarios into one optimizer call, not per-scenario.
103+
104+
## References
105+
106+
- Karpathy autoresearch pattern: `/Users/ataylor/code/tools/autoresearch/program.md`
107+
- Current eval harness: `packages/studyctl/src/studyctl/eval/`
108+
- Current persona: `agents/shared/personas/study.md`
109+
- Judge rubric: `packages/studyctl/src/studyctl/eval/prompts/judge-rubric.md`

‎docs/plans/2026-04-06-session-handoff.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
title: Session Handoff — feature-autoresearch worktree
3+
date: 2026-04-06
4+
session: feature-autoresearch
5+
---
6+
7+
# Session Handoff
8+
9+
## What was accomplished
10+
11+
This session completed the remaining items from the v2.3 handoff plus significant
12+
eval harness work. Starting from a clean worktree branched off main.
13+
14+
### Completed Tasks
15+
16+
| # | Task | Commits |
17+
|---|------|---------|
18+
| 1 | **Autoresearch Phase 3** — reporting, regression detection, CI workflow | `36ae833` |
19+
| 2 | **Content pipeline integration tests** — 9 new CLI tests + 9 async tests | `db426ac` |
20+
| 3 | **Pre-push hook asyncio fix** — `-p no:playwright` eliminates Textual/Playwright conflict | In commit 1 |
21+
| 4 | **Eval: pane targeting** — capture agent pane by explicit ID, not session name | `a70cbad` |
22+
| 5 | **Eval: two-phase capture** — wait for content growth before stability check | `25b0b29` |
23+
| 6 | **Eval: trust dialog** — auto-accept Claude Code trust prompt + trust specific dirs | `cc29fd3` |
24+
| 7 | **Eval: Bedrock provider** — LLM judge via AWS Bedrock Converse API (Sonnet 4.6) | `e184652` |
25+
| 8 | **Eval: judge feedback** — 2-3 actionable suggestions per scenario in rubric response | `0eb446b` |
26+
| 9 | **Eval: claude -p print mode** — replaced entire tmux capture with headless print mode | `cb246d4` |
27+
| 10 | **Eval: logging** — agent response preview, dimension scores, feedback in output | `0ade1ea` |
28+
29+
### Key Metrics
30+
- **Test count**: 17 new autoresearch tests + 18 new content tests = 35 new tests
31+
- **First passing eval scores**: confused-student 75%, parking-lot 71.9%
32+
- **Eval approach**: `claude -p` print mode (no tmux) + Bedrock Sonnet 4.6 judge
33+
34+
### Config Changes (user's machine)
35+
- `~/.config/studyctl/config.yaml` — added eval judge config (Bedrock, us-west-2)
36+
- boto3 installed in main repo venv (not in pyproject.toml — optional eval dep)
37+
38+
## What remains
39+
40+
### Autoresearch Persona Optimizer Loop (NEW — plan written)
41+
42+
Plan at: `docs/plans/2026-04-06-feat-autoresearch-persona-optimizer-loop-plan.md`
43+
44+
The core autoresearch iteration loop:
45+
1. `studyctl eval optimize --max-iterations N` command
46+
2. LLM-driven persona modification based on judge feedback
47+
3. Keep/discard via git commit/reset
48+
4. Loop until all 7 scenarios pass or max iterations
49+
50+
This is the **main remaining work** — the eval harness scores personas,
51+
but doesn't yet automatically iterate on them.
52+
53+
### Eval Tuning (minor)
54+
- Some scenarios score 0% due to heuristic pre-flight failures (the response
55+
is good but fails a structural check like "contains_question"). May need to
56+
relax heuristics or make them configurable per scenario.
57+
- `CLAUDE_TIMEOUT = 120` in persona.py may need increasing for Opus responses.
58+
59+
### Lower Priority
60+
- Add boto3 as an optional eval dependency in pyproject.toml (`[project.optional-dependencies] eval = ["boto3"]`)
61+
- Clean up old tmux capture code in `capture.py` (now unused by PersonaTarget but
62+
may still be used by other tests)
63+
- The `eval-results.tsv` file generated by eval runs should be in `.gitignore`
64+
65+
## Architecture Notes
66+
67+
### Eval flow (current)
68+
```
69+
studyctl eval run
70+
→ PersonaTarget.setup() — builds canonical persona, writes to temp file
71+
→ PersonaTarget.run() — claude -p with --append-system-prompt-file
72+
→ LLMJudge.score() — heuristic pre-flight → Bedrock Sonnet 4.6 rubric
73+
→ JudgeResult with dimensions + feedback
74+
→ TSVReporter.record() → eval-results.tsv
75+
→ Markdown report → docs/reports/
76+
```
77+
78+
### Key files
79+
- `packages/studyctl/src/studyctl/eval/targets/persona.py` — claude -p based eval target
80+
- `packages/studyctl/src/studyctl/eval/judge/llm.py` — heuristic + LLM rubric judge
81+
- `packages/studyctl/src/studyctl/eval/llm_client.py` — Ollama/OpenAI/Bedrock client
82+
- `packages/studyctl/src/studyctl/eval/prompts/judge-rubric.md` — 7-dimension rubric + feedback
83+
- `packages/studyctl/src/studyctl/eval/orchestrator.py` — scenario loop
84+
- `packages/studyctl/src/studyctl/cli/_eval.py` — CLI commands (run, setup, history)
85+
86+
### Bedrock config
87+
```yaml
88+
eval:
89+
judge:
90+
provider: bedrock
91+
model: us.anthropic.claude-sonnet-4-6
92+
region: us-west-2
93+
```
94+
Uses default AWS credential chain (env vars / ~/.aws/credentials).

0 commit comments

Comments
 (0)