Skip to content

README.md

Latest commit

 

History

History
109 lines (81 loc) · 3.96 KB

README.md

File metadata and controls

109 lines (81 loc) · 3.96 KB
Raw

autoforge

Autonomous optimization of anything. Inspired by Karpathy's autoresearch.

autoresearch uses an AI agent to autonomously optimize ML training code. autoforge generalizes this pattern to optimize any artifact — articles, code, prompts, resumes, configurations — anything with a measurable quality signal.

The Pattern

┌─────────────────────────────────────────────┐
│                                             │
│   ┌──────────┐    ┌──────────┐    ┌─────┐  │
│   │  Modify   │───▶│ Evaluate │───▶│Keep?│  │
│   │ artifact  │    │ artifact │    │     │  │
│   └──────────┘    └──────────┘    └──┬──┘  │
│        ▲                             │      │
│        └─────────────────────────────┘      │
│                                             │
└─────────────────────────────────────────────┘
Concept autoresearch autoforge
Artifact train.py anything — article, code, prompt, ...
Metric val_bpb pluggable — LLM judge, script, human score
Instructions program.md program.md (templated)
Loop AI agent follows instructions same

Quick Start

1. Clone and install

git clone https://github.com/yourname/autoforge.git
cd autoforge
uv sync

2. Try the article optimization example

# Set your API key
export ANTHROPIC_API_KEY=sk-...

# Run a baseline evaluation
uv run -m autoforge.evaluate examples/article/article.md --rubric examples/article/rubric.md

# Start Claude Code and tell it to follow the program
claude
> Follow the instructions in examples/article/program.md

The agent will enter an autonomous loop: read the article, modify it, evaluate, keep or discard, repeat. Come back to find a better article and a log of every iteration in results.tsv.

Creating Your Own Optimization Target

  1. Define your artifact — the file to be optimized (e.g., my_prompt.txt)
  2. Write a rubric — evaluation dimensions and scoring criteria (see examples/article/rubric.md)
  3. Copy program_template.md and fill in the blanks:
    • {{ARTIFACT_PATH}} — path to your artifact
    • {{RUBRIC_PATH}} — path to your rubric
    • {{EVALUATE_COMMAND}} — how to score it
    • {{MODIFICATION_GUIDELINES}} — what makes a good/bad edit
  4. Run it — point Claude Code at your program.md

Evaluators

LLM Judge (default)

Uses an LLM to score the artifact against a rubric. Fully automatic.

uv run -m autoforge.evaluate artifact.md --rubric rubric.md
uv run -m autoforge.evaluate artifact.md --rubric rubric.md --model gpt-4o --provider openai

Script Evaluator

Run any command that outputs a JSON score.

uv run -m autoforge.evaluate artifact.md --script "python my_scorer.py {artifact}"

Expected output format:

{"score": 72.5, "dimensions": {"clarity": 8, "depth": 7}, "reasoning": "..."}

Custom Evaluator (Python)

Subclass autoforge.evaluate.Evaluator:

from autoforge.evaluate import Evaluator, EvalResult

class MyEvaluator(Evaluator):
    def evaluate(self, artifact_path: str) -> EvalResult:
        # your logic here
        return EvalResult(score=85.0, dimensions={"quality": 8.5})

Philosophy

  • One file, one metric, one loop — keep it simple
  • The agent is the loop — no complex orchestration framework needed
  • program.md is the program — humans write instructions, agents execute
  • Pluggable evaluation — the only thing that changes between domains

License

MIT