Findings · Example run · Retrospective · Architecture · Evaluation · Quickstart · Watch a run · Deploying
An autonomous software-engineering agent. Give it a public GitHub issue URL; it clones the repository, explores the codebase, writes a plan for you to approve, edits code in an isolated container, runs the test suite, repairs its own failures, and — once you approve a second time — forks, pushes, and opens the pull request.
Issue URL → Clone → Retrieve → Plan → [your approval] → Code → Test → ⟲ Debug
→ Review → PR summary → [your approval] → Fork → Push → Pull request
Built on LangGraph, with tools discovered dynamically over MCP and all execution confined to a throwaway Docker container. It runs from a terminal, from an HTTP API, or from a page you can watch it work on.
Validated on a real repository. It completed an end-to-end repair of a genuine
bug in pallets/click — 924k characters of
source, 7.7× more than fits in the context window it would need to read everything
— for about $1.22, with the repair loop firing live. That is one of six
ground-truth cases run end to end; the other five are built and unrun.
But the more useful part of this project is why it works, and the several confident assumptions that turned out to be wrong on the way. The one-line summary is below and the full detail further down; that's the part worth reading.
Two documents go deeper than this one:
- docs/example-run.md — one real bug, annotated end to end: what retrieval surfaced, the first patch, the failure that killed it, the second patch, and the passing suite. Every excerpt is a copied artifact.
- docs/engineering-retrospective.md — the full narrative: what was tried, what was disproved, what it cost, and what I would build differently.
Which components of an autonomous software-engineering agent materially improve correctness, and which merely add complexity?
Every experiment here answers part of that. Four of them answered in the opposite direction to what I expected, which is why the architecture looks the way it does rather than the way it was planned.
| Outcome | |
|---|---|
| Real repository | End-to-end repair of a genuine pallets/click bug, repair loop fired live — 1 of 6 ground-truth cases run |
| Issue URL → pull request | Runs unattended on a deployed host and opens the PR itself — DFS replaced with Kahn's algorithm, cycle detection fixed as a side effect |
| Retrieval benchmark | P@3 0.33 → 0.83 on click, 0.25 → 0.67 across both repositories, offline and free |
| Second repository | Generalises in direction, not magnitude — rich 0.17 → 0.50, flat layout, no src/ to exploit |
| Correctness driver | Ranking, not recall. The right file was already in the candidate set in 6/6 cases |
| Cost finding | Better retrieval bought correctness and not cost — $1.219 to pass vs $1.226 to fail |
| Toy fixture | 5/5 solved — and a one-shot prompt with no tools, tests or repair loop tied it at 40% of the cost |
| Tests | 497 offline (no Docker, no datastore, no API key) + 30 gated behind real infrastructure |
| Spend through v1.0 | $8.43, of which $3.77 bought four failed runs and one lesson about instruments |
❌ marks a belief the measurement destroyed, ✅ one that survived, ⚠ a defect in how the work was being measured. Each links to the experiment that produced it.
| Finding | |
|---|---|
| ❌ | Whole-file retrieval was not the dominant cost — windowing moved total tokens 2.6%, because a cap upstream was already binding |
| ❌ | Reducing context increased total cost — 406k → 680k tokens and 19 → 39 calls; with less state the agent re-explores |
| ❌ | Cumulative token budgets are the wrong instrument for a tool loop — a re-sent transcript bills a cached prefix once per turn |
| ❌ | Candidate recall was already perfect — 6/6 before ranking, so improving search was worth nothing |
| ✅ | The ranking objective was wrong, not the search — content features beat mention-counting with no embeddings, vector store or reranker |
| ❌ | Query cleaning improved context size, not accuracy — 34% less context, P@3 changed by exactly zero |
| ⚠ | A weak objective was gating the strong one — pipeline ordering, neither a retrieval nor a ranking bug |
| ⚠ | Offline and production disagreed about what a regex means — the benchmark and the system it measured had divergent semantics |
| ❌ | Better retrieval buys correctness, not efficiency — the headline: outcome flipped, spend did not move |
| ❌ | Prompt length is inversely related to signal density — a 42-character commit subject beat a 1,321-character bug report |
| ⚠ | On the toy fixture, the whole agent tied a one-shot prompt — a property of the fixture, and the reason the project moved to a real repository |
Rather than treat the agent as one black box, it's decomposed into subsystems that are benchmarked independently.
Retrieval, offline, no model calls — two repositories, ground truth from real bugfix commits:
| strategy | P@1 | P@3 | MRR | impl rank | ctx bytes | retrieval calls |
|---|---|---|---|---|---|---|
filesystem (control) |
0.17 | 0.25 | 0.232 | 3.2 | 183,796 | 16 |
filesystem+clean-query |
0.17 | 0.25 | 0.228 | 4.2 | 127,445 | 16 |
clean-query+content-rank |
0.42 | 0.67 | 0.542 | 2.4 | 120,988 | 43 |
Per repository, because an average would hide a layout-specific win:
click (src/ layout) |
rich (flat, no src/) |
|
|---|---|---|
| control P@3 | 0.33 | 0.17 |
| best P@3 | 0.83 | 0.50 |
End to end, one paid run per configuration, same case, only the retriever changed:
| retriever | solved | tokens | cost | model calls |
|---|---|---|---|---|
clean-query+content-rank |
2/2 PASS | 552,989 | $1.2190 | 31 |
filesystem (control) |
0/2 FAIL | 583,029 | $1.2255 | 29 |
Every one of these came from an experiment that contradicted what I expected.
The first real-repository run died at 417k tokens against a 400k ceiling, and
core.py alone is ~35k tokens — so returning whole files looked like the obvious
culprit. Windowing retrieval to ±40 lines around each match changed total tokens
by 2.6%.
The cap upstream was already binding. render_context truncated to 24k characters
regardless, so windowing improved which 24k the model saw without changing how
much. Worth keeping for relevance; worthless for cost.
Cutting per-call context from 24k/60k to 9k/18k characters took tokens from 406k to 680k and calls from 19 to 39. With less state in hand the agent re-explores.
Per-call context and iteration count trade against each other, so trimming context is not a cost lever. Stated carefully: this is an interaction effect under the current planner/coder coupling, not a clean causal law — that run changed two variables.
The loop re-sends its transcript every turn, so one retrieval context counted once per iteration — 19 calls × the same 24k block. With prompt caching that prefix costs a tenth of list price, which is why 406k tokens billed $0.87 instead of $1.22. Cost is the honest ceiling; the token count is now only a runaway backstop.
In all 6 click cases, the file the real fix touched was already in the candidate set before ranking. Retrieval wasn't missing it; ranking was burying it.
That single measurement is what made the rest of the work targeted instead of speculative — there was no point improving search at all.
Ranking by "how many queried symbols does this file mention" hands the win to
tests and changelogs by construction: a unit test writes click.confirm(...)
five times where the implementation writes def confirm(...) once, and a
changelog mentions every symbol that ever existed.
Replacing the objective with content features — defines ≫ imports ≫ calls,
minus penalties for markup, changelog shape and assertion density — took click's
P@3 from 0.33 to 0.83. No embeddings, no vector store, no reranker, and
deliberately no path prior: src/** beating tests/** would score perfectly
on a benchmark whose every answer lives in src/, which measures the benchmark
rather than the ranker.
Bug reports contain reproduction scripts and pasted terminal sessions, and the
heuristic "backticked spans are the important ones" is precisely backwards for
those. The extractor was searching for False, Hello, World, CliRunner, and
in one case Python, help, copyright, credits, license — the interpreter's
start-up banner.
Fixing it with region classification (prose / code / console / REPL banner / traceback, each weighted) changed P@3 by exactly zero. It cut context 34%. The queries were bad and were not the accuracy bottleneck.
The content ranker was only shown candidates that survived a top-18 pre-filter ranked by the old mention-count objective. The correct file sat at rank 44 and 30 in two cases and never reached the ranker at all.
This wasn't a retrieval problem or a ranking problem — it was pipeline ordering.
Retriever → top-18 → good ranker instead of Retriever → good ranker → top-18.
The definition pattern used [[:space:]], a POSIX class Python's re rejects, so
every definition search silently failed in the offline benchmark. Production
called grep without -E, where (def|class) is a literal string — so it would
have failed there too, differently, for a different reason.
The benchmark and the system it measures had divergent semantics. Both now use ERE, with the pattern verified against both engines by a test.
The headline result. Holding everything else constant and changing only the retriever: 2/2 PASS versus 0/2 FAIL, at 31 versus 29 model calls and $1.219 versus $1.226.
With poor retrieval the agent confidently patched the wrong place, failed its tests, and the debugger correctly declined to retry. It was misdirected, not lazy — and being misdirected costs the same as being right.
Stated at the width of the evidence: under this architecture, on this case, changing only the retriever changed the outcome while leaving spend flat. One controlled pair is not a solve rate. What it is — and what the four earlier runs were not — is a single-variable comparison, which is the only kind that can attribute anything.
A 42-character commit subject retrieved its target at rank 1. A 1,321-character real bug report retrieved nothing. Terse text is almost all signal; a long report is mostly repro script, console paste and issue-template boilerplate.
Before any of the above, the full pipeline scored 5/5 against a one-shot Claude call with no tools, no tests and no repair loop — which also scored 5/5, at 40% of the cost.
That result is a property of the fixture, not the architecture: 20 files fit in
one prompt, so the control never needed retrieval, and every defect was fixable
first try so the repair loop never engaged. It's why the project moved to a real
repository, and eval/baseline.py prints that caveat itself so nobody quotes the
tie as a win.
Four seams, each built up front because retrofitting it would mean rewriting every node:
| Seam | Where | What it buys |
|---|---|---|
| Typed contracts | contracts.py |
Routing reads typed fields, not prose. Nodes testable without a model. |
| Run lifecycle | lifecycle.py |
RunPhase state machine with a legal-transition table; illegal transitions raise. Resume reads one enum. |
ToolRegistry |
tools/registry.py |
Nodes never import LangChain or MCP types. Fakes in tests, no MCP process needed. |
Retriever protocol |
retrieval/base.py |
The graph is retrieval-agnostic; swapping strategies is a config value. |
The supervisor is a pure function, not an LLM call. The next node follows from
(phase, last typed output) — deterministic, unit-testable, and free. An LLM
router earns its place when a decision is ambiguous; a red suite going to the
debugger never is.
Retrieval is decomposed further, because that's where the measurement pointed:
Issue → Query Generator → Retriever → Ranker → Context Builder
(query.py) (ranker.py)
Modules are benchmarked independently. Module 1 was fixed and falsified as the bottleneck; module 2's recall measured 100%; module 3 was the whole problem.
Proof the seams are real: tests/test_graph.py runs a complete agent loop —
retrieve, plan, approve, code, test, debug, re-code, test, review, summarise — with
no network and no container. If any abstraction were decorative, that file
could not exist.
- Agent commands never reach a shell. Parsed with
shlex, passed as argv straight toexecve, so chaining is structurally impossible rather than blocklist-dependent. A Docker test proves it:echo $(id -u)returns the literal string. - Executable allowlist, with a regression test asserting
bash,sh,curl,wget,nc,ssh,sudoare absent. - Path validation, 28 pure tests — rejects
..escapes, absolute paths outside the worktree, null bytes, and/workshop(which a naive/workprefix check accepts). - Network cut after dependency install, verified unreachable.
- Non-root uid 10001,
cap_drop=ALL,no-new-privileges, 2 GB / 2 CPU / pids caps. - A reaper removes orphaned containers at startup, label-scoped.
Scoring is baseline-aware (FAIL_TO_PASS / PASS_TO_PASS). A baseline suite runs before any edit; success means fixed something, broke nothing. Requiring a fully green suite was wrong — with five independent seeded defects, no single correct patch could ever achieve it.
Three integrity rules, each closing a way to pass dishonestly:
- Test edits are reverted before scoring — for the agent and the baseline.
- A shrinking suite is disqualifying. Deleting a passing test creates no regression, so a pass/fail diff cannot see it; the collected count can.
- Unparseable output is never success. "We couldn't tell" ≠ "it passed."
Real-repository cases follow SWE-bench: revert a bugfix commit's source only, keep its tests, so FAIL_TO_PASS is known rather than guessed. Every end-to-end case is verified in a container — red with the bug, green with the real fix — before it enters the set.
Two confounds are measured rather than assumed away: issues that name the file the fix must touch (retrieval handed over, not tested), and text that may describe the fix. Both are reported per case so results can be segmented instead of averaged.
- Docker (datastores + the per-run sandbox)
- Python 3.13, via
uv - An Anthropic API key
gh, authenticated — only to publish pull requests- Node 22 — only for the browser frontend
cp .env.example .env # then set ANTHROPIC_API_KEY
docker compose up -d # postgres + redis
uv sync
uv run fpilot doctor # check everything is reachable
uv run fpilot solve --issue fixtures/issues/01-off-by-one.mduv run fpilot solve https://github.com/<owner>/<repo>/issues/<n> --push --draftThe repository is shallow-cloned to .fp/clones/<run_id>/, solved, and — after
the plan gate and a second gate that shows the exact branch, title, and body —
published. Publishing forks the upstream repository, force-pushes
feature-pilot/issue-<n>-<slug>, and opens a cross-repo PR; when you already
own the repository there is nothing to fork, so the branch goes straight there.
Re-publishing the same issue updates that one PR rather than opening another.
--push implies nothing on its own: without it, the run stops at a diff and a
summary, exactly as before. --yes skips both gates for unattended runs, and
--draft opens the PR as a draft.
The same flow over HTTP, where the publish request is the approval:
curl -XPOST localhost:8080/runs -H 'content-type: application/json' \
-d '{"issue_url":"https://github.com/<owner>/<repo>/issues/<n>"}'
# ... watch /runs/{id}/stream until it reports DONE, then:
curl -XPOST localhost:8080/runs/{id}/publish -d '{"draft":true}' \
-H 'content-type: application/json'GITHUB_TOKEN is used when set; otherwise gh's own credential is.
The claim worth making is not "it opened a pull request" — it is here is the
reasoning, the tool calls, the failing test, the repair, and then the pull
request. That claim is invisible in a terminal only its author runs, which is
what web/ exists for: paste an issue link, approve the plan when it
parks, and watch the phases, tool calls, per-role spend and diff arrive live over
SSE.
docker compose up -d --wait # postgres + redis
uv run fpilot serve # the API on 127.0.0.1:8080
cd web && npm install && npm run devNEXT_PUBLIC_FP_API_URL names the API and defaults to http://localhost:8080,
so a fresh clone runs with no env file. web/README.md covers the rest —
including the bug that justified the smoke test, where the API sends two
different JSON shapes under the same SSE event names and both silently vanished
while types and unit tests passed.
The sandbox drives the Docker API directly, so it needs a real daemon. That one fact rules out Render, Railway, Fly, Koyeb, Cloud Run and Fargate at any price — a capability limit, not a pricing one. What is left:
| Cost | Good for | |
|---|---|---|
| GitHub Actions | free on public repos | one-off runs, reproducible demos |
| Any free VM + tunnel (Oracle Always Free) | free | an always-on API a browser can call |
The second row is what actually runs: an Always Free ARM instance provisioned by
Terraform, the API under systemd, and a cloudflared
tunnel for the HTTPS a browser insists on. It deploys itself — a systemd timer
resets to main, syncs, restarts, and health-checks, and CI then confirms the
commit over that HTTPS rather than reaching in over SSH. Pull, not push, because
the host's SSH is open to one address and a runner is never at it; the first
version of that workflow tried to SSH in and timed out, which is how the design
got decided. /health reports its running commit for exactly this reason.
The frontend is a separate concern and deploys to Vercel from web/ on push.
deploy/README.md has all three, the public-mode settings
(FP_MAX_USD_PER_DAY, FP_MAX_CONCURRENT_RUNS, FP_ALLOW_LOCAL_REPOS,
FP_API_CORS_ORIGINS), how a visitor's own keys stay scoped to their run, and
the Oracle iptables trap that costs an afternoon if you meet it unwarned.
ANTHROPIC_API_KEY is the only value you must set. Everything else has a working
local default — embeddings run offline, tracing is a no-op without a LangSmith key,
and pointing the FP_MODEL_* settings at ollama/... removes the hosted
dependency entirely.
Variable naming. Third-party services keep their conventional names
(ANTHROPIC_API_KEY, LANGSMITH_API_KEY, DATABASE_URL, REDIS_URL), so a key
you already have exported works untouched and the langsmith CLI reads the same
variable the app traces with. Only settings that are genuinely ours take the FP_
prefix — MAX_ATTEMPTS unprefixed would collide with anything.
# Retrieval, offline — seconds, no model calls, no container
uv run python -m eval.retrieval_bench --per-case
# Build ground-truth cases from a real repository
git clone https://github.com/pallets/click /tmp/oss-eval/click
uv run python -m eval.oss_build --limit 200 --want 6
# Stage-by-stage ablation: code-only → +retrieve → +plan → +test → +debug → full
uv run python -m eval.ablation --suite click --rungs full
# Fixture gate, and the one-shot control it is compared against
uv run python -m eval.gate
uv run python -m eval.baseline
# What did the agent spend its tool calls on?
uv run python -m eval.search_profileTracing is automatic when LANGSMITH_API_KEY is set — LangGraph instruments its
own nodes, and tracing.py adds spans for the parts LangChain cannot see (MCP tool
execution, retrieval, the containerised test run). Metrics are attributed per role
and per node, which is what made "the coder is 96% of spend" visible and therefore
actionable.
For querying traces from the terminal, this repo expects LangChain's skills and CLI, which are not vendored here:
git clone https://github.com/langchain-ai/langsmith-skills /tmp/ls-skills
mkdir -p .claude/skills && cp -r /tmp/ls-skills/config/skills/* .claude/skills/
curl -fsSL https://cli.langsmith.com/install.sh | shuv run pytest # 497 tests: seams + units. No Docker, no datastore, no API calls
uv run pytest -m docker # 26 tests: real container, real MCP servers
uv run pytest -m postgres # 2 tests: checkpoint round-trip against the compose Postgres
uv run pytest -m llm # real model calls (costs tokens)
uv run pytest -m github # real GitHub: opens a draft PR on the fixture repo, then closes itThe default suite deliberately runs without Docker or a live MCP server. If a node
can't be exercised against a fake ToolRegistry and a stub Retriever, the
abstraction is decorative — that constraint is the test.
That constraint is also what makes CI cheap: every push runs ruff check,
ruff format --check, mypy src eval and the offline suite, with no daemon, no
datastore and no API key. CI reports 489 passed, 8 skipped — the 8 need a local
fixture virtualenv that isn't committed. mypy covers the evaluation harness as
well as the agent, because three of this project's bugs corrupted scoring
silently, and a harness that miscounts is worse than no harness.
The first CI run failed, which was the point of adding it: 24 API tests were
reading ANTHROPIC_API_KEY out of a developer's .env without declaring it, so
they passed on my machine and could not construct their subject on a clean runner.
An autouse fixture now forces a dummy credential, which also guarantees no test can
reach a real provider by accident.
v1.0 is the tag on the measurement work above, and the limitations below are still the honest ones — they are documented rather than hidden, and what remains of that line of work is filed as issues and deliberately unbuilt, because each strengthens an existing claim rather than unlocking a new one.
Since that tag the agent has stopped being a thing that produces a diff on a laptop. Added after v1.0, and not covered by any number above:
- Public issue URL through to a real pull request — clone, solve, then a second gate showing the exact branch, title and body before it forks, pushes and opens the PR. Re-publishing updates that one PR instead of opening another.
- A deployed host that runs it unattended, deploys itself from
main, and has opened a pull request of its own. - A browser frontend, so the reasoning is legible to someone who is not the author — the whole point of the project being watchable rather than merely reported.
- Multi-tenant credentials and public-mode guardrails — a visitor's keys are scoped to their run, held in memory, and absent from every event that leaves the process; runs on the operator's key are capped daily and forced to open drafts.
The spend figure above is scoped to v1.0 deliberately: it is the cost of the experiments, and the work since has been engineering rather than measurement, so folding the two together would blur the one number the findings depend on.
- The ranker's weights are hand-picked and tuned on click. The gain generalises
in direction (rich improves ~3× on P@3 with no
src/directory to exploit) but not in magnitude (0.83 vs 0.50). They are unlearned, and the offline benchmark is how any change to them gets tested. - Better retrieval does not reduce cost (Finding 9). A run is ~$1.22 on click whether it succeeds or fails.
- A reporter-named file path is extracted and then ignored.
Query.pathsexists and nothing consumes it; that is the last remaining retrieval miss. - The ablation ladder is built but not fully run. At ~$1.22 per run, all six rungs across six cases is ~$44, which has not been spent.
- Phase 1B (embeddings, BM25, hybrid, reranking) is deliberately not built. Content-based ranking got P@3 to 0.83 without it, and the offline benchmark exists so the next strategy has to prove itself rather than be assumed better.
- The fork path has unit coverage but no live proof. Every pull request opened
end to end so far has been on a repository I own, where there is nothing to fork
and the branch goes straight there. Forking, then opening a cross-repo PR from
user:branch, is implemented and tested against a fakegh, but has not yet run against a repository belonging to someone else. - Run records live in memory. The graph checkpoints to Postgres, so a killed run resumes — but the record the browser polls does not survive an API restart, which is why there is no history page. A page that is sometimes blank is worse than none.