agentfoo

A vitest-like unit test framework for agent skills. You write tests in the vitest DSL you already know; agentfoo adds the pieces that make testing an agent practical: booting the agent in a container, detecting when a skill was actually pulled in, and grading free-form output with an LLM judge.

Scope (v0.1): agentfoo tests multiple coding agents — hermes, pi, and openclaw through the acpx ACP client, plus opencode natively — and routes the judge / agent inference through several providers (anthropic, deepseek, GLM, MiniMax, Kimi). The hermes (0.18.2 / acpx 0.12.1), opencode (1.18.5) and pi (0.73.1 / pi-acp 0.0.32) adapters are each exercised end-to-end against a real container, as is the judge. openclaw (2026.7.1-2) is wired and probed against a real binary — config schema, gateway startup, trace envelope and skill detection all verified — and the suite runs green against it end-to-end, though its Gateway daemon needs ~850MB RSS, so budget ~1.5GB free on the host. Treat the API as pre-1.0 and subject to change.

What you get

• Native vitest DSL — test / expect / describe / fixtures, re-exported from agentfoo. Importing the package registers the custom matchers as a side effect.
• Real agent runs in Docker — each spec file boots an agent container (bootAgent('hermes' | 'opencode' | 'pi' | 'openclaw')), runs real model turns, and tears down. A --local escape hatch runs against a host binary for fast dev iteration.
• Skill-invocation spies — a SkillHandle is both "load this skill" and the spy target: expect(skill).toHaveBeenCalled().
• LLM-judge assertions — await expect(trace).toSatisfy(rubric, { threshold }) grades open-ended output against a weighted rubric.
• Opt-in retries — retry(fn, { attempts, policy }) for flaky live runs, explicitly per-block (never silently applied to every slow test).
• Artifacts on disk — every run writes traces and session dumps under .agentfoo/runs/<id>/ for debugging.

Requirements

• Node.js ≥ 18.19
• Docker (for the default containerized runtime; not needed for --local)
• An LLM API key for live runs (judge model + the agent's own model)

Install

npm install --save-dev agentfoo vitest

Quick start

1. agentfoo.config.ts — a thin wrapper over vitest config:

import { resolve } from 'node:path'
import { defineConfig } from 'agentfoo/config'

export default defineConfig({
  judge: { model: 'deepseek/deepseek-v4-pro' },
  agents: {
    hermes: {
      model: 'deepseek-v4-pro',
      provider: 'deepseek',
      baseUrl: 'https://api.deepseek.com',
      dockerfile: resolve(import.meta.dirname, './dockers/hermes.Dockerfile'),
      passEnv: ['DEEPSEEK_API_KEY'], // forwarded into the container at run time
    },
  },
  setupFiles: ['./test/fixtures.ts'],
})

2. Shared fixtures (test/fixtures.ts) — one container per spec file:

import { fileURLToPath } from 'node:url'
import { test as base, bootAgent } from 'agentfoo'
import type { Agent, SkillHandle } from 'agentfoo'

interface Fixtures { hermes: Agent; frontendDesign: SkillHandle }

export const test = base.extend<Fixtures>({
  hermes: [
    async ({}, use) => {
      const agent = await bootAgent('hermes')
      await use(agent)
      await agent.teardown()
    },
    { scope: 'file' },
  ],
  frontendDesign: async ({ hermes }, use) => {
    await use(await hermes.loadSkill(
      fileURLToPath(new URL('../skills/frontend-design', import.meta.url)),
    ))
  },
})

export { expect } from 'agentfoo'

3. A spec (skills/frontend-design/frontend-design.spec.ts):

import { test, expect } from '../../test/fixtures'

test('triggers on a UI design request', async ({ hermes, frontendDesign }) => {
  const trace = await hermes.run('Design a landing page for a coffee brand.')

  expect(frontendDesign).toHaveBeenCalled()

  await expect(trace).toSatisfy(
    [
      { criteria: 'Makes deliberate typography/color/layout choices', weight: 2 },
      { criteria: 'Choices relate to the coffee-brand theme', weight: 2 },
      { criteria: 'Mentions accessibility or responsiveness', weight: 1 },
    ],
    { threshold: 0.7 },
  )
})

Running

agentfoo run                      # run all specs (Docker runtime)
agentfoo run skills/frontend      # path filter
agentfoo run -t "should trigger"  # name filter (passed through to vitest)
agentfoo run -a opencode          # run the same specs against another agent
agentfoo run -a pi                # …or a third
agentfoo run --local              # use a host agent binary, skip Docker
agentfoo list                     # collect specs only — zero-cost wiring check
agentfoo run --env-file ../.env   # load an explicit .env (e.g. a sibling repo's)
agentfoo watch                    # watch mode

The CLI loads the nearest .env (provider keys / model wiring), auto-discovers agentfoo.config.ts, and forwards everything else to vitest. agentfoo list collects and prints the matched specs without booting a container or calling a model — the cheapest way to confirm config, fixtures, and specs all resolve. Use --env-file <path> when your suite lives in a subtree that can't reach the .env by walking up from the cwd.

Choosing the agent

-a <kind> (--agent) picks which agent from agents the suite runs against, so one set of specs can be pointed at several agents. Fixtures opt in by calling bootAgent() with no kind:

agent: [
  async ({}, use) => {
    const agent = await bootAgent()   // ← -a decides; no kind hard-coded
    await use(agent)
    await agent.teardown()
  },
  { scope: 'file' },
],

With no -a, bootAgent() uses the sole configured agent. If several are configured and none is selected it throws rather than guessing — running a suite against the wrong agent shows up as a skill that "stopped firing", which is an expensive thing to debug. Passing a kind explicitly (bootAgent('hermes')) pins the fixture and ignores -a, which is right for a spec that is genuinely about one agent's behaviour.

Note: skill-invocation detection is still per-worker global (setSkillDetector), and the right signal differs per agent — hermes names the skill only in its reasoning, opencode fires a real skill({name}) tool call. So retargeting a suite with -a today also means revisiting the detector.

Through an npm script the flag must come after --, because npm consumes -a itself and would forward a bare opencode that agentfoo then reads as a path filter:

npm run example -- -a opencode

Configuration

defineConfig accepts vitest's UserConfig plus:

Field	Meaning
judge.model	Model used by toSatisfy to grade output (provider/model)
judge.maxTokens	Output-token cap for one grading call (default 32768)
agents.<name>	Per-agent config: model, provider, baseUrl, dockerfile/image, passEnv, memory
retries	Default attempt count consumed by retry() (not wired into vitest's global retry)

judge.maxTokens caps the judge's output, not the transcript you feed it — a large trace costs prompt tokens, not this budget. The default is generous because reasoning judges (deepseek-reasoner, deepseek-v4-pro, …) bill their hidden chain-of-thought against the same budget as the verdict JSON, and too small a cap makes the grading call come back truncated or empty. Unused budget isn't billed. Lower it if your judge model caps output below the default and rejects it, or if the provider requires prompt + max_tokens to fit the context window and your transcripts are very large.

agents.<name>.model accepts the same provider/model prefix as judge.model (e.g. deepseek/deepseek-v4-pro); the prefix populates provider unless you set it explicitly. If you set neither dockerfile nor image, the Docker runtime falls back to the Dockerfile bundled for that agent kind (dockers/<kind>.Dockerfile), so the default containerized runtime works with zero image configuration.

Agents

bootAgent(kind) selects the coding agent under test. Each kind has a bundled reference Dockerfile under dockers/:

Kind	Adapter	CLI driven	Bundled image
hermes	AcpxAgent	acpx --agent 'hermes acp' (ACP)	dockers/hermes.Dockerfile
opencode	OpencodeAgent	opencode run (native)	dockers/opencode.Dockerfile
pi	AcpxAgent	acpx pi (ACP)	dockers/pi.Dockerfile
openclaw	AcpxAgent	acpx openclaw (ACP)	dockers/openclaw.Dockerfile

hermes, pi, and openclaw share one adapter (AcpxAgent) that shells out to the acpx headless ACP client, so the same code path reaches every ACP agent — pi / openclaw by name, hermes via the acpx --agent 'hermes acp' escape hatch. What differs is only how each one is configured, since acpx's generic --model has nowhere to put a base_url: the adapter writes each agent's own config file into its isolated home before the first run — config.yaml for hermes, models.json for pi, openclaw.json for openclaw. The API key is forwarded via passEnv and never touches disk; those files reference it by env-var name rather than embedding it.

Per-test isolation is cwd-scoped. acpx keys a saved session to the working directory, so each test boundary runs sessions new for the instance's workspace to start a fresh conversation — no named-session flag, which is what lets the one adapter also drive hermes through --agent (where -s is rejected). Multi-turn continues by reusing the same cwd session. The per-turn acpx --format json output is the ACP session/update stream the adapter parses into a trace.

Only opencode keeps a native adapter.

Note that acpx <name> is often not self-contained, in two different ways. For pi it shells out to a separate pi-acp adapter package, which the bundled Dockerfile pre-installs pinned so a test run never fetches it from npm mid-flight. For openclaw it is only a bridge: the real agent is a long-running Gateway daemon on 127.0.0.1:18789 that nothing starts implicitly, so the adapter launches it during init and waits for the port.

openclaw is memory-hungry. That Gateway reaches ~850MB RSS during a single turn even with plugins disabled, and capping V8's heap does not bound it. On a host without ~1.5GB free the OOM killer takes it mid-turn, which surfaces as Gateway disconnected: 1006 / agent needs reconnect — mentioning neither memory nor the daemon. The adapter appends a gateway liveness check and log tail to any acpx failure so this is legible from the first failed run.

Providers

The judge (judge.model) and each agent's inference (agents.<name>.provider) route through a shared registry in src/providers.ts. Naming a known provider auto-fills its endpoint and conventional API-key env var, so provider: 'glm' (or a glm/… model prefix) needs no hand-written baseUrl:

Provider	Prefix	Key env var(s)	Dialect
Anthropic	anthropic/	ANTHROPIC_API_KEY	Messages API
DeepSeek	deepseek/	DEEPSEEK_API_KEY	OpenAI-compatible
GLM (Zhipu / z.ai)	glm/	ZHIPUAI_API_KEY / GLM_API_KEY	OpenAI-compatible
MiniMax	minimax/	MINIMAX_API_KEY	OpenAI-compatible
Kimi (Moonshot)	kimi/	MOONSHOT_API_KEY / KIMI_API_KEY	OpenAI-compatible

Adding a provider is one entry in src/providers.ts. judge.baseUrl / judge.apiKeyEnv (or the per-agent baseUrl / passEnv) override the defaults for a custom gateway.

Custom skill-invocation detection

Which tool call means "a skill fired" is agent- and version-specific, so the built-in heuristic behind toHaveBeenCalled is a best guess. Once you've seen your agent's real traces, pin the signal exactly from a setupFiles module:

import { setSkillDetector } from 'agentfoo'

setSkillDetector((trace, skillName) =>
  trace.toolCalls.filter((c) => c.name === 'skill_view' && c.arguments.name === skillName),
)

When toHaveBeenCalled fails, the error lists the tool calls that were observed, so you can tell "the skill never fired" from "it fired but the detector didn't recognize the signal."

License

MIT © tangxinyao