Clean up eval config

Author: thibaudcolas

docs/evals/README.md

@@ -1,25 +1,18 @@
 # Skill evals
 
-Promptfoo evals for the two agent skills in `src/wagtail_cli/.agents/skills/`, run through the OpenCode SDK provider on TensorX with `z-ai/glm-5.3-flash` as the default model under test.
+Evaluation suites for the agent skills that are [bundled with the CLI](../agent-skills.md). Running with Promptfoo, with models taken from [Wagtail agentic engineering recommendations](https://wagtail.org/ai/agentic-engineering-recommendations/).
 
-Each suite (`wagtail_api_skill.yaml`, `wagtail_docs_skill.yaml`) runs every test against two arms that differ by exactly one tool:
+For each skill, we run two suites: a baseline with nothing loaded, and one with our skills. We test:
 
-| Arm | Tools | Question answered |
-| --- | --- | --- |
-| baseline | none | What can the model do from its own knowledge? |
-| skill | `skill`, allowed to load only that suite's skill | Does loading the skill help? |
-
-- Skill activation: rows restricted to the skill arm assert with `skill-used` that a fundamental task phrased the way a user would (no CLI named) actually loads the skill, and that an unrelated task does not.
-- Task completion: both arms are asked for the exact terminal commands, and a grader runs them. `graders/run_api_commands.py` injects `--dry-run --json` into every `wt … api` invocation and runs the block against dummy credentials, then grades the exact requests (method, URL, params, body) — a hallucinated flag exits 2 and matches nothing, and no live Wagtail site is needed. `graders/run_docs_commands.py` runs `wt docs` commands as-is and checks the retrieved content.
-- Gotchas with no command answer are graded by rubric, using a second model (`deepseek/deepseek-v4.1-flash`) from a different family so the suite is not grading itself.
-
-Caveats: dry-run grading proves the CLI accepts the commands and builds the right request, not that a live site would accept the payload; the `--dry-run` rewrite drops `VAR=$(wt …)` capture, so prompts supply concrete inputs rather than values discovered from earlier commands. The `read`, `grep`, `glob` and `list` tools are disabled explicitly on both arms: the promptfoo opencode:sdk provider turns them on by default whenever `working_dir` is set, even with `'*': false`, which let the baseline read the repo (including the eval configs, the skills and this README) and answer from there instead of from its own knowledge. The skill arm gets `read` back, scoped with an OpenCode permission rule to `src/wagtail_cli/.agents/skills/**`, so it can follow SKILL.md's links into `references/`; `grep` and `glob` stay off everywhere because their permission rules match the search pattern rather than the file path and so cannot be scoped the same way. This makes the arms differ by the skill tool plus a references-only read, not a single tool — measured, the alternative (no read at all) cost the skill arm its command syntax, which lives in `references/commands.md`. The repo's `AGENTS.md` is still injected into the context of both arms by OpenCode; it describes the project but not the CLI, so it does not teach the baseline any commands.
+- Skill activation: whether the skill activates when mentioning potential related tasks, without necessarily directly prompting for "use Wagtail CLI".
+- Task completion: whether the CLI commands provided by the agent actually exist, and work as explained, and help in completing the task.
+- Gotchas with no command answer. Graded by rubric, using a separate model family.
 
 ## Requirements
 
-- promptfoo 0.123.1 or newer (the `skill-used` assertion needs the session-history skill detection missing in 0.122) and `@opencode-ai/sdk` — `just eval-init` installs both.
-- The OpenCode CLI, from https://opencode.ai/docs/.
-- `wt` on PATH — the graders run the CLI the model names.
+- Promptfoo with the OpenCode AI SDK (`just eval-init`)
+- [OpenCode CLI](https://opencode.ai/docs/)
+- `wt` on PATH. The graders run the CLI from there.
 - `TENSORX_API_KEY` — the model under test and the rubric grader both run on TensorX.
 
 ## Running
@@ -31,10 +24,10 @@ just eval --repeat 3                            # agent runs are noisy; repeat b
 just eval-view                                  # dashboard for the latest run
 ```
 
-`WT_EVAL_MODEL` picks the model under test; it must be registered in `opencode.json` (default `z-ai/glm-5.3-flash`):
+`EVAL_MODEL` picks the model under test; it must be registered in `opencode.json` (default `z-ai/glm-5.3-flash`):
 
 ```sh
-WT_EVAL_MODEL=qwen/qwen3.8-27b just eval
+EVAL_MODEL=qwen/qwen3.8-27b just eval
 ```
 
 ## Results snapshot

docs/evals/opencode.json

@@ -3,22 +3,31 @@
   "provider": {
     "tensorx": {
       "models": {
-        "z-ai/glm-5.3-flash": {
-          "name": "GLM 5.3 Flash"
-        },
-        "qwen/qwen3.8-27b": {
-          "name": "Qwen 3.8 27B"
-        },
-        "qwen/qwen3.5-9b": {
-          "name": "Qwen 3.5 9B"
-        }
+        "z-ai/glm-5.3-flash": { "name": "GLM 5.3 Flash" },
+        "qwen/qwen3.8-27b": { "name": "Qwen 3.8 27B" },
+        "qwen/qwen3.5-9b": { "name": "Qwen 3.5 9B" }
       },
       "name": "TensorX",
       "npm": "@ai-sdk/openai-compatible",
       "options": {
         "apiKey": "{env:TENSORX_API_KEY}",
         "baseURL": "https://api.tensorx.ai/v1"
       }
+    },
+    "neuralwatt": {
+      "name": "Neuralwatt",
+      "npm": "@ai-sdk/openai-compatible",
+      "options": {
+        "apiKey": "{env:NEURALWATT_API_KEY}",
+        "baseURL": "https://api.neuralwatt.com/v1"
+      },
+
+      "models": {
+        "deepseek-v4.1-flash": { "name": "Deepseek V4.1 Flash" },
+        "glm-5.3-flash": { "name": "GLM 5.3 Flash" },
+        "glm-5.3": { "name": "GLM 5.3" },
+        "qwen-3.8-27b": { "name": "Qwen 3.8 27B" }
+      }
     }
   },
   "skills": {

docs/evals/wagtail_api_skill.yaml

@@ -21,9 +21,8 @@ providers:
   - id: opencode:sdk
     label: baseline
     config: &opencode
-      provider_id: tensorx
-      model: '{{ env.WT_EVAL_MODEL | default("z-ai/glm-5.3-flash", true) }}'
-      apiKey: '{{ env.TENSORX_API_KEY }}'
+      provider_id: '{{ env.EVAL_PROVIDER | default("tensorx", true) }}'
+      model: '{{ env.EVAL_MODEL | default("z-ai/glm-5.3-flash", true) }}'
       working_dir: ../..
       tools:
         '*': false

docs/evals/wagtail_docs_skill.yaml

@@ -10,9 +10,8 @@ providers:
   - id: opencode:sdk
     label: baseline
     config: &opencode
-      provider_id: tensorx
-      model: '{{ env.WT_EVAL_MODEL | default("z-ai/glm-5.3-flash", true) }}'
-      apiKey: '{{ env.TENSORX_API_KEY }}'
+      provider_id: '{{ env.EVAL_PROVIDER | default("tensorx", true) }}'
+      model: '{{ env.EVAL_MODEL | default("z-ai/glm-5.3-flash", true) }}'
       working_dir: ../..
       # The arms differ in the `skill` tool and permission. Note: with
       # `working_dir` set, the promptfoo opencode:sdk provider turns on the

justfile

@@ -87,7 +87,7 @@ eval *args="":
         shift
     fi
     for config in ${configs}; do
-        OPENCODE_CONFIG="$PWD/docs/evals/opencode.json" promptfoo eval -c "${config}" --no-cache "$@"
+        OPENCODE_CONFIG="$PWD/docs/evals/opencode.json" promptfoo eval -c "${config}" "$@"
     done
 
 # Open the promptfoo viewer for the most recent eval results.