Hi Benchflow team,
I found an issue with bench eval create -s/--skills-dir for Docker-based ACP runs.
Problem
When --skills-dir is provided, Benchflow first bakes the skills into the Docker image through _env_setup.py:
# benchflow/_env_setup.py
shutil.copytree(skills_dir, dest, ...)
This makes the skills available inside the container at:
However, the agent itself runs with:
For Codex, the expected discovery path is:
/home/agent/.agents/skills
In the current flow, deploy_skills() in _agent_setup.py detects that the skills were already injected via Dockerfile and logs something like:
Skills already injected via Dockerfile
but it does not continue to link /skills into the agent home. As a result, the skills exist in the container, but Codex cannot discover them.
Local patch
I patched benchflow/_agent_setup.py around the Dockerfile-injected skills branch by setting:
effective_skills = "/skills"
Then the existing later linking logic correctly generates:
ln -sfn /skills /home/agent/.agents/skills
After this, Codex can load the skills passed via --skills-dir.
Verification
I verified the local patch with:
python -m py_compile /Users/wuxiaojun/code/skillsbench/.venv/lib/python3.12/site-packages/benchflow/_agent_setup.py
uv run bench tasks check tasks/task-hybrid-electric-vehicles-driving
Output:
✓ task-hybrid-electric-vehicles-driving — valid
I also ran a minimal reproduction: before the patch, the assertion failed because no /skills -> /home/agent/.agents/skills link command was generated; after the patch, the expected link command appears.
Expected behavior
When skills are injected into the image at /skills, Benchflow should still link that path into the agent-specific skill discovery directories, such as:
/home/agent/.agents/skills
/home/agent/.claude/skills
depending on the agent.
This would make --skills-dir work consistently for agents whose skill discovery depends on paths under their runtime $HOME.
Hi Benchflow team,
I found an issue with
bench eval create -s/--skills-dirfor Docker-based ACP runs.Problem
When
--skills-diris provided, Benchflow first bakes the skills into the Docker image through_env_setup.py:This makes the skills available inside the container at:
However, the agent itself runs with:
For Codex, the expected discovery path is:
In the current flow,
deploy_skills()in_agent_setup.pydetects that the skills were already injected via Dockerfile and logs something like:but it does not continue to link
/skillsinto the agent home. As a result, the skills exist in the container, but Codex cannot discover them.Local patch
I patched
benchflow/_agent_setup.pyaround the Dockerfile-injected skills branch by setting:Then the existing later linking logic correctly generates:
After this, Codex can load the skills passed via
--skills-dir.Verification
I verified the local patch with:
Output:
I also ran a minimal reproduction: before the patch, the assertion failed because no
/skills -> /home/agent/.agents/skillslink command was generated; after the patch, the expected link command appears.Expected behavior
When skills are injected into the image at
/skills, Benchflow should still link that path into the agent-specific skill discovery directories, such as:depending on the agent.
This would make
--skills-dirwork consistently for agents whose skill discovery depends on paths under their runtime$HOME.