Skip to content

Commit 5e462dd

Browse files
committed
ci(e2e): run on push to main; drop the dead broker, Python, and run guard
- Runs on `push: [main]` in addition to PRs. The lane was PR-only, so a regression on main went unnoticed until someone opened the next PR. This matches the sibling SDKs, whose e2e jobs run on push+PR; none of them use a schedule, so cost tracks merge frequency rather than a clock, and the existing cancel-in-progress concurrency group collapses rapid pushes. - Pins CONDUCTOR_AGENT_LLM_MODEL=openai/gpt-4o-mini so CI spend cannot shift silently if BaseTest's default model changes. - Removes the mcp-testkit install/start steps and the Python setup that existed only for them. Suite4McpTools is plan()-only and asserts against a fabricated http://localhost:9999/mcp, so nothing ever connected to the broker on port 3001; `grep -rn '3001' e2e/` has no hits. This does not close the MCP gap — execution had no coverage before and still has none — it stops the job implying otherwise. - Removes the silently-empty run guard, which is now unreachable dead weight: with BaseTest failing hard instead of skipping, a green run that verified nothing cannot happen. - Corrects two false claims in the header. It said the suites never read the keys "(asserted by Suite2ToolCallingCredentials)" — that suite references neither key, and Suite7MediaTools does read OPENAI_API_KEY to gate a skip. And it said fork PRs "fail at the silently-empty guard"; they do not — /health ignores provider credentials, so the server boots, the suites run, and they fail on their LLM tasks at "Run e2e suites".
1 parent 2eb9940 commit 5e462dd

1 file changed

Lines changed: 33 additions & 45 deletions

File tree

‎.github/workflows/agent-e2e.yml‎

Lines changed: 33 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,32 @@ name: Agent E2E
55
# the agent runtime on by default from 3.32.0-rc.8 onward. A separate agent
66
# server JAR is no longer used here.
77
#
8-
# These tests call real LLMs via the OPENAI_API_KEY / ANTHROPIC_API_KEY
9-
# repo secrets. The suites themselves never read the keys (asserted by
10-
# Suite2ToolCallingCredentials) — only the server process gets them.
11-
# Fork PRs cannot see repo secrets, so for them the run fails at the
12-
# silently-empty guard rather than passing vacuously.
13-
14-
on: [pull_request, workflow_dispatch]
8+
# These tests call real LLMs via the OPENAI_API_KEY / ANTHROPIC_API_KEY repo
9+
# secrets. LLM and tool execution happens server-side, so the keys are consumed by
10+
# the server process. Only one suite reads a key at all: Suite7MediaTools gates its
11+
# live image-generation test on OPENAI_API_KEY.
12+
#
13+
# BaseTest.MODEL defaults to openai/gpt-4o-mini, so OPENAI_API_KEY is the one that
14+
# must be present; ANTHROPIC_API_KEY matters only when the model is overridden.
15+
#
16+
# Fork PRs cannot see repo secrets. The server still boots healthy — /health does not
17+
# check provider credentials — so the suites run and fail on their LLM tasks, and the
18+
# job fails at "Run e2e suites".
19+
#
20+
# There is no separate "did anything actually run?" guard: BaseTest fails hard rather
21+
# than assumeTrue-skipping when the server is unreachable, so a green run that
22+
# verified nothing is not reachable.
23+
24+
# Runs on push-to-main as well as PRs, so a regression on main is caught at merge
25+
# rather than waiting for someone to open the next PR. Matches the sibling SDKs,
26+
# whose e2e jobs run on push+PR; no workflow there uses a schedule, so cost tracks
27+
# merge frequency rather than a clock, and the concurrency group above collapses
28+
# rapid pushes.
29+
on:
30+
push:
31+
branches: [main]
32+
pull_request:
33+
workflow_dispatch:
1534

1635
permissions:
1736
contents: read
@@ -31,6 +50,9 @@ jobs:
3150
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
3251
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
3352
CONDUCTOR_SERVER_URL: http://localhost:8080/api
53+
# Pinned rather than left to BaseTest's default, so CI cost cannot shift silently
54+
# if that default changes.
55+
CONDUCTOR_AGENT_LLM_MODEL: openai/gpt-4o-mini
3456
steps:
3557
- name: Checkout code
3658
uses: actions/checkout@v4
@@ -41,13 +63,6 @@ jobs:
4163
distribution: temurin
4264
java-version: '21'
4365

44-
# Python is only needed for mcp-testkit and the XML guard.
45-
# No `cache: pip` — it hard-fails without a requirements file.
46-
- name: Set up Python
47-
uses: actions/setup-python@v5
48-
with:
49-
python-version: '3.12'
50-
5166
- name: Cache server JAR
5267
id: jar_cache
5368
uses: actions/cache@v4
@@ -61,15 +76,10 @@ jobs:
6176
curl -fL --retry 3 -o conductor-server.jar \
6277
"https://repo1.maven.org/maven2/org/conductoross/conductor-server/${CONDUCTOR_SERVER_VERSION}/conductor-server-${CONDUCTOR_SERVER_VERSION}-boot.jar"
6378
64-
- name: Install mcp-testkit
65-
run: |
66-
python -m pip install --upgrade pip
67-
pip install mcp-testkit
68-
69-
- name: Start mcp-testkit
70-
run: |
71-
mcp-testkit --transport http --port 3001 &
72-
sleep 2
79+
# No MCP broker is started: Suite4McpTools is plan()-only and asserts against a
80+
# fabricated http://localhost:9999/mcp, so nothing ever connected to the broker
81+
# this job used to run on port 3001. Wiring live MCP execution means adding one
82+
# back and pointing the suite at it.
7383

7484
- name: Start server
7585
run: |
@@ -90,28 +100,6 @@ jobs:
90100
- name: Run e2e suites
91101
run: ./gradlew :e2e:test -Pe2e
92102

93-
# BaseTest assumeTrue-skips every suite when the server is unreachable —
94-
# without this guard a boot failure after the health gate (or a future
95-
# gate regression) would yield a green job that ran nothing.
96-
- name: Guard against silently-empty runs
97-
if: always()
98-
run: |
99-
python - <<'EOF'
100-
import glob
101-
import sys
102-
import xml.etree.ElementTree as ET
103-
104-
total = executed = 0
105-
for path in glob.glob("e2e/build/test-results/test/TEST-*.xml"):
106-
root = ET.parse(path).getroot()
107-
t = int(root.get("tests", 0))
108-
sk = int(root.get("skipped", 0))
109-
total += t
110-
executed += t - sk
111-
print(f"executed {executed}/{total} tests")
112-
sys.exit(0 if executed > 0 else 1)
113-
EOF
114-
115103
- name: Upload results
116104
if: always()
117105
uses: actions/upload-artifact@v4

0 commit comments

Comments
 (0)