Skip to content

fix(test): prevent custom reporter exit-code wraparound - #2497

Merged
thymikee merged 1 commit into
callstack:mainfrom
PLASMA-FR:fix/reporter-exit-code-range-b-20260911-sx1vuy2g
Sep 11, 2026
Merged

thymikee merged 1 commit into
callstack:mainfrom
PLASMA-FR:fix/reporter-exit-code-range-b-20260911-sx1vuy2g

Conversation

@PLASMA-FR

Copy link
Copy Markdown
Contributor

Summary

Completes the reporter exit-code guarantee from #959, characterized in #1505: a custom reporter must not mask a failing suite. On current main, a reporter returning 256 or 512 makes this command exit successfully on Linux even when a test fails:

agent-device test ./workflows --reporter ./reporter.mjs

Validate getExitCode results as integers from 0 to 255, or undefined, before aggregation and JSON success output. Invalid values fail with INVALID_ARGS; valid reporters retain the highest-code rule and failed-suite floor. Eight files, including regression tests, public type documentation, help, user docs, and changelog.

Validation

Tested a745e45c6aa5d05afdb0e4659608ef6e93c40193 using Node 24.13.1, pnpm 11.17.0, and AGENT_DEVICE_VITEST_MAX_WORKERS=1.

  • Eleven regression cases fail before the fix. The JSON case also fails with only the range check applied.
  • Build, 138 focused reporter/CLI/command/startup tests, and 12 command-doc tests pass.
  • Built CLI against an owned local HTTP fixture: all 20 text/JSON cases pass, covering successful/failed suites and codes 0, 3, 255, 256, and 512. Upstream gives eight incorrect exit statuses.
  • pnpm check:affected --run passes all runnable checks: 2,805 tests across 383 files, formatting, lint, types, layering, Fallow, and build.

No device run: the changed boundary is local CLI reporting. Upstream CI is pending.

@thymikee

Copy link
Copy Markdown
Member

No findings at a745e45. Reporter codes are validated before aggregation and JSON success output, and the CLI regression covers the false-success case. Ready for human review. The remaining iOS smoke failure is an alert-replacement XCTest assertion and appears unrelated to this CLI-only change.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 11, 2026
@thymikee
thymikee merged commit bda6d42 into callstack:main Sep 11, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants