Repository navigation
Conversation
The launcher and the container-side bench commands each had an echo() helper, and only the launcher's masked *TOKEN* and *SECRET* values. The masking echo now lives in infx.bench.proc (stdlib-only, Python 3.10), which the launcher already imports; bench call() and tee() also mask the child environment's secrets. Two launcher warnings that went to stdout now go to stderr with the other diagnostics.
Debugging a failed benchmark job meant reading a multi-hour Actions log plus the server logs. Each `python -m infx.launch run` now writes one record, job_event.json, into the workspace when it exits on any path: the run, job and point identity, image, recipe, cluster, Slurm job and nodes; the launch decisions (path, DCGM power, golden acceptance length, eval flags, KV offloading); seconds per stage (prepare, submit, queue_wait, run, collect); the outcome; and for a failure the first error's stage, type, message, exit code, retriable hint and the log to read first. On GitHub Actions a failure also emits one ::error annotation titled with its stage. The benchmark templates upload the record as job_event_<RESULT_FILENAME>, and collect-results joins the records it can download into events.jsonl in the job_events artifact (python -m infx.results.collect_events). A batch-wrapped launch continues the record of the launch it re-runs. The golden acceptance planner now reports the length it simulates, and the Slurm accounting read includes the job's node list.
This was referenced Oct 8, 2026
Draft
Collect-results waits only for throughput jobs, so eval-only jobs were missing from events.jsonl and evals-only sweeps produced no job_events artifact. A dedicated collect-job-events template now runs after every sweep job in run-sweep.yml and e2e-tests.yml. Add the Chinese troubleshooting section.
…tions finish() reuses exited() for its exit-status fallback, the single-use PREPARE constant is inlined, and a cleanup's name is looked up only when its nonzero result is reported. The srt driver tests stop asserting environment values copied verbatim into the record, a non-negative stage duration check, and an annotation-absence check that could not fail without GITHUB_ACTIONS.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each benchmark job now writes one wide event,
job_event.json: a single JSON record of what happened to the job. The launcher fills it in as it runs and writes it in afinally, so it is written on success and on failure.outcome.error, with the stage, type, message, exit code, whether it is retriable, and a pointer to the most relevant log.job_event_*.python -m infx.results.collect_eventsconcatenates them intoevents.jsonl. A newcollect-job-events.ymljob runs after every benchmark and eval job inrun-sweep.ymlande2e-tests.ymland uploads it as artifactjob_events.::error title=<stage>::<type>: <message>line at its exit boundary. The batch re-entry doesn't print it.infx/launch: one maskingechohelper shared with bench, which stays stdlib-only, and warnings go to stderr instead of stdout.docs/troubleshooting.mddescribes the record and gives examplejqqueries.Verification
events.jsonlcollection and::error::escaping.Known gaps
job_eventsstays empty until this lands on main. Its steps arecontinue-on-error.job_events; the per-jobjob_event_*artifacts remain.