Skip to content

[AMD Support via SemiAnalysis][4/4] Preserve failure logs and record runtime versions - #14

Closed
cquil11 wants to merge 1 commit into
agent/atom-engine-atomeshfrom
review/runtime-portability
Closed

cquil11 wants to merge 1 commit into
agent/atom-engine-atomeshfrom
review/runtime-portability

Conversation

@cquil11

@cquil11 cquil11 commented Sep 15, 2026 •

Copy link
Copy Markdown

Changes

  • Replace invalid UTF-8 when reading worker failure logs, so one malformed byte cannot hide the log tail.
  • Record SGLang Router and MoRI package versions in runtime provenance.

The log-reading issue was identified during MI355X DeepSeek V4 SGLang validation: after a decode worker failed, strict UTF-8 decoding prevented printing its AITER log tail. This fixes failure reporting, not the underlying worker failure.

Upstream #476 and fork #14 contain the same changes. The fork PR sits on the branch-based AMD/MoRI/ATOM stack. Those three runtime layers have already merged upstream.

@cquil11
cquil11 force-pushed the review/runtime-portability branch from c4da2fd to 4e7730c Compare September 15, 2026 14:07
@cquil11 cquil11 changed the title Runtime portability: cluster deployment and Slurm job observation [AMD Support][2/4] Required shared runtime integration Sep 15, 2026
@cquil11
cquil11 changed the base branch from upstream/amd-runtime to agent/amd-multinode-runtime September 15, 2026 14:07
@cquil11
cquil11 added this pull request to stack #17 September 15, 2026 14:09
@cquil11
cquil11 force-pushed the review/runtime-portability branch from 4e7730c to 1b8e14a Compare September 16, 2026 01:11
@cquil11
cquil11 removed this pull request from stack #17 September 16, 2026 05:23
@cquil11
cquil11 force-pushed the review/runtime-portability branch from 1b8e14a to c3d17a7 Compare September 16, 2026 05:23
@cquil11
cquil11 changed the base branch from agent/amd-multinode-runtime to agent/atom-engine-atomesh September 16, 2026 05:23
@cquil11
cquil11 added this pull request to stack #18 September 16, 2026 05:23
@cquil11 cquil11 changed the title [AMD Support][2/4] Required shared runtime integration [AMD Support][4/4] Optional job observation and diagnostics Sep 16, 2026
@cquil11
cquil11 marked this pull request as ready for review September 17, 2026 18:41

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/srtctl/core/slurm.py Outdated
Comment thread src/srtctl/benchmarks/scripts/lm-eval/bench.sh Outdated
@cquil11
cquil11 force-pushed the review/runtime-portability branch 2 times, most recently from 3f2c787 to 4c9c948 Compare September 17, 2026 22:00
@cquil11
cquil11 force-pushed the review/runtime-portability branch from 5c8ca12 to 03863ab Compare September 18, 2026 21:36
@cquil11 cquil11 changed the title [AMD Support][4/4] Optional job observation and diagnostics feat: [AMD Support via SemiAnalysis][4/4] Optional job observation and diagnostics Sep 18, 2026
@cquil11
cquil11 removed this pull request from stack #18 September 22, 2026 14:38
@cquil11
cquil11 added this pull request to stack #25 September 22, 2026 14:38

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/srtctl/core/slurm.py Outdated
@cquil11
cquil11 force-pushed the review/runtime-portability branch from 049a6b5 to 7a7ca65 Compare September 22, 2026 18:13
@cquil11
cquil11 force-pushed the review/runtime-portability branch 2 times, most recently from caa16c6 to ded29b8 Compare September 23, 2026 16:11

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/srtctl/core/slurm.py Outdated
@cquil11
cquil11 force-pushed the review/runtime-portability branch from ded29b8 to aa39dd9 Compare September 23, 2026 16:24
@cquil11
cquil11 force-pushed the review/runtime-portability branch from aa39dd9 to caad901 Compare September 23, 2026 16:29
@cquil11
cquil11 force-pushed the review/runtime-portability branch from caad901 to dfc51ef Compare September 23, 2026 16:32
@cquil11
cquil11 force-pushed the review/runtime-portability branch from dfc51ef to 425cf2d Compare September 23, 2026 17:18

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 425cf2d. Configure here.

Comment thread src/srtctl/core/slurm.py Outdated
stream.seek(offset)
while chunk := stream.read(65536):
output.write(chunk)
offset = stream.tell()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log streaming reprints on each poll

Medium Severity

stream_log stores a text-mode tell() cookie in offset and later compares it to stat().st_size. That cookie is not a byte position, and progress-bar \r or a partial UTF-8 sequence at EOF makes it much larger than the file, so the next poll rewinds to the start and reprints the whole log. LogOutputStreamer already tracks a binary byte offset for this.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 425cf2d. Configure here.

@cquil11
cquil11 force-pushed the review/runtime-portability branch from 425cf2d to 379f11e Compare September 23, 2026 17:31
@cquil11 cquil11 changed the title feat: [AMD Support via SemiAnalysis][4/4] Optional job observation and diagnostics [AMD Support via SemiAnalysis][4/4] Optional runtime diagnostics Sep 23, 2026
@cquil11
cquil11 force-pushed the review/runtime-portability branch from 379f11e to e645b57 Compare September 23, 2026 17:32
@cquil11
cquil11 force-pushed the review/runtime-portability branch from e645b57 to e42ae4b Compare September 23, 2026 17:33
@cquil11 cquil11 changed the title [AMD Support via SemiAnalysis][4/4] Optional runtime diagnostics [AMD Support via SemiAnalysis][4/4] Preserve failure logs and record runtime versions Sep 23, 2026
@cquil11

cquil11 commented Oct 5, 2026

Copy link
Copy Markdown
Author

Closing because these failure-log and runtime-version changes were merged upstream in NVIDIA#476.

@cquil11 cquil11 closed this Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant