Skip to content

fix: use named temp file for writability probe to avoid EIO on FUSE/NAS - #1774

Merged
SAKURA-CAT merged 5 commits into
mainfrom
fix/temp-file-create
Sep 21, 2026
Merged

SAKURA-CAT merged 5 commits into
mainfrom
fix/temp-file-create

Conversation

@Nexisato

@Nexisato Nexisato commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Description

修复 safe_mkdir 可写性探针在 FUSE/NAS 文件系统上误报 EIO 导致训练崩溃的问题。

根因: 探针使用 tempfile.TemporaryFile,其依赖匿名临时文件(O_TMPFILE)语义。部分 FUSE/NAS 实现不支持该语义,且返回的是 EIO 而非可触发 fallback 的错误码,Python 不会降级处理。探针把这个硬错误当作「NAS 同步延迟」不断重试,最终撞 5 秒超时抛 TimeoutError,训练进程挂掉。而普通命名文件写入在这类存储上完全正常。

改动:

  • swanlab/sdk/internal/pkg/fs/dir.py
    • 探针由 tempfile.TemporaryFile 改为命名临时文件(mkstemp → 写入 → 关闭 → 删除),新增 _probe_writable()
    • TimeoutError 通过 raise ... from last_error 链上最后一次 OSError,避免硬错误被误报为单纯超时
  • 测试:更新/新增命名文件探针、残留清理、异常链相关用例

Releated

@Nexisato Nexisato added the 🐛 bug Something isn't working label Sep 14, 2026
@Nexisato Nexisato changed the title fix/temp file create fix: use named temp file for writability probe to avoid EIO on FUSE/NAS Sep 14, 2026
@Nexisato Nexisato self-assigned this Sep 14, 2026
Comment thread swanlab/sdk/internal/pkg/fs/dir.py Outdated
@SAKURA-CAT
SAKURA-CAT merged commit 78101f8 into main Sep 21, 2026
20 checks passed
@SAKURA-CAT
SAKURA-CAT deleted the fix/temp-file-create branch September 21, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🐛 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants