This is a follow-up to my #4472. That change fixed the recurring clcache [WinError 5] failures, but it intentionally left %USERPROFILE%\clcache behind when the active cache moved to C:\clcache_dir.
The old directory is no longer managed by clcache, while the new cache has its own 20 GB limit. The Windows workers already had low disk space before #4472, so this is not necessarily the only cause. However, in equal 59-hour windows around the merge, the fleet's low-disk alert time increased from 16.6% to 49.4%; x64-2 later reached 100%, and the reliability reports recorded disk-space failures on September 19 and September 20.
Before changing the configuration again, could someone with worker access check one machine first?
- Drain the worker and record
fsutil volume diskfree C:.
- Record the logical sizes of
C:\clcache_dir, %USERPROFILE%\clcache, C:\ccache\cache, and C:\workspace. Hardlinks mean the total may double-count physical blocks.
- Confirm
C:\clcache_dir\config.txt exists and capture clcache -s.
- Remove the old
%USERPROFILE%\clcache directory as a one-off operation, then measure physical free space again.
- If that recovers useful space, repeat one worker at a time.
The active 20 GB limit also needs checking. clcache starts cleanup at the limit and targets about 18 GB, but it is not aware of free disk space or the other caches and workspaces on C:. A fresh 18–20 GB cache may itself be too large on workers that are already above 90% usage. The size and hit-rate measurements should tell us whether a smaller limit is warranted.
I cannot collect these values myself because my account has no Jenkins Overall/Read permission. If removing the old cache is insufficient, the existing git-clean-windows job is the next immediate step.
Related: #4442, #3897.
AI disclosure: OpenAI Codex assisted with the investigation, analysis, and drafting. I reviewed and revised the conclusions and final text.
This is a follow-up to my #4472. That change fixed the recurring clcache
[WinError 5]failures, but it intentionally left%USERPROFILE%\clcachebehind when the active cache moved toC:\clcache_dir.The old directory is no longer managed by clcache, while the new cache has its own 20 GB limit. The Windows workers already had low disk space before #4472, so this is not necessarily the only cause. However, in equal 59-hour windows around the merge, the fleet's low-disk alert time increased from 16.6% to 49.4%; x64-2 later reached 100%, and the reliability reports recorded disk-space failures on September 19 and September 20.
Before changing the configuration again, could someone with worker access check one machine first?
fsutil volume diskfree C:.C:\clcache_dir,%USERPROFILE%\clcache,C:\ccache\cache, andC:\workspace. Hardlinks mean the total may double-count physical blocks.C:\clcache_dir\config.txtexists and captureclcache -s.%USERPROFILE%\clcachedirectory as a one-off operation, then measure physical free space again.The active 20 GB limit also needs checking. clcache starts cleanup at the limit and targets about 18 GB, but it is not aware of free disk space or the other caches and workspaces on C:. A fresh 18–20 GB cache may itself be too large on workers that are already above 90% usage. The size and hit-rate measurements should tell us whether a smaller limit is warranted.
I cannot collect these values myself because my account has no Jenkins Overall/Read permission. If removing the old cache is insufficient, the existing
git-clean-windowsjob is the next immediate step.Related: #4442, #3897.
AI disclosure: OpenAI Codex assisted with the investigation, analysis, and drafting. I reviewed and revised the conclusions and final text.