Windows CI times out (35m) when Kubo MFS files.rm hangs in community.delete() cleanup

[Rinse12]

Summary

Windows CI jobs (test-pkc-js-node-local (windows-latest)) intermittently hit the 35-minute workflow timeout and get cancelled. Root cause is Kubo's files.rm hanging in community.delete() cleanup, which brings down the whole test job (and effectively the in-process Kubo node interaction) until the workflow killer fires.

This appears to be an upstream Kubo bug, tracked as ipfs/kubo#10842 (MFS bug: 'ipfs files rm' hanging).

Most recent occurrence

• Workflow run: https://github.com/pkcprotocol/pkc-js/actions/runs/26329302325
• Failing job: test-pkc-js-node-local (windows-latest) (job 77512464910)
• PR: feat(challenges): {success, error, reason, comment?, commentUpdate?} result shape (#88) #99
• Job started 09:30:40, cancelled at 10:05:48 (exact 35-min timeout-minutes budget in .github/workflows/CI.yml)

Evidence

From the per-test stderr log for test/node/community/incoming.community.address.rejection.community.test.ts (test file is unrelated to PR #99 — pre-existing test that just happened to be the unlucky one this run):

• 09:49:13 — test starts
• 09:51:56 — community stop completes; afterAll calls community.delete()
• 09:53:56 — first MFS rm attempt times out:

PKCError: Timed out removing MFS paths. We may need to nuke the whole MFS directory and republish everything
Details:
  {
    \"toDeleteMfsPaths\": [\"/12D3KooW...\"],
    \"kuboRpcUrl\": \"http://localhost:15001/api/v0\"
  }
    at RetryOperation._fn (src/util.ts:954:34)
    at removeMfsFilesSafely (src/util.ts:936:12)
    at deleteCommunity (src/runtime/node/community/local-community/lifecycle.ts:458:35)

• removeMfsFilesSafely (in src/util.ts) retries 3 more times with pTimeout of 120s each plus exponential backoff → roughly 8 more minutes of hang per delete
• Workflow timeout hits before the test suite can move on

Why it's not the PR

• The test file is not touched on the PR branch and uses createSubWithNoChallenge, so the challenge-result-shape work doesn't intersect with it
• Same code path on master completes in ~26 minutes (see run 26288400419)
• Hang is on Windows-only Kubo MFS RPC, independent of pkc-js logic

Mitigation options

Pick one (or more) depending on appetite:

1. Wait on upstream — track MFS bug: ipfs files rm hanging ipfs/kubo#10842 and bump pinned Kubo version once a fix lands
2. Reduce retries in removeMfsFilesSafely — 4 attempts × 120s is a lot of dead time during teardown; dropping inputNumOfRetries (default 3) or milliseconds (currently 120000) for the cleanup path would surface the failure faster and let the rest of the suite continue
3. Increase the Windows job timeout-minutes — masks the symptom but won't fix the underlying hang
4. Fall back to nuking the whole MFS directory when the targeted rm fails enough times (the error message itself hints at this)

Upstream reference

• MFS bug: ipfs files rm hanging ipfs/kubo#10842 — MFS bug: ipfs files rm hanging

[Rinse12]

Investigation notes + suggested directions

I built a faithful repro of the pkc-js MFS lifecycle in mfs-timeout-repro/repro-pkc-js-pattern.js to figure out which set of files.* calls actually triggers the hang and where the best place to mitigate it is. Sharing what I learned so we can pick a direction.

What pkc-js does (the MFS command sequence)

Path layout: /{community.address}/postUpdates/{timestamp_bucket}/{commentCid}/update — 4 levels deep.

Phase	Calls
Sync (per batch of 50 comments, repeated)	pLimit(50) × files.write(path, { create:true, truncate:true, parents:true, flush:true }) → then files.flush(/{community}) (comment-updates.ts:264-292)
Stale-path cleanup	files.rm([oldPath1, oldPath2, …], { recursive:true, force:true }) (cleanup.ts:85)
Auto-nuke on sync timeout	files.rm("/" + community, { recursive:true, force:true }) (ipns-publishing.ts:161)
Delete	files.rm("/" + community, { recursive:true, force:true }) with pTimeout(120_000) × 3 retries (util.ts:946, lifecycle.ts:458)

Bisection results (Linux, kubo 0.41.0)

Scenario	Outcome
Faithful baseline (2 communities × 3k comments × pLimit50 × per-batch flush)	clean
Drop per-batch files.flush(/community)	clean
Drop flush:true on writes	kubo returns clean reached limit of 256 unflushed MFS operations error — not the hang
1 community × 20k comments × 20 buckets × 3 sync rounds + recursive rm	clean (rm finished in 6.4ms)
3 communities × 15k comments × 15 buckets × 4 sync rounds + stale-rm interleaved	clean (every delete < 14ms)
25k concurrent files.write (unrealistic — the original reproduce-mfs-timeout.js)	reliable hang in files.write; daemon stays alive, MFS subsystem dies

So pkc-js's pattern at realistic scale is robust on Linux. The Windows CI hang is upstream ipfs/kubo#10842 firing intermittently — pkc-js doesn't deterministically trigger it.

Note: the current auto-nuke fallback uses the same broken call

ipns-publishing.ts:161 calls files.rm("/" + community, recursive:true, force:true) to recover from a write timeout — but that is the same recursive-rm operation that hangs in delete. So when the bug fires, the "recovery" can wedge on the same bug.

Suggested directions to test

Each can be developed and validated against the repro before porting here. Listed in order of expected ROI:

1. Per-bucket rm in delete + auto-nuke (recommended first)

   • Replace the single files.rm("/" + community, recursive:true, force:true) with: for (const bucket of await ls(/community/postUpdates)) await rm(/community/postUpdates/{bucket}, recursive:true, force:true), then rm(/community) last. Each call carries its own pTimeout, so one stuck bucket doesn't block the whole job. kubo only has to walk a smaller subtree per HTTP request.
   • Touch points: removeMfsFilesSafely (util.ts:920) callsite at lifecycle.ts:458, and the auto-nuke at ipns-publishing.ts:161.

2. Rename-then-rm safety net

   • In deleteCommunity: files.mv(/{community}, /_trash_${address}_${Date.now()}) first, then rm the trash. The community is gone from the user/test's perspective in a single fast op; the trash rm can be best-effort or run in background or on next startup. Lets test teardown unblock even if the underlying rm hangs.

3. Reduce retries on the delete path

   • removeMfsFilesSafely defaults to 3 retries × 120s = 8 minutes per delete (util.ts:934, 953). On a 35-minute Windows job that's most of the budget for a single test. Pass inputNumOfRetries: 0 at the delete callsite (lifecycle.ts:458) so a single failure surfaces fast and the rest of the suite continues. This is the smallest, lowest-risk change and addresses the CI symptom directly even before the real fix lands.

Happy to draft a PR for any/all of these — say the word and I'll start with #1 + #3 (the high-impact pair) wired up behind a single deleteCommunity rewrite.