[Bug]: Large repository clones time out after 5 minutes and restart from scratch

[hannesrudolph]

Preflight

• I searched existing issues and docs before opening this.
• I removed secrets, tokens, private keys, customer data, and private repository details.
• This report describes observed behavior, not a speculative rewrite request.

Summary

A worker cannot prepare a repository whose full Git clone takes longer than five minutes. The clone command is killed at the fixed timeout, the partial checkout is deleted, and the next retry repeats the full transfer from zero. The task remains in preparing and will eventually fail after all retries are exhausted.

Affected surface

• Roomote agent task
• Worker or controller
• Setup, deployment, or upgrade

Steps to reproduce

1. Run Roomote from a source checkout with Docker compute enabled.
2. Create a task that prepares a sufficiently large repository from GitHub.
3. Observe worker setup while the initial full clone needs more than five minutes.

Expected behavior

Repository preparation should either complete or fail with actionable diagnostics. Self-hosted deployments should have a way to accommodate large repositories without restarting the same full clone repeatedly.

Actual behavior

WorkspaceManager executes a full git clone with timeout: 300 and retries: 4:

run: `rm -rf -- '${shellEscape(fullName)}' && git clone '${shellEscape(cloneUrl)}' '${shellEscape(fullName)}'`,
retries: 4,
timeout: 300,

The retry loop deletes the partial target before every attempt. In one observed run, the worker logged failures exactly five minutes apart:

2026-07-31T05:19:03Z Cloning <redacted>
2026-07-31T05:24:03Z [Git clone] Attempt 1/5 failed. Retrying in 1000ms...
2026-07-31T05:29:04Z [Git clone] Attempt 2/5 failed. Retrying in 1000ms...
2026-07-31T05:34:05Z [Git clone] Attempt 3/5 failed. Retrying in 1000ms...

The worker was actively downloading and indexing the pack and had sufficient disk space, but no attempt could reach completion before the timeout. The standard retry log did not retain the underlying Git stderr, so the UI and logs do not make the cutoff obvious.

Version

main@327961f

Installation method

Source checkout / development

Deployment context

Self-hosted Docker compute, with the worker running a Linux ARM64 image.

Diagnostics

Relevant code:

• apps/worker/src/workspace/workspace-manager.ts (Git clone command, timeout, and partial-target cleanup)
• apps/worker/src/command-executor/command-executor.ts (generic retry logging)

Notes

Potential fixes:

• Make the clone timeout configurable or adaptive for self-hosted workers.
• Preserve or resume a partial clone where safe instead of deleting it before every retry.
• Consider a shallow or partial clone option for initial workspace setup.
• Include the failed Git command's sanitized stderr and timeout cause in task diagnostics.

[hannesrudolph]

@roomote fix it

[roomote-roomote]

I saw the mention, but I need your GitHub account linked to Roomote before issue and PR comments can start work here. Link it from Settings -> Linked Accounts and then mention me again.

[roomote-roomote]

I've analyzed this issue and here's my proposed implementation plan:

1. Add a validated deployment setting such as WORKER_REPOSITORY_CLONE_TIMEOUT_SECONDS in packages/env/src/index.ts, defaulting to the existing 10-minute command timeout, and forward it through the shared worker environment builder in packages/compute-providers/src/worker-env/base.ts so it works consistently for Docker and hosted compute providers.
2. Capture that value as worker-internal configuration in apps/worker/src/env/worker-env.ts and pass it through repository setup into WorkspaceManager, without exposing the deployment setting as a user-project environment variable.
3. Replace the initial clone's generic retries: 4 behavior in apps/worker/src/workspace/workspace-manager.ts with a clone-specific retry loop, following the existing fetch-retry pattern. Keep cleanup and retries for transient failures, but fail immediately when Execa reports that the configured clone deadline expired. This prevents five identical full transfers from restarting after a deterministic timeout while preserving the existing auth/visibility propagation retries.
4. Preserve full-clone semantics rather than switching to shallow or partial clones. git clone does not provide a safe general-purpose resume contract for an interrupted destination, while shallow/partial clones can break later branch, tag, and history operations.
5. Extend ExecutionResult in apps/worker/src/command-executor/command-executor.ts to retain timeout classification, and make clone failure/retry logs state the configured deadline and a sanitized failure reason. The final structured repository-preparation diagnostics should continue to include bounded stderr, but clearly identify timeout as the cause.
6. Add coverage in workspace-manager-clone.test.ts, command-executor.test.ts, worker-env tests, compute-provider worker-env tests, and env-schema tests for the default/override path, fail-fast timeout behavior, transient retries, cleanup, and actionable diagnostics. Document the new operator setting in apps/docs/environment-variables.mdx and the Docker troubleshooting guidance.

This approach lets self-hosted operators raise the deadline for genuinely large repositories, improves the default from five to ten minutes, and avoids repeatedly discarding an active multi-minute transfer when the configured deadline itself is the reason it failed, without changing repository history semantics.

Please tag @roomote if you'd like me to implement this, or reply with feedback on the plan.

[mrubens]

@roomote listen to the man

[roomote-roomote]

I'm on it. I started a task for this issue, and I'll keep updates here.

See task

[roomote-roomote]

Implemented the proposed fix in draft PR #934: #934

It adds a configurable 10-minute clone deadline, fails fast on deterministic timeouts, preserves transient retries with cleanup/backoff, and improves sanitized timeout diagnostics. Targeted tests, typechecks, lint, docs validation, and pre-push checks pass.