feat(server): remove the 50 MiB upload size cap and stream uploads to disk - #2312
Conversation
… disk - add writeStream/putStream to the storage and blob store interfaces so large values are written incrementally (tmp + fsync + rename) instead of being buffered in memory - FileService.save now streams the request body straight to the blob store and only counts bytes for FileMeta.size - drop the multipart fileSize limit and the file.too_large error from the /files upload path (41301 stays in the wire protocol for session export)
🦋 Changeset detectedLatest commit: 418c6a6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 418c6a675d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| /** | ||
| * Streamed variant of `atomicWrite`: same tmp + fsync + rename discipline, but | ||
| * the content arrives as an `AsyncIterable` so arbitrarily large values never | ||
| * sit in memory at once. | ||
| */ |
There was a problem hiding this comment.
In agent-core-v2 source, this new block comment is outside the top-of-file module header, but the scoped guide requires comments to live only in that header and not narrate implementation details. Please move or remove this note so the file follows the v2 comment convention.
AGENTS.md reference: packages/agent-core-v2/AGENTS.md:L13-L13
Useful? React with 👍 / 👎.
1 participant
Related Issue
No linked issue — the problem is explained below.
Problem
The built-in server's
/filesupload endpoint rejected anything over 50 MiB with an upload-too-large error, which blocked large attachments (for example videos or archives added through the web UI). The cap also acted as a memory guard: uploads were buffered fully in memory before being written to disk, so simply raising the limit would have made large uploads a memory hazard. The server targets local single-user deployments, so the concurrency/abuse rationale for a hard cap does not apply.What changed
/filesendpoint (both the multipartfileSizelimit and the service-level check), and deleted the now-unreachablefile.too_largeerror. The wire code 41301 stays in the protocol because session export still maps its own too-large error onto it.writeStreamto the filesystem storage service andputStreamto the blob store, writing through tmp-file + fsync + rename so a failed upload leaves no partial file behind.FileService.savenow pipes the request body straight to disk and only counts bytes forFileMeta.size, so upload memory usage no longer scales with file size.fileSizetoNumber.MAX_SAFE_INTEGER: when left unset,@fastify/multipartsilently falls back to Fastify's 1 MiBbodyLimitand truncates the file stream (caught by the e2e test).Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.