Parallelize archive build and size-aware retry hint - #91
Merged
Conversation
Prefetch small source objects concurrently (bounded by count + in-flight bytes) a window ahead of the single-threaded zip writer, collapsing the serial per-object GetObject latency that dominates runs with thousands of tiny files. Large/unknown-size files still stream inline to keep memory bounded. Replace the flat 5s MCP retry hint with an estimate derived from file count and total run size (clamped to [5s, 30s]), threading per-file sizeBytes from the web layer into the Lambda invoke payload. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Trim over-length comments, reference constants by name instead of restating values, backtick symbols, and drop an inaccurate note about boto3's connection pool. Comments only; no behavior change. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_PREFETCH_CONCURRENCY=16and a_PREFETCH_MAX_INFLIGHT_BYTES=256 MBbudget) a window ahead of the single-threaded zip writer. This collapses the serial per-objectGetObjectlatency that dominated runs with thousands of tiny files. Large/unknown-size files still stream inline so the memory-bounded design (e.g. a 200 GB run in a 10 GB Lambda) is preserved, and zip entry ordering is unchanged.get_run_archiveretry hint withestimateRetryAfterSeconds({ fileCount, totalBytes })—base + fileCount*perFile + bytes/throughput, clamped to[5s, 30s]. This better reflects build time for both many-small-files and few-large-files runs.sizeBytesfrom the web layer (loadDownloadableFiles->InvokeBuildArchiveInput) into the Lambda invoke payload assize_bytes, used both for the retry estimate and for the builder's prefetch-eligibility decision. Parsing is lenient/backward-compatible (missing or malformed -> unknown -> inline stream).Driveby changes
npm audit fixto address security vulnerabilities in Next.js project.Test plan
size_bytesparsing.estimateRetryAfterSecondsunit tests.download-archive(7) andmcp(15).make check-all(format, lint, typecheck) clean across Python and frontend.Made with Cursor