getblobs parallelization - #502
Open
mathieu-plak wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds caller-controlled parallelization to Repository.GetBlobs, allowing packfile range reads and blob decoding to be performed concurrently while attempting to preserve the existing “set semantics” (deduped requests) and error-per-blob delivery model.
Changes:
- Introduces
GetBlobsOptswith aConcurrencysetting (defaulting to 1) and threads it intoGetBlobs. - Refactors
GetBlobsinto a feeder/worker pipeline using channels and cancellation to avoid goroutine leaks on early iterator exit. - Adds a dedicated concurrency test suite covering roundtrip correctness, range read errors, early break/cancellation, and worker/read count mismatches.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| repository/getblobs.go | Adds GetBlobsOpts and implements concurrent range-read execution with cancellation-aware shutdown. |
| repository/getblobs_concurrency_test.go | Adds tests validating correctness under concurrency and checking for goroutine leaks on early exit/cancel. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+125
to
+149
| returnChannel := make(chan iterItem, opts.Concurrency) | ||
| var wg sync.WaitGroup | ||
| for range opts.Concurrency { | ||
| wg.Go(func() { | ||
| for rp := range readsChannel { | ||
| data, err := r.GetPackfileRange(rp.loc) | ||
|
|
||
| for _, b := range rp.blobs { | ||
| it := iterItem{b: BlobResp{b.BlobReq, nil}} | ||
| if err != nil { | ||
| it.err = err | ||
| } else { | ||
| blobStart := b.Offset - rp.loc.Offset | ||
| blobBytes := data[blobStart : blobStart+uint64(b.Length)] | ||
| it.b.Data, it.err = r.decodeBuffer(blobBytes) | ||
| } | ||
|
|
||
| select { | ||
| case returnChannel <- it: | ||
| case <-innerCtx.Done(): | ||
| return | ||
| } | ||
| } | ||
| } | ||
| }) |
Contributor
Author
There was a problem hiding this comment.
You might want to refresh your memories on the WaitGroup API :).
* This adds a new know that let the caller specify how much concurrency they want for the actual Gets.
mathieu-plak
force-pushed
the
mm/getblobs_parallelization
branch
from
July 24, 2026 10:25
25b049d to
7fad0c1
Compare
Comment on lines
+56
to
+62
| if opts == nil { | ||
| opts = makeDefaultGetBlobsOpts() | ||
| } | ||
|
|
||
| if opts.Concurrency == 0 { | ||
| opts.Concurrency = 1 | ||
| } |
Contributor
Author
There was a problem hiding this comment.
It's a pointer.... caller should think it'll be mutated. Thoughts from humans?
Comment on lines
+99
to
+107
| // Derived context to handle cancellation from the iterator loop, | ||
| // otherwise an early exit will leak all goroutines. | ||
| innerCtx, cancel := context.WithCancel(ctx) | ||
| defer cancel() | ||
|
|
||
| for _, b := range g.blobs { | ||
| if err != nil { | ||
| if !yield(BlobResp{b.BlobReq, nil}, err) { | ||
| return | ||
| } | ||
| // Feed the read plans to the workers. | ||
| readsChannel := make(chan *rangeRead) | ||
| go func() { | ||
| defer close(readsChannel) |
Contributor
Author
There was a problem hiding this comment.
This is all dominated by i/o so no I don't think it's worth the added complexity... Thoughts from humans?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for GetBlobs parallelization (chosen by the caller) of the reads.