Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow reuse of slices exposed through Ready #51

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

CaojiamingAlan
Copy link
Contributor

Solve #15:
Use pooled slice to reduce the total bytes allocated.
Benchmark result:

 % benchdiff --old 177ef28a -r BenchmarkRawNode -c 5 .
test binaries already exist for '177ef28'; skipping build
checking out 'e5ac368'
building benchmark binaries for 'e5ac368' 1/1 -

  pkg=1/1 iter=5/5 go.etcd.io/raft/v3 |

name                    old time/op        new time/op        delta
RawNode/single-voter-8         922ns ± 3%         883ns ± 2%   -4.14%  (p=0.008 n=5+5)
RawNode/two-voters-8          1.33µs ± 1%        1.36µs ± 1%   +1.79%  (p=0.032 n=4+5)

name                    old firstIndex/op  new firstIndex/op  delta
RawNode/single-voter-8          1.00 ± 0%          1.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            3.00 ± 0%          3.00 ± 0%     ~     (all equal)

name                    old lastIndex/op   new lastIndex/op   delta
RawNode/single-voter-8          2.00 ± 0%          2.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            2.00 ± 0%          2.00 ± 0%     ~     (all equal)

name                    old ready/op       new ready/op       delta
RawNode/single-voter-8          2.00 ± 0%          2.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            2.00 ± 0%          2.00 ± 0%     ~     (all equal)

name                    old term/op        new term/op        delta
RawNode/single-voter-8          0.00 ± 8%          0.00 ± 0%     ~     (p=0.444 n=5+5)
RawNode/two-voters-8            1.00 ± 0%          1.00 ± 0%     ~     (all equal)

name                    old alloc/op       new alloc/op       delta
RawNode/single-voter-8          551B ± 0%          410B ± 1%  -25.59%  (p=0.000 n=4+5)
RawNode/two-voters-8            843B ± 1%          702B ± 0%  -16.68%  (p=0.008 n=5+5)

name                    old allocs/op      new allocs/op      delta
RawNode/single-voter-8          4.00 ± 0%          4.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            6.00 ± 0%          6.00 ± 0%     ~     (all equal)

Copy link
Contributor

@pav-kv pav-kv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flushing some quick cosmetic comments while I'm looking at the allocations story. Couldn't quickly grasp where the savings come from: we still reallocate and copy the slice.

raft.go Outdated Show resolved Hide resolved
rawnode.go Outdated Show resolved Hide resolved
rawnode.go Outdated Show resolved Hide resolved
rawnode.go Outdated Show resolved Hide resolved
@CaojiamingAlan
Copy link
Contributor Author

@pavelkalinnikov Thank you for your comments! I've resolved them.

we still reallocate and copy the slice.

The trick is at when we put back the slice:

rn.raft.msgsAfterAppendPool.Put(rn.raft.msgsAfterAppend[:0])

This clears the slice but reserves the underlying array, which we can reuse so that we don't need to allocate a new one each iteration.

@CaojiamingAlan CaojiamingAlan requested a review from pav-kv May 17, 2023 16:57
@@ -428,6 +438,10 @@ func (rn *RawNode) acceptReady(rd Ready) {
}
}
rn.raft.msgs = nil
// Release the msgsAfterAppend slice and put it back into the msgsAfterAppendPool
if cap(rn.raft.msgsAfterAppend) > 0 {
rn.raft.msgsAfterAppendPool.Put(rn.raft.msgsAfterAppend[:0])
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add a comment explaining the logic of this optimization. Similar to what you've said (#51 (comment))

rawnode.go Outdated
m.Responses = r.msgsAfterAppend

// msgsAfterAppend is a pooled slice, so we need to make a deep copy
msgs, needResp := len(r.msgsAfterAppend), false
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: msgsCount

@@ -424,6 +426,11 @@ func newRaft(c *Config) *raft {
preVote: c.PreVote,
readOnly: newReadOnly(c.ReadOnlyOption),
disableProposalForwarding: c.DisableProposalForwarding,
msgsAfterAppendPool: &sync.Pool{
New: func() interface{} {
return make([]pb.Message, 0, 8)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why 8?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the number used in the example given by #15 (comment) . Actually I don't really know what might be a good number, so adopted that directly.

Signed-off-by: caojiamingalan <[email protected]>
@CaojiamingAlan CaojiamingAlan force-pushed the reuse_of_slices_exposed_through_Ready branch from b8758fe to 72e5c3d Compare June 7, 2023 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants