Heap profiler with leak tracking #5763
Conversation
|
I ran some tests both during indexing (ingest V2) and search (float aggregation). On a pretty system (10 indexes with 1 milion docs ingested to each index): Without the feature flag:
With the feature flag but without starting the profiling:
With the profiling activated with min_alloc_size=64KiB
With the profiling activated for search only and min_alloc_size=1KiB
With the profiling activeted for indexing and min_alloc_size=1KiB
|
rdettai
force-pushed
the
custom-heap-pprof2
branch
2 times, most recently
from
725eb82 to
010e717
Compare
|
New measurement with the tokio tracing: Without the feature flag: With the profiling activated with min_alloc_size=64KiB |
rdettai
force-pushed
the
custom-heap-pprof2
branch
3 times, most recently
from
2e4b375 to
13a7447
Compare
| } | ||
| } | ||
|
|
||
| /// Updates the the memory location and size of an existing allocation. Only |
There was a problem hiding this comment.
| /// Updates the the memory location and size of an existing allocation. Only | |
| /// Updates the memory location and size of an existing allocation. Only |
| }; | ||
| let (callsite_hash, old_size_bytes) = if old_ptr != new_ptr { | ||
| let Some(old_alloc) = self.memory_locations.remove(&(old_ptr as usize)) else { | ||
| return ReallocRecordingResponse::ThresholdNotExceeded; |
There was a problem hiding this comment.
is this supposed to never happen or something like that?
There was a problem hiding this comment.
Allocations that exist before the profiler is started are not tracked in self.memory_locations. For those untracked allocations we decide not to track reallocations either. It's a small blind spot in the profiler's capability. For instance we are not able to pick up the growth of a long lived memory arena.
| let Some(old_alloc) = self.memory_locations.remove(&(old_ptr as usize)) else { | ||
| return ReallocRecordingResponse::ThresholdNotExceeded; | ||
| }; | ||
| self.memory_locations.insert( |
There was a problem hiding this comment.
how are we certain that this insert does not allocate?
There was a problem hiding this comment.
ah I assume this is because you have done a remove before?
There was a problem hiding this comment.
Indeed, this insert does not grow the map because of the remove right before. At a load factor of 0.5 I'm pretty confident no resize will be triggered.
rdettai
force-pushed
the
custom-heap-pprof2
branch
from
a16a059 to
1459304
Compare
Description
This PR adds a very lightweight, feature gated, heap profiling endpoint. It superseeds the previous attempt in #5724
Using the jemalloc native profiling is complicated, it's hard to asses its performance impact, and the resulting traces are hard to interpret.
This PR proposes an custom and simple heap profiler:
When started, the profiling allocates ~30MB of symbols.
How was this PR tested?
Python scripts, see results below.