feat(diag): persistent memory trajectory + query-leak CI soak (#581 observability)#675
Merged
Conversation
#581 (query-path memory growth) does not reproduce in our soak: a 10-min, 200-file query-only run shows RSS flat-to-shrinking and committed plateauing (mimalloc already purges on free, purge_delay=0). Rather than ship speculative mitigations for a leak we cannot measure, this adds the observability to find it where it actually occurs, plus a continuous guard. - diag: the diagnostics writer now appends a PERSISTENT NDJSON memory trajectory (one sample / 5s: rss, committed, peak fields, page_faults, fd, queries), kept on exit (rotates to a .1 generation past 8 MB) so users can send the time series post-mortem. The previous latest-snapshot file was overwritten every 5s and deleted on stop. - ci: the soak now also runs a read-only query-leak leg (CBM_SOAK_MODE query-leak, no reindex/mutate) on every platform including Windows, so a regression that introduces a query-path leak is caught. soak-test.sh RESULTS_DIR is now env-overridable so both legs keep separate artifacts. - docs: README Troubleshooting and Diagnostics section + a bug-report field explain the no-telemetry stance and how to capture and share the trajectory. Build clean; unit suite 5714/0; trajectory verified to persist post-exit. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds observability + a CI guard to find and prevent query-path memory leaks (#581), based on a diagnostic finding that the leak does NOT reproduce in our soak.
Diagnostic finding
A 10-minute, 200-file query-only soak (CBM_SOAK_MODE=query-leak) shows RSS flat-to-shrinking and committed memory plateauing. With mimalloc purge_delay=0 (mem.c), freed memory is already returned to the OS immediately. Speculative per-query collect/checkpoint mitigations are therefore not justified, and a WAL TRUNCATE is unsafe here (ftruncate can SIGBUS an mmap'd sibling, see store.c). So this PR ships observability to find the leak where it actually occurs, plus a continuous regression guard — not a speculative fix.
Changes
Verification