mempressure: Fix Linux memory detection and reduce memory check overhead #219

bigshiny90 · 2025-10-14T15:22:16Z

Use /proc/meminfo MemAvailable instead of sysinfo() freeram+bufferram on Linux. MemAvailable accounts for reclaimable memory (page cache, etc), providing accurate memory availability.

Added support for macOS using kern.memorystatus_vm_pressure_level sysctl

Added support for containers on both linux and windows. (new macOS native containers to TBD)

linux - (edit: proc/self/meminfo doesn't exist.. was misinformed) using cgroup code instead
Windows - uses Job Objects (QueryInformationJobObject)

Memory pressure is periodically via Scheduler (10s interval). SystemNeedsMemoryReleased() (used in FlushStateToDisk) now reads a cached flag instead of rechecking available memory each time. After a Flush/Sync, Update flag immediately after to prevent repeated flushes.

Fixes #188
Addresses #216

Regarding #216 point:

First, try to free already-dumped coins (maybe only as needed). If that's insufficient, dump and free dirty ones.

After much experimentation found that the custom memory allocator (PoolAllocator -> Pool.h - which uses a singly-linked list for its free list), is fairly hard to use in context of trying to free partial amounts of allocated memory. The current way of freeing this memory is to completely destroy and recreate it (in ReallocateCache which happens in Flush). All or nothing. So, doing anything other than a full Flush, will not help with memory.

Using Flush when we have a low available memory seems to work just fine though. (tested multiple times on linux, with <64MB of available memory).

Adding some instructions for creating a Docker image to test containers, in post below.

Needs further testing altogether. Specifically:
Windows testing
Linux testing (Tested)
MacOS testing
Container testing: Linux container, Windows container, Mac container

Edit 10/20/25: Still working on best metrics for determining container mempressure.

kwsantiago · 2025-10-14T16:18:41Z

src/util/mempressure.cpp

+                // Format: "MemAvailable:    1234567 kB"
+                size_t pos = line.find_first_of("0123456789");
+                if (pos != std::string::npos) {
+                    uint64_t mem_kb = std::stoull(line.substr(pos));


We should have exception handling for invalid number format in case std::stoull throws an error.

added a try/catch block around this

kwsantiago · 2025-10-14T16:19:28Z

src/init.cpp

+    // During normal operation: check every 10 minutes (between block arrivals).
+    // FlushStateToDisk calls SystemNeedsMemoryReleased() which uses the cached flag.
+    std::function<void()> check_memory_pressure;
+    check_memory_pressure = [&node, &scheduler, &check_memory_pressure]() {


Self-referential lambda capture is undefined behavior.

definitely need to fix this

kwsantiago · 2025-10-14T16:20:32Z

src/validation.cpp


+            // After flushing, immediately recheck memory pressure to update the flag
+            // This prevents repeated flushes on the same memory pressure event
+            if (SystemNeedsMemoryReleased()) {


There's a race condition where the flag could be set again between check and update.

Thanks for the feedback Kyle… all of this needs more work. Simply a proof of concept sketch for now… making sure it’s addressing Luke’s issue first. Looks like it’s on the right track conceptually, I need to put some real work into it now.

Yes definitely on the right track, great job!

There's a race condition where the flag could be set again between check and update.

good consideration though.. i'm reviewing your points now.

regarding the race condition - though yes, technically it exists but is benign. the actual code is changing a bit as I move forward (i.e. the recheck for mempressure will only happen IF we are releasing cache memory because of mempressure - it will no longer happen at the end of ANY flush) but it will still exist. not sure there is more to do here. the flag being set is atomic. don't want to add mutex in the middle of FlushStateToDisk, etc..

that said, if you have a suggestion please share. I'm not a c++ expert! ;)

Makes sense.

kwsantiago

Solid implementation. The overflow check is the only real concern, and it's an edge case (would need 16+ exabytes in /proc/meminfo).

For macOS implementation, consider adding:

#ifdef __APPLE__
    // Use vm_stat or host_statistics64()
#endif

kwsantiago · 2025-10-15T02:09:58Z

src/util/mempressure.cpp

+                if (pos != std::string::npos) {
+                    try {
+                        uint64_t mem_kb = std::stoull(line.substr(pos));
+                        uint64_t available_memory = mem_kb * 1024; // Convert kB to bytes


Consider ading an overflow check before multiplication:

if (mem_kb > UINT64_MAX / 1024) { // Treat huge values as "plenty of memory" break; }

easy enough. done.

1440000bytes · 2025-10-19T17:41:31Z

Needs further testing altogether. Specifically:
Windows testing

Can you share the steps to test this on Windows?

bigshiny90 · 2025-10-19T17:51:23Z

Needs further testing altogether. Specifically:
Windows testing

Can you share the steps to test this on Windows?

No idea (yet) specifically for windows. Generally testing has been setting up an environment that allows easier testing of our required conditions. Conditions being simply low available memory (default setting is <64MB). In my Linux testing, I have been using an Ubuntu VM with 4GB RAM and setting dbcachesize to 3.5GB. Fairly guaranteed to run out of memory during IBD.

probably easier would be a VM with 2GB mem, etc…

other potentials on a system with more total RAM, would be to artificially stress the memory to trigger low mem while running bitcoind in IBD… something like stress-ng or a script to use up/allocate a bunch of memory.

generally I’m watching debug.log for the Memory pressure output..

bigshiny90 · 2025-10-19T21:47:49Z

Adding some instructions for creating a good Docker image to test containers, in post below.

Docker Memory Pressure Testing Guide

This guide shows how to test the memory pressure detection features in a Docker container with limited memory.

Prerequisites

Docker installed and running
Bitcoin Knots source code with memory pressure detection changes

Step 1: Build the Docker Image

From the Bitcoin repository root, build the Docker image:

docker build -t bitcoin-knots:mempressure -f contrib/docker/Dockerfile .

This compiles your modified code into a Docker image based on Alpine Linux.

Note: The build may take 10-30 minutes depending on your system.

Step 2: Prepare Test Directory and Config

Create a directory for the blockchain data and a minimal config file:

# Create data directory
mkdir -p ~/bitcoin_mempressure_test

# Create bitcoin.conf with high dbcache to trigger memory pressure
cat > ~/bitcoin_mempressure_test/bitcoin.conf << 'EOF'
dbcache=1800
EOF

Step 3: Run Container with Memory Limit

Run the container with a 2GB memory limit:

docker run -it --rm --memory=2g --name bitcoin-test \
  -v ~/bitcoin_mempressure_test:/var/lib/bitcoind \
  bitcoin-knots:mempressure \
  -conf=/var/lib/bitcoind/bitcoin.conf -printtoconsole

Command explanation:

--memory=2g - Limits container to 2GB RAM (creates Job Object on Windows or cgroup on Linux)
-v ~/bitcoin_mempressure_test:/var/lib/bitcoind - Mounts data directory
-conf=/var/lib/bitcoind/bitcoin.conf - Overrides default config path
-printtoconsole - Shows logs in terminal

Step 4: Monitor Memory Pressure Detection

Watch the logs for memory pressure events. You should see messages like:

On Linux (container detected via cgroup):

CheckMemoryPressure: YES: 50000000 available memory

On Windows (container detected via Job Object):

GetJobObjectMemoryLimit: Detected Job Object memory limit: 2147483648 bytes
CheckMemoryPressure: YES: 45000000 available memory (in container)

Note: Memory pressure triggers when available memory drops below the threshold (default 64MB = 67108864 bytes). With dbcache=1800 in a 2GB container, you should see pressure events as the cache fills up during IBD or block processing.

Make sure that after a memory pressure event in log, we then see a Flush and the next UpdateTip cache should be 0. we should NOT immediately get another Flush attempt.

If you want to go wild and watch mem usage to confirm memory release and filter debug.log to useful info:

watch -n 1 "docker stats bitcoin-test --no-stream"
tail -f ~/bitcoin_mempressure_test/debug.log | grep -E '(CheckMemoryPressure|Flushing large)'

Cleanup

Remove test data when done:

rm -rf ~/bitcoin_mempressure_test

Remove Docker image:

docker rmi bitcoin-knots:mempressure

… for macOS Linux: Use /proc/meminfo MemAvailable with cgroup memory limit detection for container environments Windows: Use GlobalMemoryStatusEx with Job Object support for container macOS: Use kern.memorystatus_vm_pressure_level sysctl to detect kernel CheckMemoryPressure(): updates atomic flag SystemNeedsMemoryReleased(): read of cached flag Schedule CheckMemoryPressure to run every 10 seconds

kwsantiago suggested changes Oct 14, 2025

View reviewed changes

bigshiny90 requested a review from kwsantiago October 14, 2025 18:01

bigshiny90 force-pushed the 29.2-knots-mempressure-refactor branch 2 times, most recently from fee03eb to 775b944 Compare October 14, 2025 18:34

kwsantiago reviewed Oct 15, 2025

View reviewed changes

bigshiny90 force-pushed the 29.2-knots-mempressure-refactor branch 2 times, most recently from b000f56 to 8310027 Compare October 19, 2025 15:45

bigshiny90 force-pushed the 29.2-knots-mempressure-refactor branch 4 times, most recently from 3df8675 to e293be4 Compare October 21, 2025 17:53

bigshiny90 force-pushed the 29.2-knots-mempressure-refactor branch 2 times, most recently from 7889ded to 83d51c7 Compare October 24, 2025 15:44

systemmemory: flush memory profiling

100bad0

bigshiny90 force-pushed the 29.2-knots-mempressure-refactor branch from 83d51c7 to 100bad0 Compare October 26, 2025 15:19

bigshiny90 closed this Oct 27, 2025

bigshiny90 deleted the 29.2-knots-mempressure-refactor branch October 27, 2025 21:17

mempressure: Fix Linux memory detection and reduce memory check overhead #219

mempressure: Fix Linux memory detection and reduce memory check overhead #219

Uh oh!

Conversation

bigshiny90 commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kwsantiago left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1440000bytes commented Oct 19, 2025

Uh oh!

bigshiny90 commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bigshiny90 commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docker Memory Pressure Testing Guide

Prerequisites

Step 1: Build the Docker Image

Step 2: Prepare Test Directory and Config

Step 3: Run Container with Memory Limit

Command explanation:

Step 4: Monitor Memory Pressure Detection

Cleanup

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bigshiny90 commented Oct 14, 2025 •

edited

Loading

bigshiny90 commented Oct 19, 2025 •

edited

Loading

bigshiny90 commented Oct 19, 2025 •

edited

Loading