Repository navigation
Reduce memory usage footprint #3490
Description
We still have a large memory footprint for each shape. We think this comes memory not all memory allocated reclaimed during the lifetime of a consumer.
With the new process shutdown approach (#3455), we expect improvements as processes recycle more often. We should observe that, otherwise need to dig further.
The next thing would be to enforce a cap on memory usage, since larger workloads might have more "active" shapes. The idea would need to enforce a max number of consumers at any point. The challenge is that transaction handling must be resilient to shapes being opened/closed multiple times during the handling of a transactions, as large transactions may touch many rows.
Alternative approach: Replace ETS with SQLite for automatic memory management via disk offloading. Use a single SQLite per stack - previous benchmarks showed comparable performance to ETS.
Activity
magnetised commentedon Nov 24, 2025
@balegas re our conversation I think the initial state of the system, with no shapes, is actually reasonable. My local issue (a fresh instance with 0 shapes taking ~2GB RAM) was due to my absurdly large "max open files" setting, which was causing the beam to create a massive port lookup table. In production we run with max files set to
524288. When I launch electric with this value for the max ports (+Q 524288) then the initial memory is a very respectable 200MB or something.Looking at actual instances with low shape counts, e.g. volta, the memory usage is not from shapes but various lower level/system/telemetry sources. this is top memory usage with 18 shapes:
So
tlsand telemetry.As for runtime usage, with ~150,000 defined shapes and ~700 active electric's memory usage is ~1.2GB, of which
the remaining ~200MB is for the consumers. So most of our ram is going to the knowledge that the shapes exist and their metadata, not the consumers (under this usage pattern).
I'm going to look into the
PublicationManager.RelationTrackeras a next step because that seems extremely large.I'm also seeing ~142KB of memory per request that's waiting in
Shapes.Api.hold_until_change/1which is not a huge amount but in an active system with lots of live requests that would add up.balegas commentedon Nov 24, 2025
Right.
Amperememory seems to climb quite quickly. This is after a restart after 10K shapes.I'm still interested about why the memory usage for the
Consumeris so high, but maybe we can look into it afer @robacourt work lands as we'll be more optimal then.I think the plan remains. Tasks look good.