[Bug] Memory leak when re-evaluating modified pattern with nested templates

[skyfloogle]

Operating System

Linux

What's the issue you encountered?

While putting together a pattern, I noticed that every time I modified the pattern and re-evaluated it, ImHex's memory usage went up by about 4 gigabytes. Over time, this leads to my system running low on RAM.
I have narrowed it down to the CMDL struct (quick access with CMDL cmdl @ 0xEC;) and its DictInfo<RelPtr<T>> and ListPtr<RelPtr<T>> fields. The RelPtr<T> is an optional relative pointer to a T, and the ListPtr is a relative pointer to an sized array of T. DictInfo<T> contains multiple layers of RelPtr<T>. Replacing T with char either prevents the memory increase or reduces it so much I can't measure it. I added [[inline]] attributes in many places for legibility, but removing these does not resolve the issue. I haven't been able to narrow it down further, and I'm not completely confident that this is the root cause.
I've attempted to investigate the issue myself using -fsanitize and Valgrind, but neither has given me much insight (a heap dump from Valgrind's "massif" tool identified under 200MB of data, while htop indicated memory usage was over 2GB at the time). I haven't used these tools before though, so I could be doing something wrong. I can at least verify that the issue is there in debug builds on the latest master commit as of this writing.
In the meantime, I've resorted to restarting ImHex whenever I notice memory usage is low. My 16GB system with other stuff open can only handle 2 rounds of evaluation with the full pattern, but if I replace the CMDL pointer with a char pointer it's usable.

How can the issue be reproduced?

1. Load the provided binary and pattern
2. Evaluate the pattern (▶️ button)
3. Make a change in the pattern editor (e.g. remove a later field in the DATA struct, or even remove the entire thing)
4. Re-evaluate the pattern
5. Observe in htop or similar that memory usage goes up by several gigabytes (or, in the case of removing the entire pattern, doesn't go down at all)

ImHex Version

1.36.2 and 2ba7db1

ImHex Build Type

• Nightly or built from sources

Installation type

Tried imhex-bin off AUR, the nightly AppImage, and a local build

Additional context?

I'm running EndeavourOS with KDE/Wayland on an AMD Ryzen 5 4500U with integrated graphics.

cgfx.zip contains model.cgfx (the binary, an intermediate build artifact of one of my projects) and cgfx.hexpat (the pattern).

[WerWolv]

Hey, thanks for the report! I'll definitely look into what's causing this. There's various issues currently when nesting template types but I wasn't aware of such a massive memory leak

[skyfloogle]

I'm not sure whether this is related, but after further work on my pattern, memory appears to be ballooning during parsing, and doesn't depend on the pattern data being present at all. My current pattern uses 11.3GB of RAM in plcli when applied to an empty file.

After investigating further, it seems to be at least partly because when template parameters are involved, the entire ASTNodeTypeDecl is cloned, including the type parameters.

I'm not very familiar with the code's structure and how easy it is to change things, but would it be possible to have template implementations be represented by some object with weak references to the definitions of its type parameters and to the relevant template definition?
It's also possible that the data being cloned really shouldn't be this big, and maybe some stuff could be shrunk or shared there (e.g. maybe strings could be shared with some kind of string pool, or stored as slices of the original source code).

Here's my current pattern. It seems to be particularly caused by a rather deep nesting of templates: the name in the LutInfo struct is potentially quite deep, possibly 20+ levels of structs inside structs, many of which are templates. It can also be worked around by removing the DictInfo<CMDL> from the DATA at the bottom, as that is presumably where the biggest clone happens.

[BlueFalconHD]

Got a very similar issue with an incredibly small and simple pattern and 3kb file. Here is my pattern:

enum SerializationFormat: u8 {
    a = 0,
    b = 1,
    c = 2
}

struct MyFileFormat {
    char magic[8];
    be u32 version;
    be SerializationFormat fmt;
    be u64 size;
    char data[size];
}

MyFileFormat f @ 0x00;

I am on macOS, and I used the memory trace tool to monitor the usage, it seems to be linearly increasing, with small decreases at times. Slowly but surely, it makes its way to incredibly high usage. Before I had the idea to start the trace, my Mac forced me to close an application or else my system would restart, and ImHex was using 51GB (not a typo) of RAM with that single pattern and the latest version (1.37.4). In my trace, the allocations are also measured per function, so once the analysis finishes I will come back and share where the allocations are happening. There probably is a leak somewhere, and this might not even be the same issue as the ones above.