Memory corruption on Debian 13 linux 6.12.63+deb13-powerpc64le

[jfiusdq]

The issue happens in a reproducible way, the device is filled with around 850GB of data so a rather big device, it used to work OK but now it does not. Note that I activated dedup on and off on a live vdo using lvm for testing before, not sure if related because this error used to not appear and now it appears in a completely reproducible manner on every boot and if I deactivate auto-activation of the vdo lv on boot it stops happening so it's clearly the source. I have verified all other causes such as disk related, it's not disk corruption, also my server is running ECC memory. It's definitely memory corruption caused by dm-vdo in the kernel. I'm using all default settings for lvm vdo on Debian 13 beside block map cache being 4096MB. The vdo device is on a separate disk than root which the system boots from and where the kernel modules are located (again, their hashes match on disk with official repos, they're not corrupted on disk).

There is something in dm-vdo that writes arbitrary memory address in kernel space and corrupts other module loading.

[    8.453342] module_64: garp: Address 00000000bba0fdc3 of stub out of range of 00000000a8dc9270.
[    8.612851] module_64: cfg80211: REL32 4996086348 out of range!
[    8.670602] module_64: 8021q: Address 00000000d4f483ce of stub out of range of 00000000a8dc9270.
[    8.794161] module_64: cfg80211: REL32 5020002892 out of range!
[   10.007360] device-mapper: vdo: vdo0:physQ0: VDO commencing normal operation
[   10.007409] device-mapper: vdo: vdo0:journalQ: Setting UDS index target state to online
[   10.007457] device-mapper: vdo0:vgchange: device '253:4' started
[   10.007466] device-mapper: vdo0:vgchange: resuming device '253:4'
[   10.007976] device-mapper: vdo: vdo0:dedupeQ: loading or rebuilding index: 253:3
[   10.007986] device-mapper: vdo: vdo0:dedupeQ: Using 8 indexing zones for concurrency.
[   10.025552] device-mapper: vdo0:vgchange: device '253:4' resumed
[   13.354193] device-mapper: vdo: vdo0:dedupeQ: loaded index from chapter 4598 through chapter 5620
[   38.723094] systemd-journald[451]: Time jumped backwards, rotating.
[  198.155103] module_64: dm_crypt: REL32 -4996348216 out of range!
[  201.898237] module_64: xfs: REL32 5252766060 out of range!

I don't really have more details to give since beside those logs there's no other visible memory errors.

[lorelei-sakai]

When you say "this used to work OK", do you know what kernel (or what version of dm-vdo) you were running before? Was it on the same hardware? It might be helpful to compare the previous working scenario with the current one to see what might have changed.

I have never seen this error before, so I did some digging. The error itself comes from module_64, which appears to be involved with the way kernel modules are loaded:
https://elixir.bootlin.com/linux/v6.12.63/source/arch/powerpc/kernel/module_64.c#L923

It looks to me like what may be happening is that the module loader is generating jumps that are too large for the kernel to handle. I don't think this is memory corruption per se, but more likely the module loader is not taking all the limitations into account when loading modules. I don't really see how this can be the result of anything the dm-vdo module is doing on its own. However, it may be a side-effect of loading the dm-vdo module, which is not small. dm-vdo is built as a dynamic module by default so it won't get loaded until something (like device-mapper) actually tries to use it.

Lots of the suggestions for dealing with this seem to involve recompiling the kernel with appropriate flags, but unless you built this kernel yourself I'm going to assume that's not a viable option. You might check that the kernel you are running was build for the hardware you're running it on, though.

You might also experiment with different sets of modules, or loading modules in a different order, to see how it affects this issue. For example, you could disable vdo auto-activation, and then activate the vdo volume after boot. You could try loading the dm-vdo module without activating the module at all. It may also be worth contacting whatever Debian support you have (if any) to see if you can get help from someone with more expertise on the module loader than we have.

Let me know if you get information that might help narrow down where the issue is, too.

[jfiusdq]

When you say "this used to work OK", do you know what kernel (or what version of dm-vdo) you were running before? Was it on the same hardware? It might be helpful to compare the previous working scenario with the current one to see what might have changed.

same hardware, same version, same system, just a few days ago

I did deactivate dm-vdo using lvm as I said in my first post and the issue disappeared, I have since migrated my data off of it because it's been too slow and this weird module corruption error scared me off, ppc64le isnt that well tested for all those I believe there might be some undefined behavior which triggers such errors on ppc64le and not on other archs. I think it's definitely corruption, it has nothing to do with the module loader itself, if you look at the addresses, they're just random, and the modules affected depending on reboot are also changing even though it's often the same ones for example it has happened that the xfs module works, and next boot it doesnt, but it's always related to loading a volume with dm-vdo, if such volume is never loaded there is never such corruption and the module loader works just fine.

[jfiusdq]

To debug this issue, I think what would have to be done is run dm-vdo extensively with kasan and https://docs.kernel.org/next/dev-tools/ubsan.html

[jfiusdq]

@lorelei-sakai I can't say for sure if it's the root cause but I found this and confirmed even on x86 machine and produced repro scenario: #417

[jfiusdq]

@lorelei-sakai found another kernel panic: #418

[raeburn]

The other kernel panic issues you found are definitely very concerning, but relate to loading a corrupted device image. This issue, as far as I can see, doesn't involve any known storage corruption, and aside from running on PPC would seem to be the sort of thing we test regularly.

Matt's analysis looks like my initial impressions from the messages logged. Both the "Address...of stub out of range" and the "REL32...out of range" are triggered by trying to apply relocations to store a relative offset of something over 2GB away from where the stub is being stored (REL32) or the TOC address (stub). Since you're allocating 4GB of cache space that does seem entirely plausible. If you load the modules before starting VDO, or use the default cache size, do you see any problems?

The “stub” stuff comes into play when processing REL24 relocations in some cases, or when dealing with some ftrace hooks.

Why might this come up on ppc and not x86-64? This is at a level where the instruction encoding, calling conventions, object file format, and compiler flags matter. The two architectures are quite different.

Curiously, though, in the Debian kernel tree package I just unpacked (linux-image-6.12.63+deb13-powerpc64le_6.12.63-1_ppc64el.deb), I don't see REL32 relocations in xfs.ko (“objdump -r xfs.ko”); I see ENTRY, REL24, REL64, and some TOC relocations. Is this a stock debian tree or did you rebuild your kernel?

Note that some of the “addresses” printed are really relative offsets:

[ 8.453342] module_64: garp: Address 00000000bba0fdc3 of stub out of range of 00000000a8dc9270.

00000000bba0fdc3 is a relative offset of nearly 3GB.

[ 8.612851] module_64: cfg80211: REL32 4996086348 out of range!

4996086348 is a relative offset of over 4.6GB.

So they’ll be affected by the allocation patterns on each boot, which (in part for security reasons) I would not expect to be consistent.

[jfiusdq]

@raeburn I had to erase the block device which contained my buggy dm-vdo but no I didnt see any problems with the default block map cache size before but also I was still filling up my file system with my data so it might come from data volume too (800GB+), and loading those modules without vdo on boot works, I didnt test manually loading vdo after the fact but it did reproduce reliably on boot. Yes the other issues are memory safety issues that, under the conditions of my research and testing assisted by AI code analysis, can only be reproduced with specially crafted image, I have not been able to identify a scenario which can happen solely from normal operation even if it could be found, also normal disk corruption could do it, however it still likely avoids one to fix them now that we know they exist.

Note that also on ppc64le, there is 4k and 64k page size kernels (RHEL and Fedora use 64k), if I remember correctly, I was on 4k page size on debian when it happened, I never tested with 64k page size with vdo on Debian (I've since now switched to 64k kernel due to other issues), but it may be related too. I never rebuilt my kernel, it's stock. And I verified all the files on disk and they match checksums across multiple reboots so it's not disk corruption (I am on raid also). Maybe my log contained REL32 because of further corruption of memory which even changed the descriptor? Also maybe it could be stack corruption rather than heap? Not sure.

Yes there is KASLR but it ended up being always more of less the same modules affected by this loading issues, the ones you see.

Whether or not we find the root cause of this issue (we probably can't now because I had to destroy the bug environment because of operational constraints, furthermore I decided to dismiss dm-vdo for performance reasons for now, I was only getting 10-20MB/s write speed with ssd cached hdd, and read speeds not amazing either, and I don't have the resources to tune it and wasnt sure if I could depend on it and if it's not as tested, that it would not corrupt my data at some point.). I will evaluate compressing my data on software level rather, it seems more supported and more resource efficient that way.

Thanks

[raeburn]

The memory use of VDO is mainly determined by the configuration, not how full it is. We allocate the memory we need at startup. Though, the amount used will affect the amount we read in at startup, which in turn will affect timing and maybe allocations in the I/O system below us.

I'll try to reproduce these problems on a ppc system here.

I'm sorry to hear about the performance problems. We do often see a factor of 2 slowdown -- or not -- depending on the platform configuration. Once in a while we hear about much worse performance but usually wind up being unable to dig into it -- e.g., if it's slow enough the user isn't willing to put up with it...

[jfiusdq]

@raeburn OK thanks!

Another reason is also that I learned dm-vdo does copy-on-write internally as I understand and paired with a database like postgresql apparently this increases disk fragmentation over time and slows down performance even more, but I wasnt testing R/W speeds with this database just raw sequential file copy, so I was worried it would be even worse then.

About the performance tuning, as suggested in RHEL 10 docs, I wish this was abit more automatic, that dm-vdo detects and prints itself in kernel log that it thinks it could be faster by increasing X setting, or better, does it automatically. Because it's really complex to understand what does what and tune it, and I have a good background in systems programming, I don't think we should have to understand the software so deeply just to use it at acceptable performance out of the box (even though it could be just a bug on PPC that makes performance worse in that case this feedback isnt so relevant but you get the idea).

It would be extremely helpful if all tuning options could be changed live, so that it's easy to directly see the impact of changing those options, without reboots or remounting, this would help tuning immensely. At the fingertips of 1 lvm command, to change the options and see how it affects I/O, because the main reason that tuning is a PITA is that you need to test so many times and remount the file system, remove everything that's running on it etc etc..

I was thinking at the end that dm-vdo might just be more suitable to use on plain SSD storage rather than HDD backed by SSD cache.

Also on IBM POWER9 there is PowerNX Nest for accelerated GZIP, it would be good to support that for even better compression ratios: https://github.com/libnxz/power-gzip (only works on 64k page size apparently)

[lorelei-sakai]

It's unclear to me what your intended use cases for dm-vdo were, but the primary feature of dm-vdo is efficient data deduplication. If you are using this for a database workload, they do not typically contain much duplicated data, and dm-vdo is not going to be a good choice for that scenario.

If you are only interested in compression and not deduplication, then compressing your data at the file level is likely to be both more efficient and easier to implement.

[jfiusdq]

It's unclear to me what your intended use cases for dm-vdo were, but the primary feature of dm-vdo is efficient data deduplication. If you are using this for a database workload, they do not typically contain much duplicated data, and dm-vdo is not going to be a good choice for that scenario.

If you are only interested in compression and not deduplication, then compressing your data at the file level is likely to be both more efficient and easier to implement.

For compressing a postgresql database, they generally recommend file system compression: https://dba.stackexchange.com/a/197138

I tried BTRFS and on HDD it causes extreme fragmentation which harms performance alot. Since apparently it's due to copy-on-write causing fragmentation on HDD with the database doing alot of small writes, any file system doing this would likely suffer from similar issues so in conclusion it's unclear why they recommend that and no one really tells about this speed problem, different people report "success" with dm-vdo for compressing postgresql such as: https://www.rushworth.us/lisa/?p=10961

All other solutions for compressing whole postgresql databases are proprietary. Evidently, compressing at file level is good but then I also need query capabilities so that means reimplementing something to fit that, too.

[raeburn]

Another reason is also that I learned dm-vdo does copy-on-write internally as I understand and paired with a database like postgresql…

Yes, that’s probably not a great use case for VDO.

About the performance tuning, as suggested in RHEL 10 docs, I wish this was abit more automatic, that dm-vdo detects and prints itself in kernel log that it thinks it could be faster by increasing X setting, or better, does it automatically.

I’d like to see automatic adjustments, but as you can see from the docs, it’s hard to figure out what the right values are -- and they’ll likely change with changes in the workload. Cache contention and overall system load will factor into it.

Because it's really complex to understand what does what and tune it, and I have a good background in systems programming, I don't think we should have to understand the software so deeply just to use it at acceptable performance out of the box (even though it could be just a bug on PPC that makes performance worse in that case this feedback isnt so relevant but you get the idea).

I agree.

It would be extremely helpful if all tuning options could be changed live, so that it's easy to directly see the impact of changing those options, without reboots or remounting, this would help tuning immensely. At the fingertips of 1 lvm command, to change the options and see how it affects I/O, because the main reason that tuning is a PITA is that you need to test so many times and remount the file system, remove everything that's running on it etc etc..

Rebuilding some of the data structures that are split across threads using “address mod thread_count” type calculations would require a bunch of new code to run at reconfiguration time. Or a significant reorganization of the data so as to not require such rebuilding, but without bringing back the constant locking and unlocking we put some effort into avoiding.

I was thinking at the end that dm-vdo might just be more suitable to use on plain SSD storage rather than HDD backed by SSD cache.

Yes, probably so.

Also on IBM POWER9 there is PowerNX Nest for accelerated GZIP, it would be good to support that for even better compression ratios: https://github.com/libnxz/power-gzip (only works on 64k page size apparently)

Yes, and other architectures have compression acceleration features too (s390 dfltcc instruction, Intel’s QAT). There’s an open pull request where some changes were started to make the compression algorithm configurable, though it hasn’t seen much work lately; some initial superblock changes to support it are on a development branch in the vdo-devel repo. Even cranking up the compression level for LZ4 might be an improvement, when the CPU cycles are available.