Why the "OS-agnostic" driver cannot show full stacktrace

[wxx213]

When I trace the kernel function "filp_close" by eBPF tools, I got the stack as below:

__nvoc_dtor_MIGConfigSession;
nv_cap_close_fd;
filp_close;

The "__nvoc_dtor_MIGConfigSession" function belongs to nvidia driver, but the stacktrace didn't show who called the "__nvoc_dtor_MIGConfigSession". The possible reason is that the "OS-agnostic" driver disable the "stacktrace" by gcc option "-fomit-frame-pointer", I tried to build the driver with gcc option "-fno-omit-frame-pointer", however it didn't work.

Is anyone can help with this? How to build the driver with "stacktrace" enabled so that I would debug the nvidia driver issue?

[wxx213]

I suppose the driver use the "-O2" option which made most of the functions were inlined.
And when I use the "-O0" option, the kernel panic occured when the stack was overflow.

[mtijanic]

Hi, sorry for the late response, I was unexpectedly out of office for a few days.

I just installed the 550.54.15 driver from nvidia.com using -m=kernel-open argument, and tried:

1. In shell 1: sudo bpftrace -e 'kprobe:nv_cap_close_fd { print(kstack); }'
2. In shell 2: nvidia-smi

and got a good stack trace:

Attaching 1 probe...

        nv_cap_close_fd+1
        __nvoc_dtor_MIGMonitorSession+21
        __nvoc_objDelete+44
        clientFreeResource_IMPL+1728
        rmclientFreeResource_IMPL+77
        serverFreeResourceTreeUnderLock+187
        serverFreeResourceTree+806
        rmapiFreeWithSecInfo+290
        rmapiFreeWithSecInfoTls+73
        Nv01FreeWithSecInfo+78
        RmIoctl+693
        rm_ioctl+73
        nvidia_unlocked_ioctl+2013
        __x64_sys_ioctl+160
        do_syscall_64+89
        entry_SYSCALL_64_after_hwframe+110

bpftrace is the only BCC/eBPF tool I know really, but it shouldn't be any different for any other bcc approach. Could be a kernel config maybe? I tested this on a stock Ubuntu 23.10

Linux ipp1-1391 6.5.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Here's the full kernel config of that machine: https://gist.github.com/mtijanic/aaf7955fb822969b65116c54907a3d15

[wxx213]

I suppose it's not related to the kernel config for the other driver stacktrace works fine, The nvidia driver versions we used are 535.129.03 and 535.86.10, could you have a try with them in you Ubuntu 23.10 machine? @mtijanic

What's more I will check the kernel config diff carefully, Thanks.

[mtijanic]

Works fine for me:

local-mtijanic@ipp1-1391:~$ wget https://us.download.nvidia.com/tesla/535.129.03/NVIDIA-Linux-x86_64-535.129.03.run > /dev/null 2>&1
local-mtijanic@ipp1-1391:~$ chmod +x NVIDIA-Linux-x86_64-535.129.03.run 
local-mtijanic@ipp1-1391:~$ sudo ./NVIDIA-Linux-x86_64-535.129.03.run -m=kernel-open -q
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 535.129.03........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
local-mtijanic@ipp1-1391:~$ sudo bpftrace -e 'kprobe:nv_cap_close_fd { print(kstack); }' &
[1] 97489
Attaching 1 probe...

local-mtijanic@ipp1-1391:~$ nvidia-smi > /dev/null
        nv_cap_close_fd+1
        __nvoc_dtor_MIGMonitorSession+21
        __nvoc_objDelete+35
        clientFreeResource_IMPL+1600
        rmclientFreeResource_IMPL+77
        serverFreeResourceTreeUnderLock+180
        serverFreeResourceTree+806
        rmapiFreeWithSecInfo+269
        rmapiFreeWithSecInfoTls+73
        Nv01FreeWithSecInfo+78
        RmIoctl+693
        rm_ioctl+73
        nvidia_ioctl+1515
        nvidia_frontend_unlocked_ioctl+88
        __x64_sys_ioctl+160
        do_syscall_64+89
        entry_SYSCALL_64_after_hwframe+110


local-mtijanic@ipp1-1391:~$ sudo killall bpftrace 
[1]+  Done                    sudo bpftrace -e 'kprobe:nv_cap_close_fd { print(kstack); }'

Tried both versions, on two devices (desktop+A16 and notebook), but both with Ubuntu 23.10 since that's what my default test environment uses. If you can give me a bit more info about your setup, I can try something closer to it (but it may take a while).

[wxx213]

Thanks very much.

It seems that the kernel config caused the issue, here I give the config file in our kernel which is from /boot/config-4.18.0-147.mt20200626.413.el8_1.x86_64:

文件分享链接：https://wenshu.sankuai.com/file/share/download/98E1DD807099A5D1C323422C260E95307CCB0A98
提取密码：hg3aup

I hope you could get this file from our company server. @mtijanic

[mtijanic]

4.18 is a bit old, but according to https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md it should still (just barely) support stacktrace decoding. Maybe there was a bug in the initial implementation that was fixed in later kernel versions?

I'll give it a try with some older kernels when I get a chance, but it won't be for another few days at least. In the meantime, maybe you can try a newer kernel on some machine and see if it works for you.

[timur-tabi]

Kernel code needs to be compiled with -O2. That's just a general rule that's been around forever. There are functions that depend on being inlined in order to work properly.

[NeilZhy]

I encountered the same problem. I use the bcc tool stackcount to view the function __alloc_pages_nodemask opened by the dynamic physical memory of the algorithm module, but I can only obtain the function call stack in the kernel mode, and I cannot obtain the function call stack in the user mode. Is there any way to get the function call stack in user mode? Only in this way can I make targeted optimizations.The picture below is my flame graph. There is only the function call stack in the kernel mode and no function call stack in the user mode.