Kernel kworker processes stored with a different name instead of the kworker specific name for that pID

[cpl-cmd]

System

OS: Red Hat 9.5 (and before)
collectl version: collectl V4.3.20 (and before)

Problem

Having a Linux Red Hat host and the collectl configured to generate prc files, the kworker processes are stored with other names instead of the actual kworker process name.

Example

Here is an example on how to reproduce the issue with a Red Hat 9.5 and collectl V4.3.20:

1. Update the default collectl.conf to generate prc files, how often to store the data, etc. Before starting the collect, for example:
   DaemonCommands = -f /var/log/collectl -r00:00,15 -m -F60 -s+cdmnCDMNZ -i5:10 -P

2. Start the collectl normally (in this example we are running it as a service) with the following command:
   systemctl start collectl

3. Check that the following example in /var/log/collect/ the expected prc files are being generated:

[root@redhat collectl]# ls -lrt
total 232
-rw-r--r--. 1 root root    275 May 29 14:48 redhat-collectl-202505.log
-rw-r--r--. 1 root root  17413 May 29 15:22 redhat-20250529.tab.gz
-rw-r--r--. 1 root root 172036 May 29 15:22 redhat-20250529.prc.gz
-rw-r--r--. 1 root root   5437 May 29 15:22 redhat-20250529.numa.gz
-rw-r--r--. 1 root root   3111 May 29 15:22 redhat-20250529.net.gz
-rw-r--r--. 1 root root   7365 May 29 15:22 redhat-20250529.dsk.gz
-rw-r--r--. 1 root root   4964 May 29 15:22 redhat-20250529.cpu.gz

4. To trigger a kworker process, a small script like the following one was used to trigger I/O in a user home directory (a user named “system” was used in this example):

[root@redhat collectl]# cat /home/system/generate_big_files.sh
while true; do
  date
  dd if=/dev/zero of=/home/system/testfile bs=10M count=100
  rm /home/system/testfile
done

5. The script is started in background or in another terminal with the user “system”

6. Check the prc log file, see that there are “root” and user “system” processes with different pIDs:

[root@redhat collectl]# zcat redhat-20250529.prc.gz | grep testfile
20250529 15:14:00 2689 system 20 2684 0 D 15868 0 12164 10476 132 40 1652 0 0.51 0.00 5 0:00:00 0 92614 93184 92160 10 9 0 0 12 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:10 2720 system 20 2684 0 R 15868 0 12160 10476 132 40 1652 1 0.16 0.00 1 0:00:00 0 33966 34816 33792 4 3 0 0 12 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:20 2743 root 20 2 0 I 15868 0 12164 10476 132 40 1652 0 0.00 0.00 0 0:00:00 0 0 0 0 0 0 0 0 0 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:20 2749 system 20 2684 0 D 15868 0 12164 10476 132 40 1652 1 0.49 0.00 4 0:00:00 0 81186 81920 80896 9 7 0 0 12 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:30 2743 root 20 2 0 I 15868 0 12164 10476 132 40 1652 0 0.00 0.00 0 0:00:00 0 0 0 0 0 0 0 0 0 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:30 2780 system 20 2684 0 R 15868 0 12160 10476 132 40 1652 1 0.05 0.00 0 0:00:00 0 12446 13312 12288 2 1 0 0 12 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:40 2743 root 20 2 0 I 15868 0 12164 10476 132 40 1652 0 0.00 0.00 0 0:00:00 0 0 0 0 0 0 0 0 0 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:40 2808 system 20 2684 0 R 15868 0 12164 10476 132 40 1652 1 0.48 0.00 4 0:00:00 0 91136 91136 91136 10 8 0 0 12 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:50 2743 root 20 2 0 I 15868 0 12164 10476 132 40 1652 0 0.00 0.00 0 0:00:00 0 0 0 0 0 0 0 0 0 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:14:50 2838 system 20 2684 0 R 15868 0 12160 10476 132 40 1652 1 0.47 0.00 4 0:00:00 0 93778 94208 93184 10 9 0 0 12 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:15:00 2743 root 20 2 0 I 15868 0 12164 10476 132 40 1652 0 0.00 0.00 0 0:00:00 0 0 0 0 0 0 0 0 0 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
20250529 15:15:00 2868 system 20 2684 0 R 15868 0 12164 10476 132 40 1652 0 0.51 0.00 5 0:00:00 0 91480 92160 91136 10 8 0 0 12 dd if=/dev/zero of=/home/system/testfile bs=10M count=100
 gzip: redhat-20250529.prc.gz: unexpected end of file

7. It can be checked that the root pID (2743 in this example) is the kworker process from the kernel instead of the "dd if=/dev/zero of=/home/system/testfile bs=10M count=100" of the previous step.

[root@redhat collectl]# ps -leyf | grep 2743
S system      1750    1748  0  80   0  7504 27432 do_sig 14:21 ?        00:00:00 (sd-pam)
I root        2743       2  0  80   0     0     0 worker 15:14 ?        00:00:00 [kworker/u8:3-events_unbound]
S root        3040    2627  0  80   0  2432  1601 pipe_r 15:15 pts/1    00:00:00 grep --color=auto 2743

[loberman]

Only just getting to this. Was on PTO and had lots of catchup

[loberman]

I started looking at this and I am puzzled. The kworker processes should only ever run as kernel threads as the root user and that is what I see. You are wanting to see the kworker running as the system user and that will never happen.

I will close this out.

In Linux, kworker threads are kernel worker threads, which are part of the kernel infrastructure for handling deferred work in process context (like workqueues, I/O completions, timers, etc.). They are:

Created and managed by the kernel
Always run in kernel space
Execute as user root (UID 0), since they're part of the kernel

[cpl-cmd]

I started looking at this and I am puzzled. The kworker processes should only ever run as kernel threads as the root user and that is what I see. You are wanting to see the kworker running as the system user and that will never happen.

I will close this out.

In Linux, kworker threads are kernel worker threads, which are part of the kernel infrastructure for handling deferred work in process context (like workqueues, I/O completions, timers, etc.). They are:

Created and managed by the kernel Always run in kernel space Execute as user root (UID 0), since they're part of the kernel

I don't understand your answer.
What I meant is that the kernel root process "[kworker/u8:3-events_unbound]" which is shown clearly with the ps command with PID 2743 should be stored with that process name ("[kworker/u8:3-events_unbound]") instead of another name of another process in this case "dd if=/dev/zero of=/home/system/testfile bs=10M count=100".

I feel that there is a misunderstanding of the issue. The stored information of the process names is not coherent.

[loberman]

Indeed, misunderstanding was mine
I read it as you saying the kworker was not running as the system user so that was my mistake not yours.
I will revisit to understand what could be happening here

[loberman]

re-opened the issue and investigating further

[loberman]

OK, issue is reproduced, looking to see why this is happening.
It only happens with the prc capture and not if playing back a standard raw.gz using -sZ
Will work on it and see if I can figure out why it is happening.
Its not an area of the code I spend much time in as all since I took over maintaining collectl so will need some code review.

[loberman]

Correction, if I wait long enough for multiple 60s deltas. it happens also with playback -sZ of the raw.gz so common collection code has the issue
Working on it