Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxmox netdevmon does not work after boot but after a service restart it does #48

Open
docgalaxyblock opened this issue Jan 3, 2025 · 13 comments · Fixed by #51
Open

Comments

@docgalaxyblock
Copy link
Contributor

docgalaxyblock commented Jan 3, 2025

Hello folks,
first: awesome work on this 👍

I have a DXP8800Plus and just installed Proxmox today on it (proxmox-headers-6.8.12-5-pve).
The led controller install went smoothly but on a system restart my netdevmon does not sucessfully starts.
If I manually restart it some seconds later it starts sucessfully.

My journalctl -u [email protected] log
-- Boot 8c603897bebc4e41986c9a843b2a6717 --
Jan 03 12:28:32 DXP8800 systemd[1]: Starting [email protected] - UGREEN LEDs daemon for monitoring netio (of enp86s0) and blinking corresponding LEDs...
Jan 03 12:28:32 DXP8800 ugreen-probe-leds[650]: led_ugreen             24576  0
Jan 03 12:28:32 DXP8800 ugreen-probe-leds[631]: Found I2C device /dev/i2c-0
Jan 03 12:28:32 DXP8800 systemd[1]: Started [email protected] - UGREEN LEDs daemon for monitoring netio (of enp86s0) and blinking corresponding LEDs.
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 38: /sys/class/leds/netdev/trigger: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 39: /sys/class/leds/netdev/device_name: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 40: /sys/class/leds/netdev/link: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 41: /sys/class/leds/netdev/tx: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 42: /sys/class/leds/netdev/rx: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 43: /sys/class/leds/netdev/interval: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 44: /sys/class/leds/netdev/color: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 45: /sys/class/leds/netdev/brightness: No such file or directory
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[675]: cat: /sys/class/net/enp86s0/speed: Invalid argument
Jan 03 12:28:32 DXP8800 ugreen-netdevmon[666]: /usr/bin/ugreen-netdevmon: line 59: /sys/class/leds/netdev/color: No such file or directory
Jan 03 12:30:44 DXP8800 systemd[1]: Stopping [email protected] - UGREEN LEDs daemon for monitoring netio (of enp86s0) and blinking corresponding LEDs...
Jan 03 12:30:44 DXP8800 ugreen-netdevmon[666]: Terminated
Jan 03 12:30:44 DXP8800 systemd[1]: [email protected]: Deactivated successfully.
Jan 03 12:30:44 DXP8800 systemd[1]: Stopped [email protected] - UGREEN LEDs daemon for monitoring netio (of enp86s0) and blinking corresponding LEDs.
Jan 03 12:30:44 DXP8800 systemd[1]: Starting [email protected] - UGREEN LEDs daemon for monitoring netio (of enp86s0) and blinking corresponding LEDs...
Jan 03 12:30:44 DXP8800 ugreen-probe-leds[1650]: led_ugreen             24576  0
Jan 03 12:30:44 DXP8800 ugreen-probe-leds[1644]: Found I2C device /dev/i2c-0
Jan 03 12:30:44 DXP8800 systemd[1]: Started [email protected] - UGREEN LEDs daemon for monitoring netio (of enp86s0) and blinking corresponding LEDs.
My installation steps
apt install git build-essential proxmox-default-headers i2c-tools
git clone https://github.com/miskcoo/ugreen_leds_controller
cd ugreen_leds_controller/
wget https://github.com/miskcoo/ugreen_leds_controller/releases/download/v0.2/led-ugreen-dkms_0.2_amd64.deb
apt install -f ~/led-ugreen-dkms_0.2_amd64.deb 
cat > /etc/modules-load.d/ugreen-led.conf << EOF
i2c-dev
led-ugreen
ledtrig-oneshot
ledtrig-netdev
EOF
scripts=(ugreen-diskiomon ugreen-netdevmon ugreen-probe-leds); for f in ${scripts[@]}; do chmod +x "scripts/$f"; cp "scripts/$f" /usr/bin; done
cp scripts/ugreen-leds.conf /etc/ugreen-leds.conf
cp scripts/*.service /etc/systemd/system/
cd scripts/
g++ -std=c++17 -O2 blink-disk.cpp -o ugreen-blink-disk
cp ugreen-blink-disk /usr/bin
g++ -std=c++17 -O2 check-standby.cpp -o ugreen-check-standby
cp ugreen-check-standby /usr/bin
nano /etc/ugreen-leds.conf
systemctl enable --now ugreen-diskiomon
systemctl enable --now ugreen-netdevmon@enp86s0
reboot
systemctl status ugreen-diskiomon.service [email protected]
systemctl restart ugreen-diskiomon.service [email protected]

I see systemd dependencies were included in #23 so it should work right?
Or did I missed something during installation?

EDIT: Note After some reboot and shutdowns later my services do (not) work consisent.


Greetings from 🇩🇪 👋

@miskcoo
Copy link
Owner

miskcoo commented Jan 3, 2025

If restarting [email protected] would make it work, maybe inserting a sleep 2s below this line will resolve this issue.

@jrcheshire
Copy link

I've been encountering a very similar issue with my DXP4800 Plus on Debian 12. Either the network monitor or disk I/O monitor LEDs would work, but not both, and it seemed to be random which one worked each time. Errors of the form
ugreen-probe-leds[552]: /usr/bin/ugreen-probe-leds: line 14: echo: write error: Device or resource busy, and similarly restarting the service manually could recover whichever one didn't work.

I played around a bit with adding sleep statements per the above suggestion, initially a 2-second sleep below line 14 of ugreen-probe-leds as suggested, and played around with also adding a sleep before this line and making the sleep duration longer (went up to 5 seconds max). This didn't seem to have the desired effect, but what did was adding 3-second sleep statements at the top of both the ugreen-netdevmon and ugreen-diskiomon scripts.

@docgalaxyblock
Copy link
Contributor Author

Year I experimented a bit too (I have setup a fresh pve on a test ssh on my dxp8800).
If I disable netdevmon and diskiomon, restart it and then manually run /usr/bin/ugreen-probe-leds I get no error.
I am currently guessing both services get started nearly in parallel and therefore the probe-leds ExecStartPre gets executed nearly in parallel too and the "second" instanc of probe-leds gts the Ressource bussy error.

I will try to move the probe-leds into its own service and depend the original services on it.
Let's see if I get this changed and if it works.

@docgalaxyblock
Copy link
Contributor Author

I was able to catch two different errors with the current services:

A) ugreen-probe-leds: line 14: echo: write error: File exists
root@DXP8800:~# systemctl status -o short-iso-precise ugreen-{probe-led,diskiomon,netdevmon@vmbr0}
Unit ugreen-probe-led.service could not be found.
× ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LEDs
     Loaded: loaded (/etc/systemd/system/ugreen-diskiomon.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2025-01-04 00:10:21 CET; 1min 6s ago
    Process: 678 ExecStartPre=/usr/bin/ugreen-probe-leds (code=exited, status=1/FAILURE)
        CPU: 16ms

2025-01-04T00:10:21.795178+0100 DXP8800 systemd[1]: Starting ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LEDs...
2025-01-04T00:10:21.807703+0100 DXP8800 ugreen-probe-leds[702]: led_ugreen             24576  0
2025-01-04T00:10:21.811473+0100 DXP8800 ugreen-probe-leds[678]: Found I2C device /dev/i2c-0
2025-01-04T00:10:21.812329+0100 DXP8800 ugreen-probe-leds[678]: /usr/bin/ugreen-probe-leds: line 14: echo: write error: File exists
2025-01-04T00:10:21.811780+0100 DXP8800 systemd[1]: ugreen-diskiomon.service: Control process exited, code=exited, status=1/FAILURE
2025-01-04T00:10:21.811905+0100 DXP8800 systemd[1]: ugreen-diskiomon.service: Failed with result 'exit-code'.
2025-01-04T00:10:21.812060+0100 DXP8800 systemd[1]: Failed to start ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LEDs.

○ [email protected] - UGREEN LEDs daemon for monitoring netio (of vmbr0) and blinking corresponding LEDs
     Loaded: loaded (/etc/systemd/system/[email protected]; enabled; preset: enabled)
     Active: inactive (dead) since Sat 2025-01-04 00:10:22 CET; 1min 6s ago
   Duration: 54ms
    Process: 679 ExecStartPre=/usr/bin/ugreen-probe-leds (code=exited, status=0/SUCCESS)
    Process: 782 ExecStart=/usr/bin/ugreen-netdevmon vmbr0 (code=exited, status=0/SUCCESS)
   Main PID: 782 (code=exited, status=0/SUCCESS)
        CPU: 28ms

2025-01-04T00:10:21.795612+0100 DXP8800 systemd[1]: Starting [email protected] - UGREEN LEDs daemon for monitoring netio (of vmbr0) and blinking corresponding LEDs...
2025-01-04T00:10:21.808100+0100 DXP8800 ugreen-probe-leds[700]: led_ugreen             24576  0
2025-01-04T00:10:21.811209+0100 DXP8800 ugreen-probe-leds[679]: Found I2C device /dev/i2c-0
2025-01-04T00:10:22.542812+0100 DXP8800 systemd[1]: Started [email protected] - UGREEN LEDs daemon for monitoring netio (of vmbr0) and blinking corresponding LEDs.
2025-01-04T00:10:22.597234+0100 DXP8800 systemd[1]: [email protected]: Deactivated successfully.
B) ugreen-netdevmon: line 38: /sys/class/leds/netdev/trigger: No such file or directory ...
root@DXP8800:~# systemctl status -o short-iso-precise ugreen-{probe-led,diskiomon,netdevmon@vmbr0}
Unit ugreen-probe-led.service could not be found.
● ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LEDs
     Loaded: loaded (/etc/systemd/system/ugreen-diskiomon.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-01-03 23:53:08 CET; 12min ago
    Process: 684 ExecStartPre=/usr/bin/ugreen-probe-leds (code=exited, status=0/SUCCESS)
   Main PID: 785 (ugreen-diskiomo)
      Tasks: 6 (limit: 47734)
     Memory: 3.5M
        CPU: 19.286s
     CGroup: /system.slice/ugreen-diskiomon.service
             ├─  785 /usr/bin/bash /usr/bin/ugreen-diskiomon
             ├─  962 /usr/bin/bash /usr/bin/ugreen-diskiomon
             ├─  963 /usr/bin/bash /usr/bin/ugreen-diskiomon
             ├─24383 sleep 360s
             ├─25532 sleep 5s
             └─25589 sleep 0.1s

2025-01-03T23:53:07.654387+0100 DXP8800 systemd[1]: Starting ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LEDs...
2025-01-03T23:53:07.666460+0100 DXP8800 ugreen-probe-leds[705]: led_ugreen             24576  0
2025-01-03T23:53:07.670388+0100 DXP8800 ugreen-probe-leds[684]: Found I2C device /dev/i2c-0
2025-01-03T23:53:08.341765+0100 DXP8800 systemd[1]: Started ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LEDs.
2025-01-03T23:53:08.347506+0100 DXP8800 ugreen-diskiomon[785]: Found UGREEN DXP8800 series
2025-01-03T23:53:08.351322+0100 DXP8800 ugreen-diskiomon[785]: Enumerating disks based on ata...
2025-01-03T23:53:08.353020+0100 DXP8800 ugreen-diskiomon[785]: ata ata2 >> sda

○ [email protected] - UGREEN LEDs daemon for monitoring netio (of vmbr0) and blinking corresponding LEDs
     Loaded: loaded (/etc/systemd/system/[email protected]; enabled; preset: enabled)
     Active: inactive (dead) since Fri 2025-01-03 23:53:07 CET; 12min ago
   Duration: 9ms
    Process: 685 ExecStartPre=/usr/bin/ugreen-probe-leds (code=exited, status=0/SUCCESS)
    Process: 721 ExecStart=/usr/bin/ugreen-netdevmon vmbr0 (code=exited, status=0/SUCCESS)
   Main PID: 721 (code=exited, status=0/SUCCESS)
        CPU: 23ms

2025-01-03T23:53:07.654834+0100 DXP8800 systemd[1]: Starting [email protected] - UGREEN LEDs daemon for monitoring netio (of vmbr0) and blinking corresponding LEDs...
2025-01-03T23:53:07.665316+0100 DXP8800 ugreen-probe-leds[703]: led_ugreen             24576  0
2025-01-03T23:53:07.670601+0100 DXP8800 ugreen-probe-leds[685]: Found I2C device /dev/i2c-0
2025-01-03T23:53:07.671572+0100 DXP8800 systemd[1]: Started [email protected] - UGREEN LEDs daemon for monitoring netio (of vmbr0) and blinking corresponding LEDs.
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 38: /sys/class/leds/netdev/trigger: No such file or directory
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 39: /sys/class/leds/netdev/device_name: No such file or directory
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 40: /sys/class/leds/netdev/link: No such file or directory
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 41: /sys/class/leds/netdev/tx: No such file or directory
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 42: /sys/class/leds/netdev/rx: No such file or directory
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 43: /sys/class/leds/netdev/interval: No such file or directory
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 44: /sys/class/leds/netdev/color: No such file or directory
2025-01-03T23:53:07.679941+0100 DXP8800 ugreen-netdevmon[721]: /usr/bin/ugreen-netdevmon: line 45: /sys/class/leds/netdev/brightness: No such file or directory
2025-01-03T23:53:07.680859+0100 DXP8800 systemd[1]: [email protected]: Deactivated successfully.

I am guessing case A) happens cause of the parallelism of the probe-leds ExecStartPre

In case B) I am not sure...

@miskcoo
Copy link
Owner

miskcoo commented Jan 4, 2025

I think the second issue arises because there is no additional waiting time for the driver's initialization. How about adding sleep 2s, as mentioned earlier? Or adding another ExecStartPre=sleep 2s after the probe-leds.

@docgalaxyblock
Copy link
Contributor Author

docgalaxyblock commented Jan 4, 2025

I got it - I moved the probe-leds into its own service of the type oneshot and depend the diskmon & diskiomon thanks to this post since I did not looked into systemd yet:
https://trstringer.com/simple-vs-oneshot-systemd-service

Now the dependency is correct and it correctly started multiple times in my case
root@DXP8800:~# systemctl status --full --output short-iso-precise --lines 100 systemd-modules-load ugreen-{probe-leds,diskiomon,netdevmon@{enp88s0,vmbr0}}

● systemd-modules-load.service - Load Kernel Modules
     Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static)
     Active: active (exited) since Sat 2025-01-04 17:17:32 CET; 5min ago
       Docs: man:systemd-modules-load.service(8)
             man:modules-load.d(5)
    Process: 402 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status=0/SUCCESS)
   Main PID: 402 (code=exited, status=0/SUCCESS)
        CPU: 1.252s

2025-01-04T17:17:30.773258+0100 DXP8800 systemd-modules-load[402]: Inserted module 'vhost_net'
2025-01-04T17:17:30.773599+0100 DXP8800 systemd-modules-load[402]: Module 'i2c_dev' is built in
2025-01-04T17:17:30.776889+0100 DXP8800 systemd-modules-load[402]: Inserted module 'led_ugreen'
2025-01-04T17:17:30.779842+0100 DXP8800 systemd-modules-load[402]: Inserted module 'ledtrig_oneshot'
2025-01-04T17:17:30.781978+0100 DXP8800 systemd-modules-load[402]: Inserted module 'ledtrig_netdev'
2025-01-04T17:17:32.155216+0100 DXP8800 systemd-modules-load[402]: Inserted module 'zfs'
2025-01-04T17:17:32.155955+0100 DXP8800 systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules.
Notice: journal has been rotated since unit was started, output may be incomplete.

● ugreen-probe-leds.service - UGREEN LEDs daemon for probing the hardware LEDs
     Loaded: loaded (/etc/systemd/system/ugreen-probe-leds.service; disabled; preset: enabled)
     Active: active (exited) since Sat 2025-01-04 17:17:33 CET; 5min ago
    Process: 686 ExecStart=/usr/bin/ugreen-probe-leds (code=exited, status=0/SUCCESS)
   Main PID: 686 (code=exited, status=0/SUCCESS)
        CPU: 19ms

2025-01-04T17:17:32.811992+0100 DXP8800 systemd[1]: Starting ugreen-probe-leds.service - UGREEN LEDs daemon for probing the hardware LEDs...
2025-01-04T17:17:32.822496+0100 DXP8800 ugreen-probe-leds[701]: led_ugreen             24576  0
2025-01-04T17:17:32.826199+0100 DXP8800 ugreen-probe-leds[686]: Found I2C device /dev/i2c-0
2025-01-04T17:17:33.603402+0100 DXP8800 systemd[1]: Finished ugreen-probe-leds.service - UGREEN LEDs daemon for probing the hardware LEDs.

● ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LED
     Loaded: loaded (/etc/systemd/system/ugreen-diskiomon.service; enabled; preset: enabled)
     Active: active (running) since Sat 2025-01-04 17:17:33 CET; 5min ago
   Main PID: 781 (ugreen-diskiomo)
      Tasks: 6 (limit: 47734)
     Memory: 3.2M
        CPU: 8.472s
     CGroup: /system.slice/ugreen-diskiomon.service
             ├─  781 /usr/bin/bash /usr/bin/ugreen-diskiomon
             ├─ 1019 /usr/bin/bash /usr/bin/ugreen-diskiomon
             ├─ 1020 /usr/bin/bash /usr/bin/ugreen-diskiomon
             ├─ 1028 sleep 360s
             ├─11470 sleep 5s
             └─11612 sleep 0.1s

2025-01-04T17:17:33.620952+0100 DXP8800 systemd[1]: Started ugreen-diskiomon.service - UGREEN LEDs daemon for monitoring diskio and blinking corresponding LED.
2025-01-04T17:17:33.627303+0100 DXP8800 ugreen-diskiomon[781]: Found UGREEN DXP8800 series
2025-01-04T17:17:33.631036+0100 DXP8800 ugreen-diskiomon[781]: Enumerating disks based on ata...
2025-01-04T17:17:33.632752+0100 DXP8800 ugreen-diskiomon[781]: ata ata2 >> sda

● [email protected] - UGREEN LEDs daemon for monitoring netio (of enp88s0) and blinking corresponding LEDs
     Loaded: loaded (/etc/systemd/system/[email protected]; enabled; preset: enabled)
     Active: active (running) since Sat 2025-01-04 17:17:33 CET; 5min ago
   Main PID: 782 (ugreen-netdevmo)
      Tasks: 2 (limit: 47734)
     Memory: 628.0K
        CPU: 18ms
     CGroup: /system.slice/system-ugreen\x2dnetdevmon.slice/[email protected]
             ├─  782 /usr/bin/bash /usr/bin/ugreen-netdevmon enp88s0
             └─10804 sleep 60s

2025-01-04T17:17:33.621571+0100 DXP8800 systemd[1]: Started [email protected] - UGREEN LEDs daemon for monitoring netio (of enp88s0) and blinking corresponding LEDs.

○ [email protected] - UGREEN LEDs daemon for monitoring netio (of vmbr0) and blinking corresponding LEDs
     Loaded: loaded (/etc/systemd/system/[email protected]; disabled; preset: enabled)
     Active: inactive (dead)

Since I do not know systemd yet and reading the mentioned post I think that the [email protected] could be switched to oneshot with RemainAfterExit too.

What are your opinions?

@docgalaxyblock
Copy link
Contributor Author

Hey @jrcheshire @miskcoo and maybe @KrzysztofHajdamowicz Would you like to test my changes before I open a PR?

https://github.com/docgalaxyblock/ugreen_leds_controller

@miskcoo
Copy link
Owner

miskcoo commented Jan 5, 2025

Would you like to test my changes before I open a PR?

I can test that recently.

Since I do not know systemd yet and reading the mentioned post I think that the [email protected] could be switched to oneshot with RemainAfterExit too.

I will also check that.

@miskcoo
Copy link
Owner

miskcoo commented Jan 10, 2025

Hi @docgalaxyblock, I have tested the changes in the repo, and they work well.

For the [email protected], it behaves similar to diskiomon.service since there is a loop for checking the network connectivity, so I think it should be a service of type simple.

@docgalaxyblock
Copy link
Contributor Author

For the [email protected], it behaves similar to diskiomon.service since there is a loop for checking the network connectivity, so I think it should be a service of type simple.

@miskcoo It depends if you have enabled CHECK_LINK or not. If not it is likely a "oneshot" type of service.

@miskcoo
Copy link
Owner

miskcoo commented Jan 11, 2025

For the [email protected], it behaves similar to diskiomon.service since there is a loop for checking the network connectivity, so I think it should be a service of type simple.

@miskcoo It depends if you have enabled CHECK_LINK or not. If not it is likely a "oneshot" type of service.

By looking the link about systemd, the oneshot type with remainafterexit leaves the service to the activating state with check_link enabled. It appears that the requirements for these two choices are slightly conflict?

@miskcoo miskcoo reopened this Jan 11, 2025
@docgalaxyblock
Copy link
Contributor Author

It appears that the requirements for these two choices are slightly conflict?
Year I interpret it like that too.

@docgalaxyblock
Copy link
Contributor Author

I understand it like that too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants