Skip to content

Block I/O on host monitoring counts each write up to 5x on RAID/LVM hosts (partitions and md/dm devices are summed) #5513

Description

@overflowy

To Reproduce

  1. Install Dokploy on a host with mdadm software RAID. Mine is a Hetzner box with 2 NVMe drives in RAID1, where nvme0n1pX and nvme1n1pX pair up into md0 to md3.
  2. Open Monitoring for the Dokploy server and read the Block I/O card.
  3. On the host, print lifetime writes per device:
    awk '$3 ~ /^(sd|nvme|md)/ {printf "%-10s %8.1f GB written\n", $3, $10*512/1e9}' /proc/diskstats
  4. Compare the dashboard total with the per-device numbers.

Current vs. Expected behavior

The dashboard says Write: 7292.09 GB:

Image

Here's /proc/diskstats on the same host:

nvme1n1      1466.2 GB written
nvme1n1p2     142.2 GB written
nvme1n1p4    1324.0 GB written
nvme0n1      1466.2 GB written
nvme0n1p2     142.2 GB written
nvme0n1p4    1324.0 GB written
md1           142.2 GB written
md3          1285.5 GB written
md0, md2, nvme*p1, nvme*p3 all 0.0 GB

Add up every row and you get the dashboard number almost exactly:

Rows GB written
2 whole disks 2932.4
their partitions 2932.4
md arrays 1427.7
Sum 7292.5

So one write from a filesystem shows up 5 times. It's counted on the md array, on both mirror disks, and on both mirror partitions. I spent a while chasing a 7 TB write storm that never happened.

I'd expect each byte to be counted once. That gives about 1428 GB if the card means logical I/O, which is what the filesystems wrote. It gives about 2932 GB if it means physical I/O, which is the real wear on the two drives. Either is fine, as long as it's one of them.

The cause is in packages/server/src/monitoring/utils.ts#L103-L114. It sums every row from disk.stats() and only skips loop, ram, sr and fd. Partitions go into the total, and so do stacked devices like md* and dm-*. LVM and LUKS setups create dm-* devices too, so they'd hit the same problem.

Provide environment information

Operating System:
  OS: Linux, mdadm RAID1 on 2x NVMe (md0 to md3)
Dokploy version: v0.30.7
VPS Provider: Hetzner, dedicated server
What applications/services are you tying to deploy?
  Docker Compose services: internal webhook handlers and background job workers,
  web analytics, BI dashboards, SSO, a password manager, uptime monitoring and log search

Which area(s) are affected? (Select all that apply)

Application

Are you deploying the applications where Dokploy is installed or on a remote server?

Same server where Dokploy is installed

Additional context

#5385 already reports the partition half of this ("counts each byte twice"). Someone opened #5399 to fix it, but it was closed without merging. That PR only drops partition rows. On my host the md rows would stay, so the card would still show about 4360 GB, roughly 3x the real value. Anyone on RAID, LVM or LUKS would keep seeing inflated numbers after that fix, which is why I'm filing this separately.

I'd pick devices from /sys/block rather than matching names with regexes. /sys/block only lists whole devices, so partitions drop out for free. From there:

  • For physical I/O, skip any device with entries in /sys/block/<dev>/slaves/, which catches md and dm. Also skip loop, ram and zram.
  • For logical I/O, keep only devices whose /sys/block/<dev>/holders/ is empty.

This works the same for RAID, LVM, LUKS and every naming scheme (sd, nvme, mmc), with no special cases for trailing digits.

Will you send a PR to fix it?

No

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions