periodic few hardware drops around every 1h:30min using FromDPDKDevice

[gitak83]

Hello,

I am using FastClick to capture packets at a rate of around 10 million packets per second (pps) on an E810 NIC, with a single thread and isolated cores. I am also using the FromDPDKDevice handler to monitor hardware drop counts.

The issue is that approximately every 1 hour and 30 minutes, I experience a burst of packet drops—around 300,000 to 500,000 hardware drops reported by FromDPDKDevice. After each burst, there are no further drops until the next 1.5-hour interval. This pattern repeats periodically.

There is no difference in the click script performance and drop rate with or without Real time scheduling on.

The script is so simple. it just uses FromDPDKDevice to capture the packets then it uses -> Discard() to discard the packets.
here are my DPDK configurations in the script.

DPDKInfo(NB_MBUF 33554431, MBUF_SIZE 1518, RING_SIZE 16777216, MBUF_CACHE_SIZE 512, PAYLOAD_RING_SIZE 32768)
FromDPDKDevice($rxport, VERBOSE 99, PROMISC 1, NDESC 4096, MAXTHREADS 1, N_QUEUES 1, THREADOFFSET 1, NUMA true, NUMA_NODE 0, ACTIVE 1) -> Discard();

I have allocated 1GB hugepage and isolated cores 1-16

Command line: BOOT_IMAGE=/vmlinuz-5.15.0-119-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro intel_iommu=on iommu=pt isolcpus=1-16 nohz_full=1-16 rcu_nocbs=1-16 default_hugepagesz=1G hugepages=64 acpi=on selinux=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=0 intel_pstate=on nopti nospec_store_bypass_disable nospectre_v2 nospectre_v1 nospec l1tf=off netcfg/do_not_use_netplan=true mitigations=off

below are the dpdk configs given while running the click script. DPDK version is 24.11.3
--dpdk -l 1-2 --socket-mem=64000,0 --socket-limit=64000,0 --

here is the configurations for compiling fastclick.

RTE_SDK=/home/hara/dpdk-stable-24.11.3/ ./configure --enable-dpdk --enable-intel-cpu --verbose --enable-select=poll "CFLAGS=-O3 -ls
entry_shm" "CXXFLAGS=-std=c++17 -O3 -lre2 -lsentry_shm" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --enable-local --disable-task-stats --enable-cpu-load -
-disable-clone --disable-dpdk-softqueue --enable-user-timestamp --enable-flow  --enable-dpdk-packet

Please let me know if you need any further details.

How can this problem be solved to be able to capture packets without drop.

[tbarbette]

You mean the hw_dropped handler of the element, right?
How do you read this statistic?

Eg modify to have :

info :: DPDKInfo...
fd :: FromDPDKDevice ...

Script(TYPE ACTIVE, print $(now), read fd.dropped, read fd.hw_dropped,read fd.xstats out_of_buffers, read fd.xstats rx_discards_phy, read load, read info.pool_count, wait 1s, loop);

To print a few interesting statistics every second. I draw it from memory if some counter is not found remove it or come back here, or check in the code :)

I can't think of any mechanism in FastClick that would trigger at that time scale. Maybe remove --enable-user-timestamp just to be sure. This effect likely comes from something out of FastClick.

Also I wouldn't set MBUF_SIZE 1518, leave the default. From memory, I'm not sure if that numbers includes the headroom and/or the size of the DPDK structure.

Good luck.

[gitak83]

thank you so much for your response.

I removed --enable-user-timestamp from configuration, new compilation and did what you said and test a new click script as below:

d_info :: DPDKInfo(NB_MBUF 8388607)
receiveIN1 :: FromDPDKDevice(0000:3e:00.1, VERBOSE 99, PROMISC 1, NDESC 4096, MAXTHREADS 1, N_QUEUES 1, THREADOFFSET 1);

receiveIN1 -> Discard;

Script(
    TYPE ACTIVE, 
    print "Time: $(now)\n", 

    read receiveIN1.hw_dropped,
    read receiveIN1.xstats,
    read d_info.pools,
    read d_info.pool_count,
   
    wait 1s,
    loop
);

still periodic packet miss was seen. the first packet miss was seen after 4803 seconds.

this is the output of the first iteration and the iteration after it, where packet miss has been seen:

Time: 1767115846.850843131
receiveIN1.hw_dropped:
0
receiveIN1.xstats:
rx_good_packets[0] = 39649287381
tx_good_packets[1] = 0
rx_good_bytes[2] = 3858123449899
tx_good_bytes[3] = 0
rx_missed_errors[4] = 0
rx_errors[5] = 0
tx_errors[6] = 0
rx_mbuf_allocation_errors[7] = 0
rx_q0_packets[8] = 0
rx_q0_bytes[9] = 0
rx_q0_errors[10] = 0
tx_q0_packets[11] = 0
tx_q0_bytes[12] = 0
rx_unicast_packets[13] = 39649286646
rx_multicast_packets[14] = 159
rx_broadcast_packets[15] = 0
rx_dropped_packets[16] = 0
rx_unknown_protocol_packets[17] = 0
tx_unicast_packets[18] = 0
tx_multicast_packets[19] = 0
tx_broadcast_packets[20] = 0
tx_dropped_packets[21] = 0
tx_mbuf_error_packets[22] = 0
tx_link_down_dropped[23] = 0
rx_crc_errors[24] = 0
rx_illegal_byte_errors[25] = 0
rx_error_bytes[26] = 0
mac_local_errors[27] = 0
mac_remote_errors[28] = 0
rx_len_errors[29] = 0
tx_xon_packets[30] = 0
rx_xon_packets[31] = 0
tx_xoff_packets[32] = 0
rx_xoff_packets[33] = 0
rx_size_64_packets[34] = 0
rx_size_65_to_127_packets[35] = 37336920545
rx_size_128_to_255_packets[36] = 2281287701
rx_size_256_to_511_packets[37] = 26772759
rx_size_512_to_1023_packets[38] = 4305928
rx_size_1024_to_1522_packets[39] = 0
rx_size_1523_to_max_packets[40] = 0
rx_undersized_errors[41] = 0
rx_oversize_errors[42] = 0
rx_mac_short_pkt_dropped[43] = 0
rx_fragmented_errors[44] = 0
rx_jabber_errors[45] = 0
tx_size_64_packets[46] = 0
tx_size_65_to_127_packets[47] = 0
tx_size_128_to_255_packets[48] = 0
tx_size_256_to_511_packets[49] = 0
tx_size_512_to_1023_packets[50] = 0
tx_size_1024_to_1522_packets[51] = 0
tx_size_1523_to_max_packets[52] = 0

d_info.pools:
click_mempool_0

d_info.pool_count:
0 8384534

Time: 1767115847.851332499
receiveIN1.hw_dropped:
322717
receiveIN1.xstats:
rx_good_packets[0] = 39657112952
tx_good_packets[1] = 0
rx_good_bytes[2] = 3858916862114
tx_good_bytes[3] = 0
rx_missed_errors[4] = 322717
rx_errors[5] = 0
tx_errors[6] = 0
rx_mbuf_allocation_errors[7] = 0
rx_q0_packets[8] = 0
rx_q0_bytes[9] = 0
rx_q0_errors[10] = 0
tx_q0_packets[11] = 0
tx_q0_bytes[12] = 0
rx_unicast_packets[13] = 39657434870
rx_multicast_packets[14] = 159
rx_broadcast_packets[15] = 0
rx_dropped_packets[16] = 0
rx_unknown_protocol_packets[17] = 0
tx_unicast_packets[18] = 0
tx_multicast_packets[19] = 0
tx_broadcast_packets[20] = 0
tx_dropped_packets[21] = 0
tx_mbuf_error_packets[22] = 0
tx_link_down_dropped[23] = 0
rx_crc_errors[24] = 0
rx_illegal_byte_errors[25] = 0
rx_error_bytes[26] = 0
mac_local_errors[27] = 0
mac_remote_errors[28] = 0
rx_len_errors[29] = 0
tx_xon_packets[30] = 0
rx_xon_packets[31] = 0
tx_xoff_packets[32] = 0
rx_xoff_packets[33] = 0
rx_size_64_packets[34] = 0
rx_size_65_to_127_packets[35] = 37344588780
rx_size_128_to_255_packets[36] = 2281761254
rx_size_256_to_511_packets[37] = 26778296
rx_size_512_to_1023_packets[38] = 4306827
rx_size_1024_to_1522_packets[39] = 0
rx_size_1523_to_max_packets[40] = 0
rx_undersized_errors[41] = 0
rx_oversize_errors[42] = 0
rx_mac_short_pkt_dropped[43] = 0
rx_fragmented_errors[44] = 0
rx_jabber_errors[45] = 0
tx_size_64_packets[46] = 0
tx_size_65_to_127_packets[47] = 0
tx_size_128_to_255_packets[48] = 0
tx_size_256_to_511_packets[49] = 0
tx_size_512_to_1023_packets[50] = 0
tx_size_1024_to_1522_packets[51] = 0
tx_size_1523_to_max_packets[52] = 0

d_info.pools:
click_mempool_0

d_info.pool_count:
0 8384563

Time: 1767115848.851843149
receiveIN1.hw_dropped:
429853
receiveIN1.xstats:
rx_good_packets[0] = 39665123513
tx_good_packets[1] = 0
rx_good_bytes[2] = 3859707326583
tx_good_bytes[3] = 0
rx_missed_errors[4] = 429853
rx_errors[5] = 0
tx_errors[6] = 0
rx_mbuf_allocation_errors[7] = 0
rx_q0_packets[8] = 0
rx_q0_bytes[9] = 0
rx_q0_errors[10] = 0
tx_q0_packets[11] = 0
tx_q0_bytes[12] = 0
rx_unicast_packets[13] = 39665552566
rx_multicast_packets[14] = 160
rx_broadcast_packets[15] = 0
rx_dropped_packets[16] = 0
rx_unknown_protocol_packets[17] = 0
tx_unicast_packets[18] = 0
tx_multicast_packets[19] = 0
tx_broadcast_packets[20] = 0
tx_dropped_packets[21] = 0
tx_mbuf_error_packets[22] = 0
tx_link_down_dropped[23] = 0
rx_crc_errors[24] = 0
rx_illegal_byte_errors[25] = 0
rx_error_bytes[26] = 0
mac_local_errors[27] = 0
mac_remote_errors[28] = 0
rx_len_errors[29] = 0
tx_xon_packets[30] = 0
rx_xon_packets[31] = 0
tx_xoff_packets[32] = 0
rx_xoff_packets[33] = 0
rx_size_64_packets[34] = 0
rx_size_65_to_127_packets[35] = 37352228766
rx_size_128_to_255_packets[36] = 2282232375
rx_size_256_to_511_packets[37] = 26783963
rx_size_512_to_1023_packets[38] = 4307750
rx_size_1024_to_1522_packets[39] = 0
rx_size_1523_to_max_packets[40] = 0
rx_undersized_errors[41] = 0
rx_oversize_errors[42] = 0
rx_mac_short_pkt_dropped[43] = 0
rx_fragmented_errors[44] = 0
rx_jabber_errors[45] = 0
tx_size_64_packets[46] = 0
tx_size_65_to_127_packets[47] = 0
tx_size_128_to_255_packets[48] = 0
tx_size_256_to_511_packets[49] = 0
tx_size_512_to_1023_packets[50] = 0
tx_size_1024_to_1522_packets[51] = 0
tx_size_1523_to_max_packets[52] = 0

d_info.pools:
click_mempool_0

d_info.pool_count:
0 8384567

Time: 1767115849.852347287
receiveIN1.hw_dropped:
429853
receiveIN1.xstats:
rx_good_packets[0] = 39673391225
tx_good_packets[1] = 0
rx_good_bytes[2] = 3860512476795
tx_good_bytes[3] = 0
rx_missed_errors[4] = 429853
rx_errors[5] = 0
tx_errors[6] = 0
rx_mbuf_allocation_errors[7] = 0
rx_q0_packets[8] = 0
rx_q0_bytes[9] = 0
rx_q0_errors[10] = 0
tx_q0_packets[11] = 0
tx_q0_bytes[12] = 0
rx_unicast_packets[13] = 39673820247
rx_multicast_packets[14] = 160
rx_broadcast_packets[15] = 0
rx_dropped_packets[16] = 0
rx_unknown_protocol_packets[17] = 0
tx_unicast_packets[18] = 0
tx_multicast_packets[19] = 0
tx_broadcast_packets[20] = 0
tx_dropped_packets[21] = 0
tx_mbuf_error_packets[22] = 0
tx_link_down_dropped[23] = 0
rx_crc_errors[24] = 0
rx_illegal_byte_errors[25] = 0
rx_error_bytes[26] = 0
mac_local_errors[27] = 0
mac_remote_errors[28] = 0
rx_len_errors[29] = 0
tx_xon_packets[30] = 0
rx_xon_packets[31] = 0
tx_xoff_packets[32] = 0
rx_xoff_packets[33] = 0
rx_size_64_packets[34] = 0
rx_size_65_to_127_packets[35] = 37360009320
rx_size_128_to_255_packets[36] = 2282712877
rx_size_256_to_511_packets[37] = 26789641
rx_size_512_to_1023_packets[38] = 4308708
rx_size_1024_to_1522_packets[39] = 0
rx_size_1523_to_max_packets[40] = 0
rx_undersized_errors[41] = 0
rx_oversize_errors[42] = 0
rx_mac_short_pkt_dropped[43] = 0
rx_fragmented_errors[44] = 0
rx_jabber_errors[45] = 0
tx_size_64_packets[46] = 0
tx_size_65_to_127_packets[47] = 0
tx_size_128_to_255_packets[48] = 0
tx_size_256_to_511_packets[49] = 0
tx_size_512_to_1023_packets[50] = 0
tx_size_1024_to_1522_packets[51] = 0
tx_size_1523_to_max_packets[52] = 0

d_info.pools:
click_mempool_0

d_info.pool_count:
0 8384567

Are there any further recommendations I can take to avoid these packet drops?

[tbarbette]

Thanks for your detailed feedback!

You can:

• Increase NDESC to 32768 if that's supported on E810.
• Use more cores (remove MAXTHREADS 1, N_QUEUES 1, THREADOFFSET 1 and just set -l 1-4 in the cmdline to start with 3 cores) and see if that helps?

Did you observe such drops with another DPDK tool? Like testpmd maybe?

I have never observed this but I rarely run click that long. There is definitely no garbage collection or hashtables that needs to be rebalanced or things like that involved here.

Could you also compile without --enable-cpu-load -
-disable-clone --disable-dpdk-softqueue --enable-user-timestamp --enable-flow --enable-dpdk-packet and be sure to do a full make clean and make to rule out any of those? But I hardly doubt it.

My best guess is something spawning on the OS side. Altough with isolcpus you should be safe. I would normally recommend profiling but profiling at that specific point after so much time seems impossible.