How to push the server to 100% CPU?

[WuBingzheng]

I wrote a simple echo service based on tokio. It spawns a task for each connect received.

Then I wrote a client program that can specify N tasks to request the echo service simultaneously. Each task establishes a persistent connection and sequentially makes several requests.

I wanted to use this client to maximize the CPU usage of the echo service, but I was never able to do so.

I'm using an M1 MacBook. The echo service automatically started 8 threads. The client program runs at the same computer.

Here is the cpu usage for different number of client tasks:

tasks of  | CPU usage(%) of
client    | echo server
----------+----------------
  1       |  34
  2       |  54
  3       |  72
  4       |  99
  5       |  116
  6       |  127
  7       |  134
  8       |  137
  9       |  137
 10       |  137

You can see than the CPU stops at 137% with 8 client-tasks.

However I wish to push the server and the client to use 400% CPU both, to run out of the total 8 cores.

Is it possible? How to make it?

---

I set the threads of echo server, and test it:

threads of  | CPU usage(%) of
echo server | echo server
------------+----------------
 1          | 86
 2          | 130
 3          | 132
 4          | 136
 5+         | 136

You can see the echo server can not use 100% CPU even for 1 thread.
2 threads is better than 1 thread. But after 2+ threads, more threads do not help to increase the echo-server's performance.

---

I found a similar discussion: #5779, but there is no further progress.

Here is the echo server's code:

use tokio::net::TcpListener;
use tokio::io::{AsyncReadExt, AsyncWriteExt};

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("127.0.0.1:2345").await.unwrap();
    loop {
        let (mut socket, _) = listener.accept().await.unwrap();
        tokio::spawn(async move {
            let mut buf = Vec::with_capacity(200);
            while let Ok(len) = socket.read_buf(&mut buf).await {
                if len == 0 {
                    break;
                }
                if let Err(_) = socket.write_all(&buf).await { // echo the request
                    break;
                }
                buf.clear();
            }
        });
    }
}

and the client's code:

use std::env;
use std::time::Instant;
use tokio::net::TcpStream;
use tokio::io::{AsyncWriteExt, AsyncReadExt, self};

#[tokio::main]
async fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() != 4 {
        println!("Usage: {} server-address #task #request", args[0]);
        return;
    }
    let ntask = args[2].parse::<usize>().unwrap();
    let nreq = args[3].parse::<usize>().unwrap();

    let mut handlers = Vec::new();
    for _ in 0..ntask {
        let mut conn = TcpStream::connect(&args[1]).await.unwrap();
        handlers.push(tokio::spawn(async move {
            let start = Instant::now();
            for _ in 0..nreq {
                request(&mut conn).await.unwrap();
            }
            println!("task time: {:?}", start.elapsed());
        }));
    }

    for h in handlers {
        let _ = h.await;
    }
}

async fn request(stream: &mut TcpStream) -> io::Result<()> {
    stream.write_all("hello, tokio!".as_bytes()).await?;

    let mut buf = [0; 100];
    stream.read(&mut buf).await?;
    Ok(())
}

[wathenjiang]

You may set the number of worker threads to a greater number, to make more worker threads are running instead of IO waiting.

And if your machine are running other process (the desktop computer always have), the tokio process cant take all cpu time because of OS preemptive schedule mechanism.

[WuBingzheng]

top - 01:56:05 up 1 day, 14:06,  6 users,  load average: 2.69, 0.86, 0.30
Tasks: 144 total,   1 running, 143 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.4 us, 40.2 sy,  0.0 ni, 29.4 id,  0.0 wa,  0.0 hi, 19.9 si,  0.0 st
MiB Mem :   1963.4 total,    338.4 free,    249.0 used,   1376.0 buff/cache
MiB Swap:   2048.0 total,   2036.3 free,     11.7 used.   1528.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  21088 wub       20   0  546088   3044   2480 S 143.9   0.2   1:37.36 tokio-ch
  21098 wub       20   0  614060   3860   2756 S 142.9   0.2   1:37.19 simple-bench
     14 root      20   0       0      0      0 I   0.3   0.0   2:27.04 rcu_sched
  20763 root      20   0       0      0      0 I   0.3   0.0   0:08.53 kworker/3:0-events

[WuBingzheng]

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all    9.75    0.00   39.67    0.00    0.00   20.16    0.00    0.00    0.00   30.42
Average:       0   11.11    0.00   39.72    0.00    0.00   21.39    0.00    0.00    0.00   27.78
Average:       1    9.62    0.00   39.55    0.00    0.00   21.02    0.00    0.00    0.00   29.81
Average:       2    8.84    0.00   40.19    0.00    0.00   19.01    0.00    0.00    0.00   31.96
Average:       3    9.42    0.00   39.21    0.00    0.00   19.19    0.00    0.00    0.00   32.18

[WuBingzheng]

I try 10000 tasks, then the CPU reaches to 100% almost!

[Noah-Kennedy]

I forgot you were only running 8 connections.

You'll never be able to get enough throughput from 8 connections without resorting to the dark arts of network programming to actually use much CPU

[Noah-Kennedy]

That CPU is gonna be like 70% softirq

[Noah-Kennedy]

probably more

[Noah-Kennedy]

If you want more throughput for something which is strictly IO-bound like this, its better to use SO_REUSEPORT and a pool of current_thread runtime threads which are pinned to different cores, with sockets pinned to core rx queues using SO_INCOMING_CPU

[Noah-Kennedy]

if you want more than that then you can look into ebpf or dpdk for the stupidly high-throughput echo server