Calculate parallel jobs based on available CPUs

We're going to enable new ARM64 runners in CI, where test-run is invoked
in a Docker container. At the same time, the runner is an LXD container
created by the service provider. In this circumstances, the Docker
container sees all the 128 online CPUs, but the runner may have
only some of them available (depending on the pricing plan).

Python 3.3+ has a function to determine available CPUs, so we can reduce
the parallelism to this value. We fall back to the online CPUs count on
Python < 3.3.

After this change, test-run follows a CPU affinity mask set by `taskset`
(and, I guess, by `numactl`).

The change is similar to replacing `nproc --all` to `nproc`.

The change only affects the default behavior, which can be overwritten
by passing the `--jobs` (or `-j`) CLI option or using the
`TEST_RUN_JOBS` environment variable.

Reported in tarantool/tarantool#10102

Author: Totktonada

lib/utils.py

@@ -8,6 +8,7 @@
 import time
 import json
 import subprocess
+import multiprocessing
 from lib.colorer import color_stdout
 
 try:
@@ -31,6 +32,12 @@
     # Python 2.7.
     get_terminal_size = None
 
+try:
+    # Python 3.3+
+    from os import sched_getaffinity
+except ImportError:
+    sched_getaffinity = None
+
 UNIX_SOCKET_LEN_LIMIT = 107
 
 # Useful for very coarse version differentiation.
@@ -384,3 +391,22 @@ def terminal_columns():
     if get_terminal_size:
         return get_terminal_size().columns
     return 80
+
+
+def cpu_count():
+    """
+    Return available CPU count available for the current process.
+
+    The result is the same as one from the `nproc` command.
+
+    It may be smaller than all the online CPUs count. For example,
+    an LXD container may have limited available CPUs or it may be
+    reduced by `taskset` or `numactl` commands.
+
+    If it is impossible to determine the available CPUs count (for
+    example on Python < 3.3), fallback to the all online CPUs
+    count.
+    """
+    if sched_getaffinity:
+        return len(sched_getaffinity(0))
+    return multiprocessing.cpu_count()

test-run.py

@@ -58,6 +58,7 @@
 from lib.colorer import color_stdout
 from lib.colorer import separator
 from lib.colorer import test_line
+from lib.utils import cpu_count
 from lib.utils import find_tags
 from lib.utils import shlex_quote
 from lib.error import TestRunInitError
@@ -86,7 +87,7 @@ def main_loop_parallel():
     jobs = args.jobs
     if jobs < 1:
         # faster result I got was with 2 * cpu_count
-        jobs = 2 * multiprocessing.cpu_count()
+        jobs = 2 * cpu_count()
 
     if jobs > 0:
         color_stdout("Running in parallel with %d workers\n\n" % jobs,