kerf: CPU/NUMA topology-aware resource allocation - #12
Open
congwang-mk wants to merge 3 commits into
Open
Conversation
The topology models, allocation policies and validators already existed but were dead code in practice: the baseline DTB generator never emitted the topology section, so every create/update reading state back from the kernel saw no topology, and nothing ever discovered it from the host. Make the support real: - Emit the topology section (NUMA nodes, per-node memory range, CPUs as 64-bit physical ID cells, distance matrix as <target distance> pairs, memory type) and device numa-node into the baseline DTB, and parse them back, so topology survives the kernel round trip. - Add kerf/topology.py to discover the host topology at kerf init time: NUMA nodes and distances from /sys/devices/system/node, per-node memory ranges from /proc/zoneinfo, and PCI device locality from sysfs. Sysfs is keyed by logical CPU while kerf speaks physical APIC IDs, so discovery translates through the processor/apicid pairs in /proc/cpuinfo. - Fix DTS parsing bugs that broke hand-written topology sections: the topology/numa-nodes/cores regexes could not handle nested braces (only the first node parsed), comments inside multi-line property values broke value parsing, and the pool memory-base regex could match a NUMA node's memory-base instead. - Keep manual allocation authoritative: explicit --cpus no longer gets a compact affinity policy attached implicitly, so deliberate topology-crossing layouts do not accumulate spurious warnings. Auto-allocation (--cpu-count) still defaults to compact. Topology violations remain warnings; only impossible requests are errors. - Fold the topology documentation into README.md, replacing the stale docs/CPU_NUMA_TOPOLOGY.md, and update the NUMA examples to the supported format. Instance memory is still allocated first-fit from a single pool; NUMA-aware per-node pools via /dev/lazy_cma's node parameter are the next step.
Instance memory used to be first-fit from one contiguous pool allocated
with no node preference, so on multi-socket hosts all instances ended up
on whichever node happened to have contiguous memory, and memory-policy
was recorded but never influenced placement.
Introduce per-node memory pools:
- kerf init accepts per-node sizes ("memory=8GB@0,8GB@1"), allocating
one pool per NUMA node through lazy_cma's node parameter, which was
wired up in the kernel module but never used. A single anonymous size
keeps the exact legacy behavior and DTB layout. On re-init, all
reserved pools in /proc/iomem are rediscovered and matched to NUMA
nodes via the discovered topology's memory ranges.
- The pool layout round-trips through the baseline as a memory-pools
section (ignored by the kernel), parsed from both DTB and DTS. The
legacy memory-base/memory-bytes properties remain as the envelope for
compatibility; allocation logic operates on the pool list, with a
synthesized single pool for old baselines.
- Allocation first-fits within each pool, never spanning the gap
between pools, and validation requires an instance region to lie
entirely inside one pool. The validator's iomem cross-check now
verifies every configured pool against all reserved regions.
- kerf create implements the placement policies: "local" hard-requires
a pool on the same node as the instance CPUs, "bind" hard-requires a
pool on the requested NUMA nodes, and with no policy kerf prefers a
CPU-local pool and silently falls back to any pool. An explicit
memory base remains authoritative and only gets locality warnings.
"interleave" stays unimplemented since instances receive a single
contiguous region; true interleaving needs kernel support for
multiple regions per instance.
Also fix a leftover from the logical-CPU-numbering era: the validator
rejected instance CPUs with id >= total, which falsely fails sparse
physical APIC IDs; it now checks membership in the hardware CPU set.
- parser: return plain lists from the memory-pool parsers and convert to None at the construction sites, so pylint can prove iterability (not-an-iterable) - lazy_cma, console, init: convert str.format() calls to f-strings (consider-using-f-string) - tests: underscore-name unused fake-interface arguments and disable redefined-outer-name for pytest fixtures (unused-argument, redefined-outer-name)
nickolaev
reviewed
Aug 4, 2026
|
|
||
| total_bytes = memory_pool_base + memory_pool_bytes | ||
| pool_list.sort(key=lambda pool: pool.base) | ||
| memory_pool_base = pool_list[0].base |
There was a problem hiding this comment.
Hmm, why only [0]? This somehow implies that we have all pools being continuous. What if they are not?
Shall we add an extra-check if the pools are continuous, and reject them if they are not?
| # boundaries, so no placement policy is attached unless asked for. | ||
| if is_count: | ||
| # Allocate CPUs automatically from available pool with topology awareness | ||
| effective_affinity = cpu_affinity or "compact" |
There was a problem hiding this comment.
We default to compact here, but that would require that Numa nodes are supplied on the command line already.
At least that is how I read _allocate_compact()
| except libfdt.FdtException: | ||
| pass | ||
|
|
||
| return InstanceResources( |
|
|
||
| # Create instance resources with topology settings | ||
| uring_enabled = uring or uring_sq_entries is not None or uring_cq_entries is not None or uring_shim_pages is not None | ||
| resources = InstanceResources( |
There was a problem hiding this comment.
We store the affinity and nodes info here -> and then drop it later in the parser
| if not cpus_match: | ||
| raise ParseError("Missing 'cpus' property in /resources") | ||
|
|
||
| available = [int(x.strip()) for x in cpus_match.group(1).split()] |
There was a problem hiding this comment.
Are we sure this is going to parse :
cpus = <0x0 0x80 0x0 0x82>
|
|
||
| return None | ||
|
|
||
| def _get_processor_to_physical_id_map(self) -> Optional[Dict[int, int]]: |
There was a problem hiding this comment.
Maybe we can re-use the new read_logical_to_physical_cpu_map here and simplify this code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9
Summary
Makes kerf topology-aware end to end: the baseline records the host's real CPU/NUMA topology, auto-allocation places CPUs and memory for locality, and validation reports topology violations as warnings while keeping manual allocation fully authoritative.
Two commits:
1. Make NUMA topology support functional end to end
The topology models, allocation policies and validators already existed but were dead code: the baseline DTB generator never emitted the topology section, and nothing discovered topology from the host.
kerf initnow discovers the host topology automatically: NUMA nodes and distances from sysfs, per-node memory ranges from /proc/zoneinfo, PCI device locality from sysfsnuma_node. Sysfs is keyed by logical CPU while kerf speaks physical APIC IDs, so discovery translates through the processor/apicid pairs in /proc/cpuinfo.numa-node) round-trips through the baseline DTB, so every later create/update sees it. The kernel ignores these nodes.memory-basebeing shadowed by a NUMA node'smemory-base.2. Per-NUMA-node memory pools and policy-driven placement
kerf initaccepts per-node pool sizes (memory=8GB@0,8GB@1), allocating one pool per NUMA node through lazy_cma's previously unused node parameter. A single anonymous size keeps the exact legacy behavior and DTB layout. Existing pools in /proc/iomem are rediscovered on re-init and matched to nodes via the topology.memory-poolssection (kernel-ignored), parsed from both DTB and DTS.memory-policynow drives placement:localhard-requires a pool on the same node as the instance CPUs,bindhard-requires a pool on the requested nodes, no policy prefers a CPU-local pool with silent fallback. Explicitmemory-baseremains authoritative with warnings only.interleavestays unimplemented pending kernel support for multiple regions per instance.Testing
59 new tests (tests/test_topology.py, tests/test_memory_pools.py) covering DTB/DTS round-trips, sysfs discovery against fake trees, pool-aware allocation/validation, and CLI placement policies via dry-run. Full suite: 136 passed. Discovery and DTB round-trip also verified against a live host.
Known follow-ups
kerf updatereuses the pool-aware allocator but does not take placement policy flags yet.topology,memory-pools) verbatim; worth one check on multikernel hardware.