radlink: link-time performance pass (~1.9x on a Fortnite link) by honkstar1 · Pull Request #830 · EpicGames/raddebugger

honkstar1 · 2026-06-16T22:00:29Z

Profiled radlink linking UnrealEditorFortnite-Engine.dll (Superluminal, MSVC release) and landed 7 optimizations plus dead-code cleanup. Cold-run wall time dropped ~45s → ~23s (~1.9x). The vendored BLAKE3 source is left untouched.

Each commit is independent and carries its own rationale + measured gain.

Gains (main-thread, same workload)

Symbol	Before	After
`get_cpu_features`	5591 ms	4 ms
`coff_parse_symbol32` (main)	3922 ms	~810 ms
`lnk_on_symbol_replace`	1306 ms	161 ms
`ReleaseSemaphore` (main)	3274 ms	940 ms
`cv_name_from_symbol`	1098 ms	201 ms
`lnk_fixup_cv_type_indices` (incl)	1445 ms	390 ms

Commits

BLAKE3 C11 atomics — _InterlockedOr(&x,0) per-dispatch barrier → plain load, via build flags only (no third_party edit).
No-name symbol parse — skip the string-table cstr scan on interp-only paths.
refs_tail — lnk_on_symbol_replace ref-merge O(n²) → O(1).
Batched worker wakeup — one ReleaseSemaphore(h, count, 0) instead of a per-worker loop.
memchr in str8_cstring_capped instead of a byte loop.
Type-index fixup single probe — store assigned ti on the leaf hash table; removes the second hash table + its build pass.

Correctness

Linker torture suite (build/torture.exe): 65/65 linker tests pass (COMDAT, weak/undef/abs, ghash type-merge, relocs, import/export), unchanged from baseline, verified after every commit.

Caveats

Commit 1 uses /std:c11 /experimental:c11atomics (scoped to the radlink target). /experimental:c11atomics is an unstable MSVC switch — happy to use a project-local ATOMIC_LOAD override instead if preferred.
radlink output is already run-to-run non-deterministic, so the torture suite (not a byte-diff) was used as the gate.

🤖 Generated with Claude Code

get_cpu_features was the top main-thread hot spot (~5.6s for one Fortnite link), 97% of it inside a single ATOMIC_LOAD(g_cpu_features). On MSVC, blake3_dispatch.c defines ATOMIC_LOAD as _InterlockedOr(&x,0) -- a lock'd RMW (full barrier) run on every BLAKE3 compress dispatch. The value is written once and read-only after, so the barrier is pointless. Enable BLAKE3's plain-load path (C11 _Atomic, a plain mov on x86) via build flags only, leaving the vendored third_party/blake3 source untouched: /std:c11 /experimental:c11atomics -DBLAKE3_ATOMICS=1 Scoped to the radlink target. MSVC C11 atomics need both /std:c11 and /experimental:c11atomics. get_cpu_features: 5591ms -> 4ms (main thread). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coff_read_symbol_name scans a cstr in the memory-mapped string table -- the dominant, page-fault-bound cost of bulk symbol parsing. Many hot callers parse a full symbol but only read scalar fields (value/section/storage_class/aux) to interpret the symbol value; the name is never used. Add name-skipping parse variants and route the interp-only paths through them: coff_parse_symbol{16,32}_no_name (coff_parse.c) -- and the full variants now call these + add the name, so the scalar logic lives in one place lnk_parsed_symbol_from_coff_symbol_idx_no_name (lnk_obj.c) lnk_interp_from_symbol / lnk_can_replace_symbol / lnk_on_symbol_replace (lnk_symbol_table.c) and the lnk_search_lib_task loop (lnk.c) Where the name is still needed (lnk_search_lib) it uses the already-cached LNK_Symbol.name instead of re-parsing. lnk_can_replace_symbol previously parsed dst/src twice (full parse + a second parse for interp); collapsed to one no-name parse each. coff_parse_symbol32 on the main thread: 3922ms -> ~810ms (name-needed callers). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

lnk_on_symbol_replace merged ref lists by walking the destination's singly linked refs list to its tail on every merge. Across repeated COMDAT merges into one accumulating leader this is O(n^2) and was 96% of the function. Add a refs_tail pointer to LNK_Symbol so the append is O(1): src->refs_tail->next = dst->refs; src->refs_tail = dst->refs_tail; maintained at all ref-list write sites (lnk_make_symbol, the null_symbol and import-stub sites in lnk.c). Order and head identity are preserved exactly, so this is a pure perf change: the head node stays the primary ref, and interior order is irrelevant (every multi-ref consumer sorts). lnk_on_symbol_replace (main thread): 1306ms -> 161ms exclusive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

tp_for_parallel woke workers with a loop of single semaphore_drop calls -- one ReleaseSemaphore syscall per worker (twice in shared mode). The main thread spent ~3.3s in ReleaseSemaphore over a Fortnite link. Add semaphore_drop_n(sem, count) (a single ReleaseSemaphore(h, count, 0) on Windows; a loop on POSIX) and wake all drop_count workers in one call. Two details keep the batched release correct: - Wake the full drop_count (NOT drop_count-1). The main thread runs as worker 0, but tasks that nest a tp_broadcast_ barrier span all workers; under-waking by one leaves that barrier a participant short and deadlocks on small dispatches. - Give the exec/task semaphores 2x max-count headroom. A single batched release can land while up to worker_count-1 previously-woken workers have not yet re-taken their permit, so the count can transiently approach 2*worker_count; a tight max would make ReleaseSemaphore fail outright and deadlock at the next barrier. ReleaseSemaphore (main thread): 3274ms -> 940ms.

The capped cstr length scan was a byte-by-byte loop. Switch to memchr, which is SIMD-accelerated in the CRT. Speeds up every capped-cstr scan in the codebase; notably cv_name_from_symbol (CodeView symbol-name scan during GSI build). cv_name_from_symbol (main thread): 1098ms -> 201ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

lnk_fixup_cv_type_indices did two open-addressing probes per type-index reference: lnk_leaf_hash_table_search (leaf_ref -> canonical bucket), then lnk_assigned_type_ht_search (canonical bucket -> assigned type index, via a second hash table keyed by leaf-ref content). Both are cache-miss-bound and this ran across every type-index reference in every obj. Store the assigned type index directly on the leaf hash table: add a ti_arr parallel to bucket_arr. lnk_assign_type_indices_task writes ti = min+i into the leaf's bucket slot (each unique leaf owns a distinct slot, so worker writes never collide), and the new lnk_leaf_hash_table_search_ti recovers it in one probe. Removes the entire assigned_type_hts table and its build pass; deletes the now-dead lnk_leaf_hash_table_search and lnk_assigned_type_ht_search. Correctness: deduplicated leaves share the same ghash (debug_h value), so the fixup query and the assign-time canonical bucket hash to the same slot. lnk_fixup_cv_type_indices (main thread): 1445ms -> 390ms inclusive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Input obj/lib files are mapped copy-on-write (PAGE_WRITECOPY/FILE_MAP_COPY) so the linker can patch them in place. Pages touched during linking become private-dirty; at process exit the kernel reclaims them in single-threaded address-space rundown -- ~3s of lingering process time after the last thread exits for a large (Fortnite-scale) link. After all outputs are written and inputs are no longer read (post image-write join), unmap the whole-file CoW views in parallel on the thread pool. The same reclaim work then runs multi-threaded, off the serial post-exit path: measured ~34s of aggregate UnmapViewOfFile CPU collapsing to ~0.55s wall, and the post-exit process tail dropping from ~3s to ~0.5s. Only the is_thin whole-file views are swept (lib-member substrings and linkgen arena data are skipped), and only in the copy-on-write (read-only) mapping mode -- read-write-shared mapping would flush dirty pages back to the input files on unmap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Two micro-optimizations on hot parse helpers (profiled as the largest aggregate-CPU functions in a Fortnite link): - lnk_obj_section_from_sect_idx: split out a _no_name variant that skips the section-name string-table lookup (coff_name_from_section_header). The full variant now reuses it + adds the name. lnk_raw_directives_from_obj iterated every section of every obj building the full section struct just to test a flag, computing the name on ~all sections though only .drectve needs it -- now uses the no-name variant and resolves the name only inside the LnkInfo branch. - lnk_parsed_symbol_from_coff_symbol_idx (+_no_name): return the coff_parse_* result directly instead of zero-initializing a local and assigning to it, removing a redundant ~48-byte COFF_ParsedSymbol copy + zero-init per call (RVO). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…hotspot) Parse each COFF symbol once in lnk_obj_initer into LNK_Obj.parsed_symbols; lnk_parsed_symbol_from_coff_symbol_idx[_no_name] becomes an array index instead of re-decoding the mmapped symbol table on every access (it was the EpicGames#1 hotspot, lnk_parsed_symbol_from_coff_symbol_idx). All symbol-value patch sites (weak-replace, COMDAT-leader, regular/common fixups) write obj->parsed_symbols[idx], decoupling symbol values from the input mapping. Extracted from the entangled WIP commit 8dc5fa6 (parsed-symbol memo + .rgd staging); this is the memo half only -- no .rgd. Gated separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

LNK_Obj.parsed_symbols memoized the full COFF_ParsedSymbol per symbol -- including the 16B String8 name -- sized by total symbol count, held to exit. Store a slim LNK_ParsedSymbolLite (every field except name, ~24B vs ~40B) and re-decode the name from the read-only symbol record in the named accessor only. The hot _no_name path (can_replace/GC/resolution) and all symbol-value patching never touch the name, so they stay fully memoized; only the named/push path pays a re-decode (cold relative to total). FN no-rrt full link: peak commit 50.5GB -> 49.1GB (-1.4GB), wall flat (~8-9s no-debug), valid 5.68GB PDB, 95/0 linker torture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

lnk_leaf_hash_table_search_ti (~EpicGames#2 radlink hotspot, ~48% of its self-time) spent its time in lnk_match_leaf_ref, which is just a_hash==b_hash but fetches the bucket's hash via lnk_hash_from_leaf_ref -> input->debug_h_arr[obj].v[leaf], a scattered cache miss per probe step. Add LNK_LeafHashTable.hash_arr (parallel to bucket_arr), populated at bucket claim/update (both lnk_populate_leaf_ht and lnk_leaf_dedup_task) with the leaf's debug_h hash. search_ti now matches via hash_arr[idx] == hash -- no deref. Exact equivalent (match is pure hash compare); same value across same-hash updates. Gated 65/65 linker torture (ghash_basic/match_debug_t, determ_test, p2r_determinism). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…im ~3GB peak) The CV type-index fixup (05af760) had stored the assigned type index in arrays parallel to the dedup hash table (leaf_ht), whose cap is the TOTAL pre-dedup leaf count summed over all objs. On large links that ti_arr (+ the hash_arr added for deref-free probing) added ~3GB to peak working set vs the prior unique-sized assigned_type_ht. Split the two concerns: - LNK_LeafHashTable: just {cap, bucket_arr} for dedup (total-sized, as before / unavoidable). - LNK_AssignedTiHash {cap, ti_arr, hash_arr}: hash -> assigned ti, sized to the UNIQUE (post-dedup) leaf count. Built in lnk_assign_type_indices_task by hashing each unique leaf into its own slot (atomic claim; unique leaves have distinct hashes since dedup is by hash). search_ti probes it in one deref-free pass (occupant hash stored on the slot), exactly as before -- just on a table sized by unique instead of total. Keeps the single-probe fixup speed of 05af760; removes its peak-memory regression. ti==0 marks empty (assigned ti is always >= CV_MinComplexTypeIndex). Builds clean, 95 linker torture PASS, 0 fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The image buffer was push'd on the shared link arena and only reclaimed in the single-threaded process rundown at exit -- a multi-second kernel page-reclaim tail (observed: one thread 100% in-kernel, zero user frames). Allocate it as a standalone reserve_memory/commit_memory region and release_memory() it the instant the background image-write thread joins (image is on disk, no later reader). VirtualFree(MEM_RELEASE) returns fast; the kernel zeroes the ~1GB on its background thread, overlapping the parallel input-view release + exit instead of blocking rundown. Discard early so the kernel cleans up while the app still runs -- don't defer to exit. Gated 65/65 linker torture (determ_test + p2r_determinism: image correct). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

honkstar1 · 2026-06-18T19:12:52Z

Pushed 5 follow-up commits (perf + peak memory). Measured on a no-.rrt UnrealEditorFortnite-Engine.dll link (heaviest type-merge case); all gated 95/95 linker torture + valid 5.7GB PDB.

commit	change	improvement
memoize parsed COFF symbols	parse each symbol once into `LNK_Obj.parsed_symbols`; accessor becomes an array index	removes the #1 hotspot (`lnk_parsed_symbol_from_coff_symbol_idx` re-parse) — code-link ~~18s→~~12s MSVC (~1.5x)
slim the parsed-symbol memo	store every field except `name` (~24B vs ~40B); re-decode name on the cold/named path only	peak commit −1.4GB (50.5→49.1GB), wall flat
cache leaf hash in dedup table	store occupant hash on the bucket	deref-free `lnk_leaf_hash_table_search_ti` (avoids scattered `debug_h_arr` deref)
size assigned-ti table by unique types	the CV type-index fixup table was sized by total leaves	peak −~3GB — fixes the +3GB peak regression introduced by the earlier 'fold CV type-index fixup' commit
release ~1GB image buffer early	free the image reservation right after the write thread joins	reclaim overlaps the run instead of single-threaded exit rundown

Net: code-link ~18s → ~8s with the clang-PGO build (~~2.2x), peak −~~4-5GB. Symbol-value patching writes obj->parsed_symbols, not obj->data, so inputs stay write-free (also lets inputs be read-only mmapped — separate change).

🤖 Generated with Claude Code

Store the COFF symbol record as a U32 byte-offset into obj->data instead of an 8B pointer, and value as U32 (COFF symbol value is U32). Struct goes 24B -> 16B (no padding): raw_symbol_off(4) value(4) section_number(4) type(2) storage_class(1) aux(1). The named/no_name accessors reconstruct the pointer as obj->data.str + off (obj->data == input->data, so the offset is stable). Sized by total symbol count -> 8B/sym off peak. FN no-rrt full link: peak commit 49.1GB -> 48.4GB (-0.67GB), wall flat, valid 5.68GB PDB, 95/0 linker torture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

honkstar1 · 2026-06-18T19:24:44Z

+1 more: pack LNK_ParsedSymbolLite to 16B (was 24B) — store the symbol record as a U32 offset into obj->data instead of an 8B pointer, and value as U32. Sized by total symbols → another −0.67GB peak (48.4GB). Cumulative memo-slimming (name-drop + 16B pack) = −2.1GB peak. 95/95 torture, valid PDB.

A struct-layout sweep of the other per-total-count structs (LNK_SectionContrib, LNK_LeafRef, LNK_Symbol, LNK_SymbolHashTrie, the dedup arrays) found them already minimally packed — no further free wins there.

🤖 Generated with Claude Code

honkstar1 · 2026-06-19T23:56:40Z

Superseded by #839 (perf/radlink-link-time-plus), which contains rebased versions of all these commits plus output-size, determinism, and further link-time work. Closing in favor of #839.

honkstar1 force-pushed the perf/radlink-link-time branch from 26be2b6 to 7c99b7f Compare June 16, 2026 23:29

honkstar1 and others added 3 commits June 16, 2026 20:06

honkstar1 force-pushed the perf/radlink-link-time branch from 32cd519 to 816a840 Compare June 17, 2026 03:07

honkstar1 changed the base branch from master to dev June 17, 2026 03:07

honkstar1 and others added 5 commits June 17, 2026 11:53

honkstar1 force-pushed the perf/radlink-link-time branch from 816a840 to 8fcd84b Compare June 17, 2026 18:55

honkstar1 and others added 5 commits June 18, 2026 12:09

honkstar1 closed this Jun 19, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

radlink: link-time performance pass (~1.9x on a Fortnite link)#830

radlink: link-time performance pass (~1.9x on a Fortnite link)#830
honkstar1 wants to merge 14 commits into
EpicGames:devfrom
honkstar1:perf/radlink-link-time

honkstar1 commented Jun 16, 2026

Uh oh!

honkstar1 commented Jun 18, 2026

Uh oh!

honkstar1 commented Jun 18, 2026

Uh oh!

honkstar1 commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

honkstar1 commented Jun 16, 2026

Gains (main-thread, same workload)

Commits

Correctness

Caveats

Uh oh!

honkstar1 commented Jun 18, 2026

Uh oh!

honkstar1 commented Jun 18, 2026

Uh oh!

honkstar1 commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant