fix(ci): make ASAN sanity findings reachable and readable by dileepr1 · Pull Request #8100 · ROCm/TheRock

dileepr1 · 2026-09-09T16:12:24Z

Depends on #8123, which sets up the ASAN environment for test jobs:
configure_asan_env.py resolves ASAN_RUNTIME_PATH and ASAN_SYMBOLIZER_PATH
from the artifact tree, and the workflow scopes LD_PRELOAD and the leak
suppression to the sanity step.

This PR is the remainder: the two script-side changes needed to get past the
sanity check and read what ASAN reports once it runs. It touches no workflow
files.

Earlier revisions of this PR also carried the workflow reorder, a
BUILD_VARIANT branch inside print_driver_gpu_info.py, and the
ASAN_SYMBOLIZER_PATH fix. All three now live in #8123 — the first two after
review feedback on #7311 asking that generic utilities not learn about CI build
variants, the third as part of moving that logic out of inline bash.

Commits

print a failing command's output in the driver sanity script —
run_command pipes stdout and stderr but only logs them on success, so a
failing probe produced a bare CalledProcessError traceback. Nightly ASAN
runs failed this way for days with no indication of why amd-smi returned
non-zero. No CI or sanitizer coupling: it applies to any failing probe on
any system.
preload the ASAN runtime for hip_check in the sanity test — hip_check
is deliberately built without -fsanitize=address, so it aborts with "ASan
runtime does not come first" when it loads instrumented ROCm libraries.
Instrumenting it instead would pull in the device-side instrumentation that
the host-asan variant exists to avoid, so the runtime is preloaded for that
one subprocess, reusing the ASAN_RUNTIME_PATH that fix(ci): preload ASAN runtime for GPU sanity check in host-asan builds #8123 exports.

Kept in the project test rather than exported job-wide. A job-wide
LD_PRELOAD puts the non-instrumented Python driving pytest under
LeakSanitizer, which then reports CPython's own end-of-process allocations
and fails test steps: measured at 531244 bytes in 499 allocations, all
_PyMem_RawMalloc / _PyBytes_Resize / rpds, in run
34492719442.

Validation

Exercised against ROCm/rocm-systems#10088 on a branch carrying #7780 and #8123
plus these changes, added progressively:

Run	State	Result
34385342239	baseline	`amd-smi static` exit 1 after 0.19s, reason not logged
34395575767	+ preload (#8123)	survives 2.11s then exit 1, reason still not logged
34401743892	+ commit 1, leak suppression	sanity passes; `hip_check` aborts, now visible
34408883136	+ commit 2	real ASAN `bad-free` in `libamdhip64.so.7`, unsymbolized
34513780373	+ symbolizer fix (now #8123)	same finding, symbolized, no spurious leak report

The sanity check passes and the remaining failure is a genuine defect:

ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed
    #0 hsa_amd_memory_pool_free   compiler-rt/lib/asan/asan_interceptors.cpp:916
allocated by thread T0 here:
    #0 hsa_amd_memory_pool_allocate compiler-rt/lib/asan/asan_interceptors.cpp:908
SUMMARY: AddressSanitizer: bad-free (build/bin/../lib/libamdhip64.so.7+0x5febae)

A 4 MiB region allocated through the HSA pool allocator is released through a
path the allocator does not recognise, during static destructor teardown. That
is tracked separately; it is not addressed here.

ISSUE ID: #7202

therock-pr-bot · 2026-09-09T16:13:16Z

✅ All Checks Passed — Ready for Review

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—
🤖 therock-pr-bot	✅ Pass	—

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-09T16:13:50Z

🎉 All checks passed! This PR is ready for review.

Validate ROCm/TheRock#7780 on top of current main (which already carries ROCm/TheRock#7602 and ROCm/TheRock#7600) without the sanity-check fixes still in review as ROCm/TheRock#8100. Co-authored-by: Cursor <cursoragent@cursor.com>

run_command pipes stdout and stderr but only logs them on success, so when a probe fails the command's own diagnostics are discarded and the job shows a bare CalledProcessError traceback. Nightly ASAN runs failed this way for days with no indication of why amd-smi was returning non-zero. Print the captured output before propagating. This has no CI or sanitizer coupling; it applies to any failing probe on any system. Co-authored-by: Cursor <cursoragent@cursor.com>

hip_check is compiled without -fsanitize=address on purpose, so running it against instrumented ROCm libraries aborts with "ASan runtime does not come first in initial library list". Instrumenting it instead would pull in the device-side instrumentation that the host-asan variant exists to avoid. Preload the runtime for that one subprocess, reusing ASAN_RUNTIME_PATH which CI already resolves. LD_PRELOAD is deliberately not exported globally, to keep third-party libraries from producing spurious leak reports. Ported from #7311. Co-authored-by: Cursor <cursoragent@cursor.com>

Rebuilds the validation branch on top of current main, which already carries #8123's ASAN environment setup, so only #7780 remains on top. #8100 is left out while it is parked. The host-asan gate moved and was widened to startswith() on main since #7780 branched, so the presubmit escape is applied to that structure rather than the original elif. Content matches #7780. Co-authored-by: Cursor <cursoragent@cursor.com>

## Motivation ### In Short: #8268 fixes two independent bugs that together made every host-ASan hip-tests shard fail without running a single test. First, is_asan() matched BUILD_VARIANT exactly against "asan", so it was false for host-asan and everything gated on it silently did nothing Second, configure_asan_env.py pinned the runtime filename to libclang_rt.asan.so, which #8077 renamed to libclang_rt.asan-<arch>.so, so the ASan runtime hadn't resolved at all since that landed. ## Details GitHub issue: #7202 Found while looking at the first host-asan presubmit run that got as far as the component test matrix (ROCm/rocm-systems#10088, [run 35004915528](https://github.com/ROCm/rocm-systems/actions/runs/35004915528)). All four `hip-tests` shards failed **without running a single test**, and with zero AddressSanitizer findings: ``` catch_tests/RTC: error while loading shared libraries: libclang_rt.asan-x86_64.so: cannot open shared object file CMake Error at CatchAddTests.cmake:82: Error listing tests from executable '.../RTC': Result: 127 ctest ... returned non-zero exit status 8 ``` ctest runs `RTC --list-tests` to enumerate Catch2 tests; it cannot load the instrumented binary, CMake aborts the listing, and the shard reports failure. Two separate defects combine to produce this. ### 1. `is_asan()` excluded host-asan ```python return BUILD_VARIANT == "asan" ``` An exact match, so `host-asan` (and `asan-debug`, `host-asan-debug`) evaluated false and everything gated on it silently did nothing: | Caller | What was skipped | | --- | --- | | `test_hiptests.py` | `LD_PRELOAD` of the runtime, `HSA_XNACK`, extended timeout | | `test_rocprofiler_sdk.py` | clang resource dir on the loader path, preload, ASAN test exclusions | | `test_hipfile.py` | preload | | `test_rocblas.py`, `test_hipblas.py`, `test_hipblaslt.py` | `HSA_XNACK` | | `tests/test_rocm_sanity.py` | the #3312 rocminfo skip | The `test_hiptests.py` preload is the one that breaks discovery. Notably the machinery was already correct — `get_asan_lib_path()` there asks clang for `libclang_rt.asan-<arch>.so`, exactly the name `RTC` needs — it just never ran. The rest of the codebase already treats these as a family: `fetch_test_configurations.py` uses `"asan" in build_variant` with a comment listing all four variants, `configure_multi_arch_ci.py` uses `suffix in ("asan", "host-asan")`, and `publish_rocm_to_release_buckets.py` has `_is_asan_variant()`. `is_asan()` was the odd one out. ### 2. `configure_asan_env.py` pinned one runtime filename `ASAN_RUNTIME_LIB = "libclang_rt.asan.so"` is the per-target-runtime-dir name. #8077 (`cd8ed0c3f`) set `LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF`, moving the runtime to `lib/linux/libclang_rt.asan-<arch>.so`. Resolution has failed ever since, and every ASAN test job has been logging `##[warning]ASAN runtime not found`. The regression is visible across runs: | Run | TheRock base | Result | | --- | --- | --- | | [34549006044](https://github.com/ROCm/rocm-systems/actions/runs/34549006044) | `b85dc2119` (pre-#8077) | resolved `.../x86_64-unknown-linux-gnu/libclang_rt.asan.so` | | [34613837319](https://github.com/ROCm/rocm-systems/actions/runs/34613837319) | `24cc5c7d6` | not found | | [35004915528](https://github.com/ROCm/rocm-systems/actions/runs/35004915528) | `24cc5c7d6` | not found | ## Technical Details `is_asan()` now matches any ASAN-family variant via `"asan" in BUILD_VARIANT`, mirroring `fetch_test_configurations.py`. `_resolve_asan_runtime()` tries both filenames in turn, per-target first then the legacy arch-suffixed one, relying on the fact that clang echoes the requested name back unchanged when it cannot locate the file. The warning now names both candidates. TheRock has flipped between these layouts, so pinning either one is fragile. ## Test Plan - `python -m pytest build_tools/ tests/ -q --continue-on-collection-errors` (against baseline) - `pre-commit run --files build_tools/github_actions/amdgpu_family_matrix.py build_tools/github_actions/configure_asan_env.py build_tools/github_actions/tests/configure_asan_env_test.py` ## Test Result 2432 passed / 16 failed / 6 collection errors, against a baseline of 2431 passed / 16 failed / 6 collection errors on `origin/main`. The failures and collection errors are identical on both sides and are pre-existing local dependency gaps (`zlib_ng`, `pkginfo`, `_rocm_sdk_core_DEFAULT`); the delta is the one test added here. Targeted pre-commit passed. ## Risk Assessment Risk 3. Widening `is_asan()` activates ASAN handling for host-asan across several test scripts that have never had it, which is the intent but does change their behavior. The specific known side effect is that the TheRock#3312 rocminfo skip now applies to host-asan. That `skipif` is parametrized over three search patterns, so three sanity tests that were passing under host-asan are now skipped, taking the sanity tally from `1 failed, 4 passed, 3 skipped` to `1 failed, 1 passed, 6 skipped`: ``` test_rocm_output[rocminfo - GPU Device Type Search] SKIPPED test_rocm_output[rocminfo - GFX Name Search] SKIPPED test_rocm_output[rocminfo - AMD Vendor Name Search] SKIPPED ``` That coverage loss is accepted here in exchange for the variants behaving consistently. Worth a follow-up: #3312 was filed against full ASAN, and these tests demonstrably pass under host-asan, so narrowing that `skipif` to the full-ASAN variant may be the better long-term answer. The runtime-name change is additive: it only adds a second candidate after the existing one fails. ## Related - Work tracking: #7202 - Caused the rename: #8077 - Depends on nothing; independent of #7780 and #8100 ## Device / Architecture Coverage Validated on gfx942 via the rocm-systems host-asan presubmit. No device code changes. ## Testing Summary Added `test_resolves_the_legacy_arch_suffixed_runtime`, which stubs a clang that resolves only the arch-suffixed name and echoes the request back otherwise, matching real clang behavior. Existing coverage for the per-target name, missing clang, missing runtime, missing symbolizer and the GITHUB_ENV format is unchanged. ## Testing Checklist - [x] Full unit suite vs baseline - Status: Passed, no regressions - [x] Targeted pre-commit - Status: Passed - [x] host-asan presubmit against ROCm/rocm-systems#10088 - [run 35140846158](https://github.com/ROCm/rocm-systems/actions/runs/35140846158) - Status: Passed. hip-tests went from 0 tests discovered to ~4,900 run across 4 shards (~2,050 passing). Every remaining failure is the pre-existing `hsa_amd_memory_pool_free` bad-free. The runtime now resolves to `.../lib/clang/24/lib/linux/libclang_rt.asan-x86_64.so`, confirming the #8077 layout. - [ ] PR CI - Status: Pending ## Flags / Guardrails None. ## Adjacent Tests Considered `rocprofiler_sdk_dashboard_test.py` patches `is_asan` directly rather than driving it through `BUILD_VARIANT`, so it is unaffected by the widening and needed no change. ## Risk Acceptance / Waivers None. ## Technical Changes - `is_asan()` matches any ASAN-family `BUILD_VARIANT` instead of exactly `"asan"`. - `configure_asan_env.py` resolves the ASAN runtime under either the per-target or the legacy arch-suffixed filename. - Add a unit test for the legacy name. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/CONTRIBUTING.md. Co-authored-by: Cursor <cursoragent@cursor.com>

github-project-automation Bot added this to TheRock Triage Sep 9, 2026

github-project-automation Bot moved this to TODO in TheRock Triage Sep 9, 2026

dileepr1 force-pushed the users/dileepr1/asan-sanity-fixes branch from ab67b82 to 170ade3 Compare September 10, 2026 20:01

dileepr1 changed the title ~~fix(ci): make the ASAN driver sanity check work under instrumentation~~ fix(ci): make ASAN sanity findings reachable and readable Sep 10, 2026

dileepr1 and others added 2 commits September 10, 2026 16:05

dileepr1 force-pushed the users/dileepr1/asan-sanity-fixes branch from 170ade3 to d719e3c Compare September 10, 2026 21:06

sa-faizal mentioned this pull request Sep 11, 2026

[Issue] ASAN bad-free in libamdhip64.so.7 HSA memory pool allocator during static destructor teardown #8142

Open

This was referenced Sep 16, 2026

fix(ci): make host-asan test jobs find and use the ASAN runtime #8268

Merged

fix(ci): preload ASAN runtime for rocm-sdk wheel tests #8341

Open

sa-faizal mentioned this pull request Oct 8, 2026

[Issue] Linux gfx94X-dcgpu nightly (asan): Test Sanity Check fails — amd-smi static exits 1, runner container hook failure #7515

Open

Navigation Menu

Repository navigation

fix(ci): make ASAN sanity findings reachable and readable - #8100

fix(ci): make ASAN sanity findings reachable and readable#8100
dileepr1 wants to merge 2 commits into
mainfrom
users/dileepr1/asan-sanity-fixes

dileepr1 commented Sep 9, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dileepr1 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commits

Validation

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Checks Passed — Ready for Review

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dileepr1 commented Sep 9, 2026 •

edited

Loading

therock-pr-bot Bot commented Sep 9, 2026 •

edited

Loading