Skip to content

feat: sm_121a (GB10) build lane + aarch64 c_char portability - #26

Closed
AnsenIO wants to merge 1 commit into
avifenesh:mainfrom
AnsenIO:gb10-sm121a-bringup
Closed

feat: sm_121a (GB10) build lane + aarch64 c_char portability#26
AnsenIO wants to merge 1 commit into
avifenesh:mainfrom
AnsenIO:gb10-sm121a-bringup

Conversation

@AnsenIO

@AnsenIO AnsenIO commented Aug 19, 2026

Copy link
Copy Markdown

What

Bring-up for NVIDIA GB10 (DGX Spark; aarch64, compute capability 12.1) — two small fixes that take memra from "does not compile / does not run" to "all gates green" on this hardware:

  1. aarch64 c_char portability (cpu_experts.rs, cpu_native_check.rs): error buffers and two FFI fn-pointer declarations (PrefetchFn, RowsFn) hardcoded i8 where the ABI type is c_char. On aarch64 c_char is u8, so memra-engine did not compile at all. Switched to std::ffi::c_char throughout. No behavior change on x86_64 (c_char == i8 there; this is a no-op on the primary lane).

  2. sm_121a build lane (build.rs, lib.rs): detect_arch() mapped compute_cap 12.1 to the 120a gencode. The resulting fatbins contain only sm_120a SASS, which is not binary-compatible with sm_121 — every run died at kernel launch with CUDA_ERROR_NO_BINARY_FOR_GPU, and the runtime arch guard explicitly allowed this combination (("120a", 12, 1)), so the failure surfaced late and cryptically.

    • detect_arch: 12.1 now maps to a new 121a arch (arch=compute_121a,code=sm_121a); MEMRA_CUDA_ARCH=121a accepted as an explicit override.
    • Runtime guard: accepts ("121a", 12, 1); no longer accepts ("120a", 12, 1) — fails closed with the rebuild hint instead of crashing at launch. 12.0 behavior unchanged.
    • No kernel source changes were needed: arch guards are __CUDA_ARCH__ >= style, and the mxf4nvf4 / kind::f8f6f4 inline PTX assembles cleanly under sm_121a (CUDA 13.0.88).

Evidence (all on one GB10, CUDA 13.0.88, driver 580+, rustc 1.96)

kernel-checkALL GREEN (82 cells, 22 skipped) (skips are the two Qwen3.6-35B-A3B-UD-IQ4_XS.gguf-dependent cells; model not on this box):

moe sigmoid router vs host oracle (cases=68, ...): idx_mismatch=0 ... OK
fp8-blk-gpu Q8_0 bit-parity [256x512] bytes=139264 bad=0 OK
attn_head_gate [hd128 nh96 T7] maxdiff=2.38e-7 f16_mismatch=0 ... OK
swiglu_clamped [limit=16 gs=0.75 us=1.25] maxdiff=7.63e-6 ... OK
ALL GREEN (82 cells, 22 skipped)

run-gen prime gates, Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF (Q5K-mtp):

prefill argmax=760  decode argmax=760  logit maxdiff=8.857e-1  MATCH
batched-prime argmax=760  tokenwise argmax=760  logit maxdiff=8.819e-1  MATCH

run-spec K=1..8 self-consistency, same model + frspec-mixed32768 trim, real 18-token chat prompt — 8/8 PASS, token-identical to plain decode:

[generate_spec K=1] acceptance: 14/17 = 82.4%   self-consistency: PASS
[generate_spec K=2] acceptance: 19/24 = 79.2%   self-consistency: PASS
[generate_spec K=3] acceptance: 21/33 = 63.6%   self-consistency: PASS
[generate_spec K=4] acceptance: 23/44 = 52.3%   self-consistency: PASS
[generate_spec K=5] acceptance: 21/50 = 42.0%   self-consistency: PASS
[generate_spec K=6] acceptance: 22/54 = 40.7%   self-consistency: PASS
[generate_spec K=7] acceptance: 23/56 = 41.1%   self-consistency: PASS
[generate_spec K=8] acceptance: 22/72 = 30.6%   self-consistency: PASS
=== SELF-CONSISTENCY PASS ===

First perf touchpoint (same prompt sweep, MEMRA_PROMPT_DIR, K=4, gen-only tok/s): plain 8.6 → spec 15.0 tok/s (1.74x, 46.5% acceptance). Reference point on the same box: llama.cpp llama-bench tg128 = 2.71 tok/s on Qwen3.8-27B-Q4_K_M (ngl 99). No tuning claimed — GB10 has no tuned defaults and this PR sets none.

Scope / what this PR is not

  • No defaults set for GB10. Per "a default only ships for a card class it was measured on", this PR only makes the hardware build and run correctly; tuning is a later lane with its own receipts.
  • memra-probe's build.rs still hardcodes sm_120a (documented phase-0 spine) — left as-is deliberately; say the word if you want it parameterized too.
  • MEMRA_CUTLASS remains 120a-only (unchanged assert).
  • I cannot run the x86/sm_120a regression battery (no such hardware); the changes are additive on that lane except the guard tightening called out above, and CI's compile gate covers the x86 build. Happy to adjust anything you'd rather see shaped differently.

Assisted-by: Hermione (IABAI agent) for the aarch64/sm_121a bring-up and on-device verification; hardware and gates run by the account owner.

Two bring-up fixes for NVIDIA GB10 (DGX Spark, aarch64, compute 12.1):

- memra-engine build.rs: detect_arch mapped compute_cap 12.1 to the
  120a gencode, producing SASS that fails at runtime with
  CUDA_ERROR_NO_BINARY_FOR_GPU. Add a 121a arch lane
  (arch=compute_121a,code=sm_121a) and auto-detect it.
- Engine arch guard: accept 121a builds on 12.1 devices; stop accepting
  120a builds on 12.1 (fail-closed instead of crashing at kernel launch).
- cpu_experts.rs / cpu_native_check.rs: error buffers and FFI fn-pointer
  declarations hardcoded i8 where the ABI is c_char; aarch64 c_char is
  u8, so the crate did not compile. Use std::ffi::c_char throughout.
  No behavior change on x86_64 (c_char = i8 there).

@revuto-review revuto-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


Verified the two halves of this PR against the tree:

c_char portability — consistent. Base tree confirms MoeTokenFn/RowsFn in cpu_experts.rs were already c_char and only PrefetchFn + the error buffers were i8; the fix brings them in line. Imports are present at head (cpu_experts.rs:9, cpu_native_check.rs:1), every error-buffer consumer uses CStr::from_ptr(error.as_ptr()) (type-correct under both c_char == i8 and == u8), and the only remaining vec![0i8 (lib.rs:4848) is a device int8 activation buffer through htod_i8, not FFI — correctly left alone. No behavior change on x86_64 as claimed.

sm_121a lane — coherent. The 121a string flows unchanged through gencode (arch=compute_121a,code=sm_121a), MEMRA_BUILT_CUDA_ARCH, and the lib.rs guard tuple, so guard-vs-fatbin can't skew. Dropping ("120a", 12, 1) breaks no working configuration (that combo always died at module load with NO_BINARY_FOR_GPU; the new failure is the early rebuild hint). The mmq_fp4.cu stub-on-non-120a path applies to 121a as on B200 — fail-closed, consistent with the PR scope note. memra-probe/CUTLASS exclusions are explicitly deliberate in the PR body; fine.

One consistency finding below (FLAGS.md / install.sh still encode the old 12.1→120a mapping). Everything else looks good.

let arch = match cap.as_deref() {
Some("12.0") | Some("12.1") => "120a",
Some("12.0") => "120a",
Some("12.1") => "121a",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an auto review done by revuto.


The new 12.1→121a mapping (and the 121a value accepted at build.rs:224) didn't get propagated to the surfaces that document/select this seam, and they now contradict the code:

  • docs/FLAGS.md:218 (the MEMRA_CUDA_ARCH registry row) still says "probe the GPU via nvidia-smi compute_cap (12.x→120a, …)" and enumerates the value set as 120a/90a/100a/89 — no 121a. After this PR that row is wrong on both counts: auto-detect maps 12.1→121a, and a GB10 user who follows the doc and pins MEMRA_CUDA_ARCH=120a gets the fail-closed guard at lib.rs:1102 (which no longer accepts ("120a", 12, 1)). CLAUDE.md:43-45 requires keeping docs/FLAGS.md aligned with the flag surface before integration; this is the exact lane where the doc would mislead.
  • tools/install.sh:39 still maps 12.0|12.1) ARCH=120a — i.e. it selects the sm_120a prebuilt for a compute-cap-12.1 machine, a combination this PR's guard now deterministically rejects at Engine::new (install claims success, every binary refuses to start, and the "rebuild, MEMRA_CUDA_ARCH auto-detects" hint doesn't apply since there's no sm_121a release artifact). Currently dormant because install.sh:25 refuses aarch64 and no x86_64 cc-12.1 part exists, but it re-creates the exact "wrong arch selected silently" class the PR description set out to kill — dropping the 12.1 arm (or pointing it at a source build) would close it.

A one-line FLAGS.md row update (+ optionally install.sh) in this PR would keep the registry truthful for the hardware the lane targets.

@avifenesh
avifenesh requested a lite review from Copilot August 19, 2026 10:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@avifenesh

Copy link
Copy Markdown
Owner

@AnsenIO Hi, thanks for the pr, happy to accept it, please address the review left by revuto.
Are you planning to do tune work on sm121? The strength of the project is tunnig every model on exact hardware, adding general new hardware will probably wont get you the performance results you wants, nor the exact models support.

@avifenesh

Copy link
Copy Markdown
Owner

Closing as stale — no activity since Aug 19 and the review findings (FLAGS.md registry row + install.sh 12.1 arm now contradicting the new guard) were never addressed. The sm_121a lane work itself is solid and appreciated — if you are still on GB10 and want to land this, feel free to reopen (or open a fresh PR) with those two doc/script surfaces aligned and I will review promptly.

@avifenesh avifenesh closed this Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants