Skip to content

feat(kernels): explicit hd512 sm_scale and the final-logit softcap - #873

Merged
FeathBow merged 1 commit into
pegainfer-project:mainfrom
FeathBow:feat/kernels-hd512-scale-softcap
Aug 14, 2026
Merged

feat(kernels): explicit hd512 sm_scale and the final-logit softcap#873
FeathBow merged 1 commit into
pegainfer-project:mainfrom
FeathBow:feat/kernels-hd512-scale-softcap

Conversation

@FeathBow

Copy link
Copy Markdown
Collaborator

Description

Closes #872

The four hd512 attention wrappers baked in rsqrt(512), and hd512's only prospective consumer cannot use it: Gemma 4 runs unscaled attention (scaling = 1.0 in the HF reference, for both layer kinds). This makes sm_scale an explicit parameter with a finite check, as the hd256 single-prefill wrapper already does. The entries have no current callers, so no call sites move.

Alongside it, the last elementwise piece the 12B forward needs: an in-place final-logit softcap, cap * tanh(x / cap) in f32 with a single rounding back to bf16. Every published Gemma 4 size declares final_logit_softcapping 30.0; nothing in the elementwise family could express it. The wrapper validates the cap positive and finite and the buffer extent under the crate's capacity contract; the launcher computes its grid with an overflow-free ceil-div.

Test Env

Single GPU (sm_89, x86_64), CUDA 12.9, PEGAINFER_REQUIRE_GPU=1.

Verification

  • cargo fmt --all -- --check — clean.
  • cargo build --release -p pegainfer-kernels --tests — clean.
  • cargo clippy --release -p pegainfer-kernels --all-targets -- -D warnings — clean, zero diagnostics.
  • cargo test --release -p pegainfer-kernels --test softcap_smoke — running 2 tests, 2 passed: the softcap op against a host f32 reference on constructed cases, and cap validation (rejects zero/negative/non-finite).

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Signed-off-by: Feathbow <feathbow@gmail.com>
@FeathBow
FeathBow merged commit dcbd6ba into pegainfer-project:main Aug 14, 2026
13 checks passed
@FeathBow
FeathBow deleted the feat/kernels-hd512-scale-softcap branch August 14, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gemma4: hd512 attention wrappers hardcode rsqrt(512), and no op expresses the final-logit softcap

1 participant