Skip to content

feat(strategy): add nystrom landmark-column transform for low-rank data#194

Merged
zeokin merged 1 commit into
zeokin:mainfrom
andriypolanski:feat/nystrom-transform
Jul 13, 2026
Merged

feat(strategy): add nystrom landmark-column transform for low-rank data#194
zeokin merged 1 commit into
zeokin:mainfrom
andriypolanski:feat/nystrom-transform

Conversation

@andriypolanski

Copy link
Copy Markdown
Contributor

PR kind

  • fix
  • feat

Summary

New registered transform nystrom. Splits the M budget across col(A), row(A), col(B), and row(B) — the same four spaces rsvd sketches — but gathers random landmark columns/rows instead of random projections, then thin-QR into Q.
Basis FLOPs are ~2 N M² (QR only), not rsvd's ~2 N² M sketches.

Honest trade-off (GPU-measured): on --fill lowrank accuracy is 1.0000, VRAM and FLOPs beat exact, but latency is slightly above exact so the dominance gate scores 0 (not an improvement under the one rule). Versus rsvd at the
same accuracy/VRAM it is much faster and fewer FLOPs. Does not touch rsvd.

Fixes #193

Validation

  • python -m strategy.smoke — nystrom and rsvd both PASS
  • python -m pytest strategy/tests/test_nystrom.py tests/ strategy/tests/ eval/tests/ -q
    — 240 passed, 1 failed (tests/test_correctness.py::test_fp16_incore_tiled_parity,
    pre-existing fp16 tolerance flake; unrelated to strategy/)
  • python -m eval … on NVIDIA GeForce RTX 5090 (see scorecard)

GPU Result (required for feat PRs only)

metric value
accuracy 1.0000
time complexity O(N²M) + O(N M²) QR; 2.3× fewer FLOPs vs exact
latency 2423.28 ms
VRAM usage 713.1 MiB

Regime measured: N=12000, dtype=fp32, fill=lowrank, data-rank=16, rank M=1500, device=RTX 5090, seed=0, pairs=3

Comparison (same couples)

metric exact rsvd nystrom (this)
accuracy 1.0 1.0000 1.0000
latency 2301.35 ms 5491.87 ms 2423.28 ms
VRAM 2208.1 MiB 713.1 MiB 713.1 MiB
FLOPs vs exact 1.8× fewer 2.3× fewer
verdict baseline not an improvement (latency ≥ exact) not an improvement (latency ≥ exact)
Raw scorecard (paste python -m eval … output or --json)
$ python -m eval --n 12000 --pairs 3 --fill lowrank --data-rank 16 \
    --rank-m 1500 --transforms nystrom,rsvd --seed 0

[eval] device     : NVIDIA GeForce RTX 5090 (PyTorch/CUDA)
[eval] couples     : 3 x (12000 x 12000)  fill=lowrank  dtype=fp32
[eval] rank_m (M)  : 1500
[eval] transforms  : nystrom, rsvd

=== eval report (n=12000, 3 couples, fp32, fill=lowrank, M=1500, seed=0) ===
  complexity : normal O(N^3)   smart O(N^2 * M)  (M=1500 fixed -> ~O(N^2))
  exact      :  2301.35 ms   peak   2208.1 MiB
  transform   accuracy      latency   peakVRAM   FLOPx        score
  -----------------------------------------------------------------
  nystrom       1.0000    2423.28ms    713.1MiB    2.3x            0  (not an improvement: latency ≥ exact)
  rsvd          1.0000    5491.87ms    713.1MiB    1.8x            0  (not an improvement: latency ≥ exact)
  best (highest score): nystrom

# Companion --fill decaying-spectrum --data-rank 64 (same N/M/device/seed):
  nystrom       1.0000    2425.16ms    713.1MiB    2.3x            0  (not an improvement: latency ≥ exact)
  rsvd          1.0000    5491.46ms    713.1MiB    1.8x            0  (not an improvement: latency ≥ exact)

Checklist

  • CPU-safe validation passed (strategy.smoke if relevant, plus pytest tests/ strategy/tests/ eval/tests/).
  • My commits do not include Co-authored-by footers for coding agents such as Cursor, Codex, Claude, Copilot, or similar tools.
  • If this is a feat PR, I ran the scorer on unseen couples — no hardcoding of seeds/matrices.
  • If this is a feat PR, accuracy and latency come from the same run at the same dtype.
  • If this is a feat PR, this is an improvement (every cost axis down, accuracy held) or I
    state honestly which axis it trades — see the one rule in CONTRIBUTING.md.
    (Honest: accuracy held; VRAM + FLOPs beat exact; latency is slightly above exact → score 0.)
  • Correctness gates pass:
    python eval/tests/test_eval.py,
    python strategy/tests/test_subspace.py,
    python tests/test_correctness.py.
  • If this is a feat PR, I named the device and dtype so a reviewer can reproduce the numbers.

@github-actions github-actions Bot added area:strategy Smart strategies / transforms (strategy/) area:tests Correctness gates and test suites (tests/) area:docs README, CONTRIBUTING, BENCHMARKS and other docs status:needs-review Awaiting maintainer review type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation labels Jul 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Gate chain passed. This PR is queued for the next batched GPU evaluation window.

@zeokin zeokin added eval:L Verified frontier improvement, large tier (>=25%) and removed status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation status:needs-review Awaiting maintainer review labels Jul 13, 2026

@zeokin zeokin left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — eval:L (first tiered frontier result on the low-rank track).

GPU re-measurement (RTX PRO 500 Blackwell, fp32, lowrank rank-16, n=8192, same seed 777):

  • nystrom @ M=64: accuracy 0.99999, latency 0.230s, VRAM 260 MiB, 42x fewer FLOPs, score 17.11 (dominates exact on all axes).
  • 3-way rsvd (current frontier, #156) @ M=64: score 12.85; @ M=48: score 13.51.
  • nystrom beats the frontier by +33% at equal M=64, +27% at each method's natural M -> both clear the >=25% eval:L bar. Stable across 3 seeds (17.07-17.11).

Why it wins: nystrom's landmark-column basis (~NM^2, column gather + QR) is ~100x cheaper in FLOPs than rsvd's Gaussian sketch (~N^2M), so it's faster even using more columns. Merges cleanly (additive; keeps main's 3-way rsvd). Label: eval:L.

@zeokin zeokin merged commit 246807e into zeokin:main Jul 13, 2026
5 checks passed
zeokin added a commit that referenced this pull request Jul 13, 2026
Second real frontier point and first tiered result: nystrom landmark-column
transform beats the low-rank BASELINE (#156, 3-way rsvd) by ~30% (33% at equal
M=64, 27% at natural M) — accuracy 0.99997, 0.23s vs 0.44s exact, 260 MiB, 42x
fewer FLOPs. Regenerate dashboard/results.json from the ledger (build_dashboard_data):
low-rank frontier now nystrom @ 17.11 (BASELINE #156 -> L #194).
zeokin added a commit that referenced this pull request Jul 13, 2026
The mock RTX-5090 test entries seeded the ledger during pipeline bring-up.
Now that the frontier has real, reproducible points (#156 BASELINE, #194 L on
the low-rank track), remove the mocks so the dashboard shows only genuine
results. Full-rank track is now correctly empty (unbeaten by design).
zeokin added a commit that referenced this pull request Jul 13, 2026
The scorer gated every track at --min-accuracy 0.8, but the whitepaper (6.2)
and the dashboard advertise per-track floors: full-rank 0.80, low-rank 0.95,
decaying-spectrum 0.90. Low-rank is the easy case, so it must clear a higher
bar to count.

EvalConfig.accuracy_floor now defaults to None and resolves to the fill's
track floor in __post_init__ (TRACK_FLOORS / default_floor); an explicit
--min-accuracy still overrides. No verdict already recorded changes (#156/#194
clear 0.95). Regression test added.
zeokin added a commit that referenced this pull request Jul 13, 2026
Step 1 of the automated GPU-test workflow. TRACKS pins (fill, rank, M, floor)
per track so a tier is reproducible and a submission can't cherry-pick the
rank/M that flatters its method (the M choice flipped #194 from none to L).
Low-rank pins M=64=4r for rank-16 -- the M where both 3-way and 4-way bases
fully recover, so it's inclusive, not rigged to one basis.

evaluator's per-track accuracy floor now resolves through tracks.track_floor,
so the scorer and the registry can never disagree.
zeokin added a commit that referenced this pull request Jul 13, 2026
…string

The transform-authorship check wrongly flagged an UPDATE to an existing
transform as unverified: modifying RandomizedSVDTransform.basis() (the #156
case) never touches the literal "rsvd", so the name-in-diff check missed it,
while a NEW transform (#194) passed only because it adds `name = "nystrom"` +
register_transform.

Verify by the transform's CLASS instead: map the declared name -> its class via
its `name = "..."` attribute (ast), then check whether the PR's diff hunks land
inside that class's line range. Now a NEW transform (class added -> hunk overlaps
the added range) and an UPDATE (hunk lands inside the existing class) validate
the same way. Pure + unit-tested for both.
RenzoMXD added a commit to RenzoMXD/Cuda-Compute-OSS that referenced this pull request Jul 15, 2026
…ecaying-spectrum data

Replaces this branch's original content (nystrom 3-way split): repeat
GPU testing showed changing 4-way to 3-way landmark sampling does not
reliably improve latency -- the earlier single-run 19% difference did
not reproduce. The redundancy proof (col(B) is unnecessary) is still
mathematically correct, but with no measurable benefit there's no
reason to change the merged 4-way nystrom.

Per docs/developing-new-transforms.md's track guidance: low-rank is
accuracy-saturated (~1.0 for every method already), so the only lever
left there is latency -- exactly the fragile, noise-prone claim that
just failed to hold up. decaying-spectrum is flagged as the track with
real headroom and no admitted baseline yet.

New transform combining three ideas from this project's own history:
- the 3-way space split {col(A), row(A), row(B)} (zeokin#156)
- nystrom's landmark-column seed (zeokin#194): a memory gather, not a GEMM
  against a random Gaussian projection
- power-iteration refinement (prior art: PR zeokin#185, unmerged): concentrates
  a below-rank budget on the components that carry the product's energy
  on a decaying-but-not-low-rank spectrum

zeokin#185 seeded its iterations with random Gaussian noise; its own GPU
measurement showed a real accuracy gain over rsvd on decaying-spectrum
data, but the extra streamed GEMMs were not amortized at n=8192, so it
never beat exact on latency. A landmark column is already correlated
with a matrix's dominant subspace on a decaying spectrum, unlike a
fully random projection, so seeding from landmarks instead should need
fewer refinement passes to reach a given accuracy.

At power_iters=0 this degenerates exactly to plain 3-way nystrom (same
basis_flops), so the CPU-safe suite verifies the iteration genuinely
earns its cost: test_nystrom_iter_beats_plain_nystrom_on_decaying_spectrum
confirms one pass measurably improves accuracy over the landmark seed
alone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs README, CONTRIBUTING, BENCHMARKS and other docs area:strategy Smart strategies / transforms (strategy/) area:tests Correctness gates and test suites (tests/) eval:L Verified frontier improvement, large tier (>=25%) type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[strategy] nystrom transform: landmark column sampling for low-rank matmul

2 participants