Skip to content

feat(strategy): rsvd captures 3 subspaces, not 4 (drop redundant col(B) sketch)#156

Merged
zeokin merged 1 commit into
zeokin:mainfrom
ultrahighsuper:feat/rsvd-three-way-budget
Jul 13, 2026
Merged

feat(strategy): rsvd captures 3 subspaces, not 4 (drop redundant col(B) sketch)#156
zeokin merged 1 commit into
zeokin:mainfrom
ultrahighsuper:feat/rsvd-three-way-budget

Conversation

@ultrahighsuper

Copy link
Copy Markdown
Contributor

PR kind

  • fix
  • feat

Reopened as feat/strategy per @zeokin's request on #132 ("changes core rsvd strategy behavior and accuracy … reopen as a feat/strategy PR").

Summary

The subspace reconstruction is Ĉ = P·A·P·B·P with P = QQᵀ, which equals A@B once range(Q) contains col(A), row(A), row(B):

P A = A  needs col(A);   A P = A  needs row(A);   B P = B  needs row(B)
=> P A P B P = A·B·P = A·B

col(B) is never needed. RandomizedSVDTransform split the M-column budget four ways (col A, row A, col B, row B), so a quarter of every rank went to a subspace that does not affect the result — exact recovery of a rank-r product needed M ≈ 4r. Splitting three ways over {col(A), row(A), row(B)} recovers at M ≈ 3r: a 25% smaller subspace for the same accuracy (and strictly better accuracy at any fixed M < 4r). Total sketch width is still m, so basis_flops (2n²m + 2nm²) is unchanged.

Fixes #91.

GPU Result (compressible regime — lowrank rank-16, M=3r=48, GTX 1650, fp32)

The reference regime is full-rank, where no subspace beats exact (accuracy ≈ 0) — that is unchanged and honestly still scores 0. The improvement is on compressible data, where the subspace method is designed to win; there this change turns a gated failure into a dominant improvement:

metric exact rsvd 4-way (before) rsvd 3-way (this PR)
accuracy 1.0 (truth) 0.35 → gated, score 0 1.0000
latency (N=8192) 1774 ms 749 ms
VRAM (N=8192) 1288 MiB 259 MiB
FLOPs vs exact 21–42× fewer 21–42× fewer
verdict baseline not an improvement improvement (score 5.28)

Raw scorecard (python -m eval --n 8192 --pairs 3 --fill lowrank --data-rank 16 --rank-m 48 --transforms rsvd --seed 0):

=== eval report (n=8192, 3 couples, fp32, fill=lowrank, M=48, seed=0) ===
  exact      :  1774.19 ms   peak   1288.1 MiB
  transform   accuracy      latency   peakVRAM   FLOPx        score
  rsvd          1.0000     748.66ms    259.0MiB   42.4x         5.28  (improvement)

The before/after is fully reproducible at the same seed: old 4-way rsvd at M=48 gates at accuracy 0.3491 → score 0; 3-way scores an improvement. Measured on a GTX 1650 (named per CONTRIBUTING); reproduce on the RTX 5090 reference at N=12000.

CPU-safe validation

$ python -m strategy.smoke            # 1/1 transforms pass
$ python -m pytest strategy/tests/ -q  # 15 passed

test_exact_when_m_equals_n and the rsvd low-rank recovery/monotonicity tests still pass. Device-independent proof (numpy): dropping col(B) keeps exact reconstruction (rel-error 1e-15); dropping col(A) or row(B) breaks it (~0.9).

Scope

  • strategy/transforms.py only (Open — "the main event" zone); no API change, basis_flops unchanged, no protected-path edits.

@github-actions github-actions Bot added area:strategy Smart strategies / transforms (strategy/) status:needs-review Awaiting maintainer review type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation labels Jul 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Gate chain passed. This PR is queued for the next batched GPU evaluation window.

@github-actions

Copy link
Copy Markdown
Contributor

This PR currently has merge conflicts with the base branch. Please resolve them within 12 hours or the bot will close the PR automatically.

@github-actions github-actions Bot removed the status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation label Jul 13, 2026
The subspace reconstruction is Ĉ = P A P B P with P = QQᵀ, which equals A@B
once range(Q) contains col(A), row(A), row(B):

    P A = A  needs col(A);  A P = A  needs row(A);  B P = B  needs row(B)
    => P A P B P = A B P = A B

col(B) is never needed. RandomizedSVDTransform split the M budget four ways
(col A, row A, col B, row B), so a quarter of every rank went to a subspace that
does not affect the result -- exact recovery of a rank-r product needed M ~ 4r.
Splitting three ways over {col(A), row(A), row(B)} recovers at M ~ 3r, a 25%
smaller subspace for the same accuracy (and better accuracy at any fixed
M < 4r). Total sketch width is still m, so basis_flops is unchanged.

Measured improvement over exact on compressible data at M = 3r (GTX 1650, fp32,
N=8192, lowrank rank-16, seed 0): accuracy 1.0000, 748ms vs 1774ms exact, 259
vs 1288 MiB, 42.4x fewer FLOPs. The old 4-way split at the same M=48 gates at
accuracy 0.35 (score 0); the 3-way split scores an improvement.

Fixes zeokin#91
@zeokin zeokin force-pushed the feat/rsvd-three-way-budget branch from 7c7fe70 to 25bd375 Compare July 13, 2026 12:47
@github-actions github-actions Bot added area:tests Correctness gates and test suites (tests/) status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation labels Jul 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Gate chain passed. This PR is queued for the next batched GPU evaluation window.

@zeokin zeokin left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — feat/strategy verdict after GPU re-measurement (RTX PRO 500 Blackwell, fp32).

Math verified independently (numpy) and confirmed by this run: 3-way rsvd {col(A), row(A), row(B)} recovers the low-rank product exactly (accuracy 0.99997), so col(B) is provably redundant.

GPU reproduction (lowrank rank-16, M=48, n=8192):

  • seeded (seed 0): accuracy 0.99997, latency 0.295s, peak VRAM 259 MiB, 42.4x fewer FLOPs — dominates exact (0.44s / 1032 MiB) on all three cost axes.
  • fresh unseen (2 runs, no --seed): accuracy 0.99996 / 0.99998 — stable; improvement:true both.
  • full-rank safety: gated to ~0 (no regression on the reference regime).

Rebased onto current main during review: the 3-way change was combined with #211's vram_fraction threading (the sketch block was the only conflict); #211's frac test updated 4->3 to match; 262 tests green; Eval policy CI green on the rebased head.

Label: eval:BASELINE — first admitted strategy on the low-rank track. Scope: strategy/transforms.py (+ the merged test adjustment). Thanks for the clean derivation.

@zeokin zeokin merged commit 64e1d86 into zeokin:main Jul 13, 2026
6 checks passed
@zeokin zeokin added eval:BASELINE First admitted strategy on this track and removed status:queued-gpu Feat/strategy PR passed non-GPU gates and is queued for the next batched GPU evaluation status:needs-review Awaiting maintainer review labels Jul 13, 2026
zeokin added a commit that referenced this pull request Jul 13, 2026
Change the pinned reference regime from N=12000 to N=8192 across CLI/config
defaults, docs, and the GPU-batch workflow, so it matches the actual scoring
hardware and the first admitted low-rank entry (#156). Reference device stays
RTX 5090.

Derived defaults updated to match: M = N//8 is now 1024 (was 1500), and the
example VRAM figure ~0.8 GB (was ~1.7 GB). No semantics change beyond the
default N; accuracy/dominance gates unchanged. 262 tests pass.
zeokin added a commit that referenced this pull request Jul 13, 2026
First real (non-mock) ledger entry. rsvd 3-way subspace on the low-rank track
(rank-16, n=8192, M=48): accuracy 0.99997, dominates exact on latency/VRAM/FLOPs
(0.30s vs 0.44s, 259 vs 1032 MiB, 42x fewer FLOPs). Measured on RTX PRO 500
Blackwell; reference device is RTX 5090. verdict BASELINE.
zeokin added a commit that referenced this pull request Jul 13, 2026
Second real frontier point and first tiered result: nystrom landmark-column
transform beats the low-rank BASELINE (#156, 3-way rsvd) by ~30% (33% at equal
M=64, 27% at natural M) — accuracy 0.99997, 0.23s vs 0.44s exact, 260 MiB, 42x
fewer FLOPs. Regenerate dashboard/results.json from the ledger (build_dashboard_data):
low-rank frontier now nystrom @ 17.11 (BASELINE #156 -> L #194).
zeokin added a commit that referenced this pull request Jul 13, 2026
The mock RTX-5090 test entries seeded the ledger during pipeline bring-up.
Now that the frontier has real, reproducible points (#156 BASELINE, #194 L on
the low-rank track), remove the mocks so the dashboard shows only genuine
results. Full-rank track is now correctly empty (unbeaten by design).
zeokin added a commit that referenced this pull request Jul 13, 2026
The scorer gated every track at --min-accuracy 0.8, but the whitepaper (6.2)
and the dashboard advertise per-track floors: full-rank 0.80, low-rank 0.95,
decaying-spectrum 0.90. Low-rank is the easy case, so it must clear a higher
bar to count.

EvalConfig.accuracy_floor now defaults to None and resolves to the fill's
track floor in __post_init__ (TRACK_FLOORS / default_floor); an explicit
--min-accuracy still overrides. No verdict already recorded changes (#156/#194
clear 0.95). Regression test added.
zeokin added a commit that referenced this pull request Jul 13, 2026
…string

The transform-authorship check wrongly flagged an UPDATE to an existing
transform as unverified: modifying RandomizedSVDTransform.basis() (the #156
case) never touches the literal "rsvd", so the name-in-diff check missed it,
while a NEW transform (#194) passed only because it adds `name = "nystrom"` +
register_transform.

Verify by the transform's CLASS instead: map the declared name -> its class via
its `name = "..."` attribute (ast), then check whether the PR's diff hunks land
inside that class's line range. Now a NEW transform (class added -> hunk overlaps
the added range) and an UPDATE (hunk lands inside the existing class) validate
the same way. Pure + unit-tested for both.
RenzoMXD added a commit to RenzoMXD/Cuda-Compute-OSS that referenced this pull request Jul 15, 2026
…ecaying-spectrum data

Replaces this branch's original content (nystrom 3-way split): repeat
GPU testing showed changing 4-way to 3-way landmark sampling does not
reliably improve latency -- the earlier single-run 19% difference did
not reproduce. The redundancy proof (col(B) is unnecessary) is still
mathematically correct, but with no measurable benefit there's no
reason to change the merged 4-way nystrom.

Per docs/developing-new-transforms.md's track guidance: low-rank is
accuracy-saturated (~1.0 for every method already), so the only lever
left there is latency -- exactly the fragile, noise-prone claim that
just failed to hold up. decaying-spectrum is flagged as the track with
real headroom and no admitted baseline yet.

New transform combining three ideas from this project's own history:
- the 3-way space split {col(A), row(A), row(B)} (zeokin#156)
- nystrom's landmark-column seed (zeokin#194): a memory gather, not a GEMM
  against a random Gaussian projection
- power-iteration refinement (prior art: PR zeokin#185, unmerged): concentrates
  a below-rank budget on the components that carry the product's energy
  on a decaying-but-not-low-rank spectrum

zeokin#185 seeded its iterations with random Gaussian noise; its own GPU
measurement showed a real accuracy gain over rsvd on decaying-spectrum
data, but the extra streamed GEMMs were not amortized at n=8192, so it
never beat exact on latency. A landmark column is already correlated
with a matrix's dominant subspace on a decaying spectrum, unlike a
fully random projection, so seeding from landmarks instead should need
fewer refinement passes to reach a given accuracy.

At power_iters=0 this degenerates exactly to plain 3-way nystrom (same
basis_flops), so the CPU-safe suite verifies the iteration genuinely
earns its cost: test_nystrom_iter_beats_plain_nystrom_on_decaying_spectrum
confirms one pass measurably improves accuracy over the landmark seed
alone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:strategy Smart strategies / transforms (strategy/) area:tests Correctness gates and test suites (tests/) eval:BASELINE First admitted strategy on this track type:feature Feature / improvement PR that must carry a GPU scorecard type:strategy New smart strategy / transform proposal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] rsvd wastes ~25% of the rank budget: Ĉ = P·A·P·B·P needs only 3 of its 4 sketched subspaces

2 participants