Repository navigation
[temp] DS tests against tt-metal cffd1752 + PR 59385 - #9389
Draft
bmalesevicTT wants to merge 8 commits into
Draft
bmalesevicTT wants to merge 8 commits into
bmalesevicTT wants to merge 8 commits into
Conversation
The DS path was the only place the optimizer forced a compute-kernel config on a matmul: math fidelity from the weight dtype, plus packer-L1-accumulate pinned on and fp32 dest-accumulate pinned off. Every other matmul emits none and lets tt-metal derive them. Emit none here too. tt-metal already picks LoFi for any matmul carrying a program config, and derives fp32_dest_acc_en and packer_l1_acc from the output dtype. It also means the compute-kernel-config pipeline options now reach a DS matmul: TTNNSetComputeKernelConfig only fills knobs the op has not already set.
The guard declined DS when the fitted in0_block_w fell below kPerCore/2. Both that constant and the 4 proposed to replace it were fitted to timings taken with DS on HiFi2, a tax only DS paid, which made the curve slope from the start and any early cut look justified. Sweeping in0_block_w at both fidelities on each part shows the cost tracks the block width itself, not its ratio to kPerCore. At LoFi a Wormhole part holds 86-90% of DRAM peak from w=30 down to w=3 and breaks at w=2 (72%) and w=1 (45%); Blackhole holds 96-98% down to w=6, then 78% at w=3 and 57% at w=2. Read against the ratio there is no monotone relationship at all -- ratio 3 gives 86%, ratio 18 gives 90%, ratio 45 gives 72% -- and a ratio cut declines the whole plateau on any shape with a large kPerCore: at kPerCore 90 it demanded w >= 45. Per-bank burst size is ruled out as the invariant too: Wormhole's best rung is its second-smallest burst, 51 KB, where Blackhole is already down to 78% at 38 KB. Across the 71 DS-affected matmul groups on Wormhole the two rules differ on 18. The floor keeps five high-w configs the ratio cut discarded, worth +248us of matmul device time re-measured with both kernels at LoFi, and declines thirteen w=2 groups the ratio cut kept because kPerCore was 2 there and it never fired. 3 is what the Wormhole ladder supports; Blackhole's knee sits higher, so 4 would buy margin there for ~52us of measured Wormhole time. Bfp4NeverFitsWorseThanBfp8 sweeps lower budgets now: the floor is more permissive, so the dtype term only separates the two below ~500000. Also decline DS when the matmul's result feeds a collective. A more permissive floor makes this reachable: on qb2 (Blackhole, TP=4) it flips exactly one shape per model from a 1D multicast on an 11x8 grid to DS, once per layer, and every flip regressed. llama_3_1_70b down_proj K=7168 N=8192 w=7 80 ops -7.5% qwen_2_5_coder_32b down_proj K=6912 N=5120 w=9 64 ops -5.8% mistral_small_24b gate+up K=5120 N=8192 w=5 80 ops -2.5% Prefill graphs are byte-identical between the two builds and the decode matmul counts are unchanged, so the whole regression attributes to those ops: +71, +49 and +10us each. It is not the DS config -- all three sit at ratio 3-4 with a 170-238 KB per-bank burst, inside the plateau above, and the largest burst is the worst outcome. Each flip inserts exactly one ToMemoryConfigOp, and what differs is what that reshard lands in front of: down_proj is row-parallel and feeds an all_reduce, gate/up is column-parallel and feeds nothing, a 5-7x difference for the same flip on the same silicon. On single-chip p150 the identical flip on falcon3_7b's down projection is a net +4.6%, because there is no collective. No CCL implements the op-model interface -- AllGatherOp, AllReduceOp, ReduceScatterOp and PointToPointOp all carry OpModelExempt because tt-metal exposes no constraint or runtime query for them -- so the optimizer can cost the matmul but not the collective behind it, and cannot weigh the reshard against the DS win. Revisit when they become costable. The check sits in buildDSPlan beside the disable-option gate rather than in isDSEligible: it is a property of the op's consumers, not of its operands, and buildDSPlan is the choke point every DS entry point reaches. The consumer walk looks through view-like ops, because the inserted reshard sits between the matmul and the collective once the graph has been rewritten.
applyOpSpecificAttrs still carried a compute-config branch for DS hints that no longer carry one, and the header still promised one. The L1-interleaved summary also still cited the HiFi4 fallback the .cpp rationale no longer claims.
… XFAILs Temporary probe: tt-metal branch bmalesevic/temp-cffd1752-dfb-mock-fix is the current pin with PR 59385 cherry-picked, so the DS lit tests should pass outright instead of XFAIL.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9389 +/- ##
========================================
Coverage 73.85% 73.85%
========================================
Files 515 515
Lines 108030 108130 +100
========================================
+ Hits 79785 79859 +74
- Misses 28245 28271 +26 ☔ View full report in Codecov by Harness. |
The rms_norm sharding test passes again, and sdpa_fw's query from an L1-sharded producer goes back to two reshards: the single hop encoded the broken reshard validation, not a fold.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI probe only, do not merge. Pins tt-metal to
f7c86f43c3= the current pincffd1752plus tt-metal PR 59385 (the mock-device borrowed-DFB fix for tt-metal#57022), on top of #9252, with the five DS lit XFAILs removed so they must pass outright.