Skip to content

[CUDA] Faster general copy#2873

Merged
awni merged 1 commit into
ml-explore:mainfrom
awni:cuda_faster_general_copy
Dec 5, 2025
Merged

[CUDA] Faster general copy#2873
awni merged 1 commit into
ml-explore:mainfrom
awni:cuda_faster_general_copy

Conversation

@awni

@awni awni commented Dec 5, 2025

Copy link
Copy Markdown
Member

Use a larger work per thread for general to contiguous copies when possible.

On B200:

Type Pre ms Post ms
bf16 140.536 94.853
fp32 135.735 77.529

Pretraining 4B model:

Pre: 26580.4548 tok/s
Post: 27443.1227 tok/s

Microbench script:

import mlx.core as mx
import time

B = 8
T = 2048
H = 32
D = 128

for t in [mx.float32, mx.bfloat16]:
    x = mx.random.normal(shape=(B, T, H, D)).astype(t)

    def fun(x):
        for _ in range(50):
            x = mx.contiguous(x.swapaxes(1, 2))
        return x

    for _ in range(20):
        mx.eval(fun(x))

    tic = time.time()
    for _ in range(20):
        mx.eval(fun(x))
    toc = time.time()

    s = toc - tic
    gb = x.nbytes * 2 * 50 * 20 / 1e9
    gbps = gb / s
    ms = s * 1e3
    print(f"{gbps=:.3f}, {ms=:.3f}")
    ```

@awni awni requested review from angeloskath and zcbenz December 5, 2025 03:50

@zcbenz zcbenz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

@awni awni merged commit aefc9bd into ml-explore:main Dec 5, 2025
12 checks passed
@awni awni deleted the cuda_faster_general_copy branch December 9, 2025 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants