Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions examples/microbenchmarks/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,19 @@ py_binary(
"@pypi//numpy",
],
)

py_binary(
name = "torch_d2d_read_benchmark_runner",
srcs = ["torch_d2d_read_benchmark_runner.py"],
deps = [
"//tpu_sync/api/torch:kv_cache_manager_torch_py",
"//tpu_sync/api/torch:torch_tpu_common_loader",
"//tpu_sync/rpc:coordination_helper",
"@com_google_absl_py//absl:app",
"@com_google_absl_py//absl/flags",
"@pypi//numpy",
"@torch_tpu//shims/torch",
"@torch_tpu//shims/torch:pytorch",
"@torch_tpu//src/torch_tpu",
],
)
73 changes: 69 additions & 4 deletions examples/microbenchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@
This folder contains microbenchmarks for the Raiden engine. They differ in which
part of the transfer path they isolate:

| Benchmark | Measures | Needs TPU? | Needs jax? |
| Benchmark | Measures | Needs TPU? | Framework |
| --- | --- | --- | --- |
| `jax_dma_kv_cache_benchmark.py` | Raw DMA on ONE host: D2H and H2D, against JAX baselines | yes | yes |
| `jax_d2d_read_benchmark_runner.py` | Cross-node device-to-device pull: D2H + H2H + H2D end to end | yes, on both nodes | yes |
| `h2h_benchmark_runner.cc` | The middle hop only: host memory to host memory across the NIC | no | no |
| `jax_dma_kv_cache_benchmark.py` | Raw DMA on ONE host: D2H and H2D, against JAX baselines | yes | JAX |
| `torch_dma_kv_cache_benchmark.py` | Raw DMA on ONE host: D2H and H2D, against PyTorch baselines | yes | PyTorch |
| `jax_d2d_read_benchmark_runner.py` | Cross-node device-to-device pull: D2H + H2H + H2D end to end | yes, on both nodes | JAX |
| `torch_d2d_read_benchmark_runner.py` | Cross-node device-to-device pull: D2H + H2H + H2D end to end | yes, on both nodes | PyTorch |
| `h2h_benchmark_runner.cc` | The middle hop only: host memory to host memory across the NIC | no | None (C++) |

Pick by what you are trying to localise. The C++ H2H runner gives the wire
ceiling; the D2D read runner shows what the device path delivers against that
Expand Down Expand Up @@ -44,3 +46,66 @@ The script prints the median latency (in seconds) and the calculated throughput

When evaluating the performance, look specifically at the **`KVCacheManager D2H bandwidth`** and **`KVCacheManager H2D bandwidth`** lines and compare them against the JAX baselines to observe the throughput gains achieved by bypassing the framework overhead.

---

## `torch_dma_kv_cache_benchmark.py` — single-host raw DMA (PyTorch)

To execute the PyTorch microbenchmark, run:

```bash
PYTHONPATH=../.. python torch_dma_kv_cache_benchmark.py --telemetry_log_path=/tmp/${USER}_torch_benchmark.jsonl
```

### How to Read the Output

The microbenchmark runs across different tensor shapes, layers, and dtypes (`bf16`, `fp32`, `int32`), comparing three implementations:

1. **KVCacheManager**: The TPU Raiden raw DMA engine.
2. **PyTorch Pinned Host Baseline**: Native PyTorch transfers using page-locked host memory (`pin_memory=True`, `non_blocking=True`).
3. **PyTorch Standard Baseline**: Native PyTorch transfers using standard unpinned host memory.

The script logs telemetry with 95% confidence intervals and standard deviation to `--telemetry_log_path`, recording median latency and throughput (GB/s) for both D2H and H2D.

---

## `torch_d2d_read_benchmark_runner.py` — cross-node D2D read (PyTorch)

Measures end-to-end device-to-device KV-cache pull across two nodes: D2H on the sender node, H2H over the network NIC via Raiden C++, and H2D on the receiver node, with byte-for-byte tensor verification.

### Running

Start the **Sender** first:

```bash
cd <tpu-raiden>
PYTHONUNBUFFERED=1 PYTHONPATH=$PWD python3 \
examples/microbenchmarks/torch_d2d_read_benchmark_runner.py \
--role=sender \
--grpc_port=50051 \
--parallelism=4 \
--num_blocks=512 \
--num_layers=8 \
--block_size=16
```

Then start the **Receiver** (pointing to the Sender's network IP):

```bash
cd <tpu-raiden>
PYTHONUNBUFFERED=1 PYTHONPATH=$PWD python3 \
examples/microbenchmarks/torch_d2d_read_benchmark_runner.py \
--role=receiver \
--peer=<SENDER_IP>:50051 \
--parallelism=4 \
--num_blocks=512 \
--num_layers=8 \
--block_size=16
```

### How to Read the Output

The receiver measures:
1. **Transfer Latency:** Total elapsed time from issuing `start_read` until all layers are deposited in destination TPU HBM.
2. **Throughput:** Effective bandwidth achieved in both Gbps and GB/s across the configured parallel streams (`--parallelism`).
3. **Correctness:** Byte-for-byte verification across all layers against the deterministic source sequence (0% corruption).

Loading
Loading