Dual ANE Cluster on Ultra models

[Anemll]

We have received M3 Ultra and M1 Ultra reports for parallel model execution.
Unfortunately, neither shows bandwidth increase.
Either ANE service is not placing execution on different clusters, or memory I/O is somehow limited to 120GB/s for ANE on mac studio Ultra models.

We'll need to add some TOPS and FLOPS benchmarks to investigate further or get direct access to M3 Ultra units

{
"timestamp": "2025-03-12 09:33:07",
"system_info": {
"os": {
"name": "Darwin",
"version": "Darwin Kernel Version 24.3.0: Thu Jan 2 20:31:43 PST 2025; root:xnu-11215.81.4~4/RELEASE_ARM64_T6031",
"release": "24.3.0"
},
"cpu": {
"brand": "Apple M3 Ultra",
"architecture": "arm64",
"cores": 32,
"threads": 32
},
"ram": {
"total_gb": 512.0,
"available_gb": 487.37
},
"python_version": "3.12.9",
"mac_model": "Mac15,14",
"macos_version": "macOS Sequoia 15.3 (24D2059)",
"apple_silicon": true,
"neural_engine": {
"available": true,
"cores": "Unknown",
"tflops": "Unknown"
}
},
"results": [
{
"model_name": "llama_lm_head",
"backend": "ANE",
"inference_time_ms": 8.71199369430542,
"memory_used_mb": 0.0,
"throughput_gb_s": 120.60083660147403,
"tflops": null,
"input_shape": "[1, 1, 4096]",
"model_size_mb": 1050.673728
},
{
"model_name": "DeepHermes_lm_head",
"backend": "ANE",
"inference_time_ms": 8.70786984761556,
"memory_used_mb": 0.0,
"throughput_gb_s": 120.65795038125215,
"tflops": null,
"input_shape": "[1, 1, 4096]",
"model_size_mb": 1050.673728
},
{
"model_name": "llama_lm_head (Parallel Run)",
"backend": "ANE",
"inference_time_ms": 17.02842950820923,
"memory_used_mb": 0.0,
"throughput_gb_s": 61.701152622059546,
"tflops": null,
"input_shape": "[1, 1, 4096]",
"model_size_mb": 1050.673728
},
{
"model_name": "DeepHermes_lm_head (Parallel Run)",
"backend": "ANE",
"inference_time_ms": 17.346296310424805,
"memory_used_mb": 0.0,
"throughput_gb_s": 60.570493504631564,
"tflops": null,
"input_shape": "[1, 1, 4096]",
"model_size_mb": 1050.673728
},
{
"model_name": "Combined (llama_lm_head + DeepHermes_lm_head)",
"backend": "ANE",
"inference_time_ms": 17.346296310424805,
"memory_used_mb": 0.0,
"throughput_gb_s": 120.93652731402324,
"tflops": null,
"input_shape": "[]",
"model_size_mb": 2101.347456
}
]
}

[kdkd]

I have an M3 Ultra 512GB and tried playing with this - not sure if this confirms your suspicion or not, but observing things in "asitop" while it was running the dual benchmarks, I never saw the ANE usage go above about 47%. I'm happy to try running anything you'd like to experiment here.

=== Dual Model Benchmark Results ===

Individual Performance:

• llama_lm_head: 8.84 ms, 118.87 GB/s
• DeepHermes_lm_head: 8.79 ms, 119.48 GB/s

Parallel Performance:

• llama_lm_head: 17.40 ms, 60.37 GB/s
• DeepHermes_lm_head: 17.55 ms, 59.88 GB/s
• Combined: 17.55 ms, 119.55 GB/s (total throughput)

Combined Analysis:

• Total Parallel Execution Time: 5.27 seconds
• Combined Throughput: 119.55 GB/s
• Sum of Individual Throughputs: 238.35 GB/s
• Bandwidth Utilization Factor: 0.50x
• Efficiency: 50.16%

Conclusion: Low bandwidth utilization. Models might be competing for resources.
Combined performance: 17.55 ms, 119.55 GB/s

[Anemll]

Thanks, looks right. One possible workflow is to use swift instead of python. Swift code is in this branch https://github.com/Anemll/Anemll/tree/swift-inference Perhaps you can experiment with it by launching as two separate apps Also, If you can post python examples/benchmark_all_models.py for m3 ultra, it will be useful as well.

…

On Fri, Mar 14, 2025 at 10:12 AM kdkd ***@***.***> wrote: I have an M3 Ultra 512GB and tried playing with this - not sure if this confirms your suspicion or not, but observing things in "asitop" while it was running the dual benchmarks, I never saw the ANE usage go above about 47%. I'm happy to try running anything you'd like to experiment here. === Dual Model Benchmark Results === Individual Performance: - llama_lm_head: 8.84 ms, 118.87 GB/s - DeepHermes_lm_head: 8.79 ms, 119.48 GB/s Parallel Performance: - llama_lm_head: 17.40 ms, 60.37 GB/s - DeepHermes_lm_head: 17.55 ms, 59.88 GB/s - Combined: 17.55 ms, 119.55 GB/s (total throughput) Combined Analysis: - Total Parallel Execution Time: 5.27 seconds - Combined Throughput: 119.55 GB/s - Sum of Individual Throughputs: 238.35 GB/s - Bandwidth Utilization Factor: 0.50x - Efficiency: 50.16% Conclusion: Low bandwidth utilization. Models might be competing for resources. Combined performance: 17.55 ms, 119.55 GB/s — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BOG4QS5PKWF75J3FAIBHZAT2UMEWRAVCNFSM6AAAAABY6QP4TGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRVGI4DAOJVGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: kdkd]*kdkd* left a comment (Anemll/anemll-bench#8) <#8 (comment)> I have an M3 Ultra 512GB and tried playing with this - not sure if this confirms your suspicion or not, but observing things in "asitop" while it was running the dual benchmarks, I never saw the ANE usage go above about 47%. I'm happy to try running anything you'd like to experiment here. === Dual Model Benchmark Results === Individual Performance: - llama_lm_head: 8.84 ms, 118.87 GB/s - DeepHermes_lm_head: 8.79 ms, 119.48 GB/s Parallel Performance: - llama_lm_head: 17.40 ms, 60.37 GB/s - DeepHermes_lm_head: 17.55 ms, 59.88 GB/s - Combined: 17.55 ms, 119.55 GB/s (total throughput) Combined Analysis: - Total Parallel Execution Time: 5.27 seconds - Combined Throughput: 119.55 GB/s - Sum of Individual Throughputs: 238.35 GB/s - Bandwidth Utilization Factor: 0.50x - Efficiency: 50.16% Conclusion: Low bandwidth utilization. Models might be competing for resources. Combined performance: 17.55 ms, 119.55 GB/s — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BOG4QS5PKWF75J3FAIBHZAT2UMEWRAVCNFSM6AAAAABY6QP4TGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRVGI4DAOJVGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>