UPSTREAM PR #18535: ggml: add env var GGML_OP_OFFLOAD_MIN_BATCH

[loci-dev]

Mirrored from ggml-org/llama.cpp#18535

ggml-org/llama.cpp#18530 / ggml-org/llama.cpp#17026

Makes the min_batch_size for triggering op offload configurable via env var, defaulting to the prior hardcoded value of 32 if not specified to keep current behavior intact.

This is helpful when running large MoEs with a significant size of weights stored in host buffers on CPU, causing a bottleneck when op offloading with small batches that are still larger than the default 32. The optimal value, or "break even point" here depends on characteristics of the hardware + model, and is best determined empirically (ref: ggml-org/llama.cpp#17026 (comment)).

Make sure to read the contributing guidelines before submitting a PR

[loci-agentic-ai]

Explore the complete analysis inside the Version Insights

Perfect! I've generated the summary report for your project. Here are the key findings:

Summary Report for llama.cpp PR #781

Key Highlights:

• ✅ No significant performance changes detected - All modified functions show performance changes within ±2% threshold
• ✅ Both response time and throughput remain stable
• ✅ Low risk for performance regressions

The analysis compared:

• Base version: cfc7f9d0-e773-11f0-81f2-dbb430499cb5
• Target version: 23f02221-e7a7-11f0-81f2-dbb430499cb5

This is a positive result indicating that the changes in Pull Request #781 maintain performance stability and can proceed without performance concerns.

Would you like more detailed analysis on any specific aspects of the report?