Recommended VRAM guidance is currently hand-written per playbook, which is one-size-fits-all and open to human error as models and devices vary. We need a way to consistently resolve a recommended VRAM allocation from device + model load and surface it to the user.
Considerations for whatever solution we land on:
- Inputs available from GGUF metadata + context;
overhead a tunable constant
- CI could print measured VRAM as a drift check to catch calc divergence
Recommended VRAM guidance is currently hand-written per playbook, which is one-size-fits-all and open to human error as models and devices vary. We need a way to consistently resolve a recommended VRAM allocation from device + model load and surface it to the user.
Considerations for whatever solution we land on:
overheada tunable constant