Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: automatically adjust default gpu_layers by available GPU memory #3541

Open
mudler opened this issue Sep 13, 2024 · 0 comments
Open

feat: automatically adjust default gpu_layers by available GPU memory #3541

mudler opened this issue Sep 13, 2024 · 0 comments
Labels
enhancement New feature or request roadmap

Comments

@mudler
Copy link
Owner

mudler commented Sep 13, 2024

Is your feature request related to a problem? Please describe.
Having defaults high number of GPU layers doesn't always work. For instance big models can overfit the card and constrain the user to configure gpu_layers manually

Describe the solution you'd like
With libraries like https://github.com/gpustack/gguf-parser-go we could get along and identify beforeahead how much gpu vram could be used and adjust the default settings

Describe alternatives you've considered
Keep things as is

Additional context

@mudler mudler added enhancement New feature or request roadmap labels Sep 13, 2024
@mudler mudler changed the title automatically adjust default gpu_layers by available GPU memory feat: automatically adjust default gpu_layers by available GPU memory Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request roadmap
Projects
None yet
Development

No branches or pull requests

1 participant