Skip to content
This repository has been archived by the owner on Jun 24, 2024. It is now read-only.

Support for Mistral-7b #434

Open
ranaya-formant opened this issue Sep 27, 2023 · 5 comments
Open

Support for Mistral-7b #434

ranaya-formant opened this issue Sep 27, 2023 · 5 comments
Labels
issue:enhancement New feature or request topic:model-support Support for new models

Comments

@ranaya-formant
Copy link

Do you all support this? https://mistral.ai/news/announcing-mistral-7b/

@codesoda
Copy link

huggingface/candle#983

@philpax
Copy link
Collaborator

philpax commented Oct 31, 2023

Apologies for the late reply on this. I've tested two Mistral 7B-derived models (https://huggingface.co/TheBloke/Mistral-7B-Claude-Chat-GGUF and https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF) with #412, and all seems to work. I'll keep this issue open until that PR lands.

@philpax philpax added issue:enhancement New feature or request topic:model-support Support for new models labels Oct 31, 2023
@svenstaro
Copy link

Is there an update on this?

@svenstaro
Copy link

svenstaro commented Dec 3, 2023

Ok I sort of figured it out. For others looking into this:

  • Use the GGUF branch
  • Run it like this: cargo run --release -- infer -m ~/llms/mistral-7b-instruct-v0.1.Q4_K_S.gguf -p "Write a long story" -n 5000 -r mistralai/Mistral-7B-v0.1
    The important bit is to tell it to use the tokenizer at the end.

@philpax
Copy link
Collaborator

philpax commented Dec 4, 2023

Yup - the gguf branch's embedded tokenizer support doesn't quite work, so I've disabled it for now. I'm rebuilding the library in develop to target the latest llama.cpp, but that's going to take some time.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue:enhancement New feature or request topic:model-support Support for new models
Projects
None yet
Development

No branches or pull requests

4 participants