Modifying the AutoModelForCausalLM

[RDCordova]

I just want to start by saying I love the work that has been done on this project, Here is the issue I'm having

when the model is loaded from HuggingFace using it would be great to be able to select the paramaters of the AutoModelForCausalLM.
self.model = AutoModelForCausalLM.from_pretrained(self.llm)

It works great with small models likee GPT2 but when we advance to larger models (ex mistralai/Mistral-7B-Instruct-v0.1) the GPU quickly runs out of memory . I can generally get around this by using BitsAndBytesConfig to minmize the memory requiered for the LLM but that requires passing addtinal agrumetns to AutoModelForCausalLM ex

model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.1",
quantization_config=bnb_config,
device_map="auto",
trust_remote_code=True,
)

[unnir]

Thank you, we will add it to the next update.

Would it be possible to share your bnb_config for testing?

[RDCordova]

Thank you for the quick respons. I would be happy to assist with the testing.

[RDCordova]

bnb_config = BitsAndBytesConfig(load_in_4bit= True,bnb_4bit_quant_type= "nf4",bnb_4bit_compute_dtype= torch.float16,bnb_4bit_use_double_quant= True)

[hiberfil]

@unnir I am also trying to solve this issue to be able to run Mistral but even with @RDCordova example I cant get it to run properly. Do you have a timeline when the next version of the GReaT might come out? Happy to help with testing.

Also @RDCordova did you modify and added the bnb config to the great.py or do you have a training script with bnb as arguments to run the script? Do you have a modified script snippet that you can share with us?

Again thank you so much for the awesome work on both ends.