rpc error: Error while dialing: dial tcp 127.0.0.1:44557:

[gerroon]

Hi I am getting these errors with an initial test code. I am running it under Docker (Compose) under WSL2

mistral-7b-instruct-v0.2.Q4_K_S.gguf in in the models folder

test post

curl http://DOCKER-IP:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "mistral-7b-instruct-v0.2.Q4_K_S.gguf",
          "prompt": "A long time ago in a galaxy far, far away",
               "temperature": 0.7
                  }'

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:44557: connect: connection refused"

"error":{"code":500,"message":"rpc error: code = Unknown desc = unimplemented","type":""}} 

localai-api-1  | 6:14AM INF Loading model 'mistral-7b-instruct-v0.2.Q4_K_S.gguf' greedly from all the available backends: llama-cpp, 

llama-ggml, llama, gpt4all, gptneox, bert-embeddings, falcon-ggml, gptj, gpt2, dolly, mpt, replit, starcoder, rwkv, whisper, stablediffusion, 

tinydream, piper, /build/backend/python/vllm/run.sh, /build/backend/python/diffusers/run.sh, /build/backend/python/vall-e-x/run.sh, 
/build/backend/python/exllama/run.sh, /build/backend/python/sentencetransformers/run.sh, /build/backend/python/transformers/run.sh, /build/backend/python/exllama2/run.sh, /build/backend/python/petals/run.sh, /build/backend/python/bark/run.sh, /build/backend/python/coqui/run.sh, /build/backend/python/transformers-musicgen/run.sh, /build/backend/python/sentencetransformers/run.sh, /build/backend/python/autogptq/run.sh  localai-api-1  | 6:14AM INF [llama-cpp] 

Attempting to load                                                                                                                                                     localai-api-1  | 6:14AM 

INF Loading model 'mistral-7b-instruct-v0.2.Q4_K_S.gguf' with backend llama-cpp                                                                                                        

localai-api-1  | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:39313: connect: connection refused"                                         

localai-api-1  | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:39313: connect: connection refused"               

[neowisard]

Just use config files https://github.com/mudler/LocalAI/tree/master/examples/configurations/mistral

[lunamidori5]

For installing and testing the best / known models for each size check out the all in one LocalAI Manager!

[gerroon]

Thanks, but I already have localAI docker running, I do not want this to reinstall all that? Can I use it with existing install? It says anythinllm and localai, option 1 just tries to install localai which I do not want.

[gerroon]

| Quant Mode | Description |
|Q3| Smallest, significant quality loss - not recommended|
|Q4| Medium, balanced quality|
|Q5| Large, very low quality loss|
|Q6| Very large, extremely low quality loss|
|Q8| Extremely large, extremely low quality loss, hard to use - not recommended|
|None| Extremely large, No quality loss, super hard to use - really not recommended|

Note: If the model does not support a quant mode, the server will return the next lowest one it has...


What type of quantised model would you like to setup? (q3, q4, q5, q6, q8, none):
q4
I asked What type of quantised model would you like to setup? (q3, q4, q5, q6, q8, none):  / you replied q4
Traceback (most recent call last):
  File "model_installer.py", line 438, in <module>
NameError: name 'service_image' is not defined. Did you mean: 'service_name'?
[27278] Failed to execute script 'model_installer' due to unhandled exception!

[lunamidori5]

@gerroon I have pushed a fix for this error, I am deeply sorry, please try the model installer again!

[gerroon]

I used this one curl -sSL https://tea-cup.midori-ai.xyz/download/model_installer_linux.tar.gz -o model_installer.tar.gz

It still has the issue?