Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

diffusers backend does not load in hiplas docker image (torchvision:nms does not exist) #2737

Closed
jbash opened this issue Jul 6, 2024 · 1 comment · Fixed by #3194
Closed

diffusers backend does not load in hiplas docker image (torchvision:nms does not exist) #2737

jbash opened this issue Jul 6, 2024 · 1 comment · Fixed by #3194
Labels
bug Something isn't working unconfirmed

Comments

@jbash
Copy link

jbash commented Jul 6, 2024

LocalAI version:
quay.io/go-skynet/local-ai:v2.18.1-hipblas-ffmpeg

Environment, CPU architecture, OS, and Version:

Run under rootless podman on Fedora 39.

  • uname -a: Linux mctl.kdjf.net 6.9.4-100.fc39.x86_64 &hash;1 SMP PREEMPT_DYNAMIC Wed Jun 12 13:37:46 UTC 2024 x86_64 GNU/Linux
  • System ROCM version: 5.7.1.
  • CPU: Xeon E5-2640
  • GPUs: AMD Radion RX 6500 (used for video), AMD Radeon RX 7900 XT (compute only)
  • System RAM: 128MB

Run with

podman run -p 4480:8080 \
    -v ~/local/lib/ml/llm/models/:/models:z \
    --device=/dev/kfd --device=/dev/dri/card0 --device=/dev/dri/renderD128 \
    --device=/dev/dri/card1 --device=/dev/dri/renderD129 \
    quay.io/go-skynet/local-ai:v2.18.1-hipblas-ffmpeg  --models-path=/models --log-level=debug

Describe the bug

Trying to generate an image using the built-in UI fails.

  • After a significant pause, the Web UI says "grpc service not ready".
  • The non-debug log says " failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:37429: connect: connection refused"".
  • The backend output in the debug log shows a traceback complaining (as the root cause) that "operator torchvision::nms does not exist".

To Reproduce

  • Run Docker image as above
  • Point a Web browser to http://localhost:4480
  • Select image generation
  • Choose the dreamshaper model
  • Type a prompt

Expected behavior

An image should show up.

Logs

time="2024-07-06T17:31:06-04:00" level=warning msg="Error validating CNI config file /home/jbash/.config/cni/net.d/betterchatgpt_default.conflist: [failed to find plugin \"bridge\" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin \"portmap\" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin \"firewall\" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin \"tuning\" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]"
@@@@@
Skipping rebuild
@@@@@
If you are experiencing issues with the pre-compiled builds, try setting REBUILD=true
If you are still experiencing issues with the build, try setting CMAKE_ARGS and disable the instructions set as needed:
CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF"
see the documentation at: https://localai.io/basics/build/index.html
Note: See also https://github.com/go-skynet/LocalAI/issues/288
@@@@@
CPU info:
model name      : Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts vnmi md_clear flush_l1d
CPU:    AVX    found OK
CPU:    AVX2   found OK
CPU: no AVX512 found
@@@@@
9:31PM INF env file found, loading environment variables from file envFile=.env
9:31PM DBG Setting logging to debug
9:31PM INF Starting LocalAI using 10 threads, with models path: /models
9:31PM INF LocalAI version:  ()
9:31PM DBG CPU capabilities: [3dnowprefetch abm acpi adx aes aperfmperf apic arat arch_perfmon avx avx2 bmi1 bmi2 bts cat_l3 cdp_l3 clflush cmov constant_tsc cpuid cpuid_fault cqm cqm_llc cqm_mbm_local cqm_mbm_total cqm_occup_llc cx16 cx8 dca de ds_cpl dtes64 dtherm dts epb ept ept_ad erms est f16c flexpriority flush_l1d fma fpu fsgsbase fxsr hle ht ibpb ibrs ida intel_ppin intel_pt invpcid lahf_lm lm mca mce md_clear mmx monitor movbe msr mtrr nonstop_tsc nopl nx pae pat pbe pcid pclmulqdq pdcm pdpe1gb pebs pge pln pni popcnt pse pse36 pti pts rdrand rdseed rdt_a rdtscp rep_good rtm sdbg sep smap smep smx ss ssbd sse sse2 sse4_1 sse4_2 ssse3 stibp syscall tm tm2 tpr_shadow tsc tsc_adjust tsc_deadline_timer vme vmx vnmi vpid x2apic xsave xsaveopt xtopology xtpr]
9:31PM DBG GPU count: 2
9:31PM DBG GPU: card #0  [affined to NUMA node 1]@0000:83:00.0 -> driver: 'amdgpu' class: 'Display controller' vendor: 'Advanced Micro Devices, Inc. [AMD/ATI]' product: 'unknown'
9:31PM DBG GPU: card #1  [affined to NUMA node 0]@0000:04:00.0 -> driver: 'amdgpu' class: 'Display controller' vendor: 'Advanced Micro Devices, Inc. [AMD/ATI]' product: 'unknown'
9:31PM DBG Model file loaded: deepseek-coder-33b-instruct.Q4_K_M.gguf architecture=llama bosTokenID=32013 eosTokenID=32021 modelName=deepseek-ai_deepseek-coder-33b-instruct
9:31PM DBG guessDefaultsFromFile: family not identified
9:31PM DBG Model file loaded: dolphin-2.7-mixtral-8x7b.Q4_0.gguf architecture=llama bosTokenID=1 eosTokenID=32000 modelName=cognitivecomputations_dolphin-2.7-mixtral-8x7b
9:31PM DBG guessDefaultsFromFile: family not identified
9:31PM DBG guessDefaultsFromFile: not a GGUF file
9:31PM DBG guessDefaultsFromFile: template already set name=poppy_porpoise-v1.4-l3-8b-iq-imatrix
9:31PM DBG Model file loaded: wizardcoder-python-34b-v1.0.Q4_K_M.gguf architecture=llama bosTokenID=1 eosTokenID=2 modelName=wizardlm_wizardcoder-python-34b-v1.0
9:31PM DBG guessDefaultsFromFile: guessed template {Chat:{{.Input -}}
<|im_start|>assistant ChatMessage:<|im_start|>{{ .RoleName }}
{{ if .FunctionCall -}}
Function call:
{{ else if eq .RoleName "tool" -}}
Function response:
{{ end -}}
{{ if .Content -}}
{{.Content }}
{{ end -}}
{{ if .FunctionCall -}}
{{toJson .FunctionCall}}
{{ end -}}<|im_end|> Completion: Edit: Functions:<|im_start|>system
You are a function calling AI model. You are provided with functions to execute. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
{{range .Functions}}
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
{{end}}
For each function call return a json object with function name and arguments
<|im_end|>
{{.Input -}}
<|im_start|>assistant UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} family=4
9:31PM INF Preloading models from /models

  Model name: deepseek-coder-33b-instruct.Q4_K_M                              



  Model name: dolphin-2.7-mixtral-8x7b.Q4_0                                   



  Model name: dreamshaper                                                     



  Model name: poppy_porpoise-v1.4-l3-8b-iq-imatrix                            



  Model name: wizardcoder-python-34b-v1.0.Q4_K_M                              


9:31PM DBG Model: deepseek-coder-33b-instruct.Q4_K_M (config: {PredictionOptions:{Model:deepseek-coder-33b-instruct.Q4_K_M.gguf Language: Translate:false N:0 TopP:0xc000d29c68 TopK:0xc000d29c70 Temperature:0xc000d29c78 Maxtokens:0xc000d29ca8 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000d29ca0 TypicalP:0xc000d29c98 Seed:0xc000d29cc0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:deepseek-coder-33b-instruct.Q4_K_M F16:0xc000d29c60 Threads:0xc000d29c38 Debug:0xc000d29cb8 Roles:map[] Embeddings:false Backend:llama-cpp TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000d29c90 MirostatTAU:0xc000d29c88 Mirostat:0xc000d29c80 NGPULayers:0xc000d29c30 MMap:0xc000d29cb8 MMlock:0xc000d29cb9 LowVRAM:0xc000d29cb9 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000d29c40 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:})
9:31PM DBG Model: dolphin-2.7-mixtral-8x7b.Q4_0 (config: {PredictionOptions:{Model:dolphin-2.7-mixtral-8x7b.Q4_0.gguf Language: Translate:false N:0 TopP:0xc001ab0d08 TopK:0xc001ab0d10 Temperature:0xc001ab0d18 Maxtokens:0xc001ab0d48 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc001ab0d40 TypicalP:0xc001ab0d38 Seed:0xc001ab0d60 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:dolphin-2.7-mixtral-8x7b.Q4_0 F16:0xc001ab0d00 Threads:0xc001ab0ce8 Debug:0xc001ab0d58 Roles:map[] Embeddings:false Backend:llama-cpp TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc001ab0d30 MirostatTAU:0xc001ab0d28 Mirostat:0xc001ab0d20 NGPULayers:0xc001ab0cd0 MMap:0xc001ab0d58 MMlock:0xc001ab0d59 LowVRAM:0xc001ab0d59 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc001ab0cd8 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:})
9:31PM DBG Model: dreamshaper (config: {PredictionOptions:{Model:DreamShaper_8_pruned.safetensors Language: Translate:false N:0 TopP:0xc001be1588 TopK:0xc001be1640 Temperature:0xc001be1648 Maxtokens:0xc001be1768 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc001be1760 TypicalP:0xc001be1708 Seed:0xc001be1830 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:dreamshaper F16:0xc001be14fa Threads:0xc001be1538 Debug:0xc001be17d8 Roles:map[] Embeddings:false Backend:diffusers TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc001be1700 MirostatTAU:0xc001be1678 Mirostat:0xc001be1670 NGPULayers:0xc001be17d0 MMap:0xc001be17d8 MMlock:0xc001be17d9 LowVRAM:0xc001be17d9 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc001be1530 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:true PipelineType:StableDiffusionPipeline SchedulerType:k_dpmpp_2m EnableParameters:negative_prompt,num_inference_steps CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:25 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:})
9:31PM DBG Model: poppy_porpoise-v1.4-l3-8b-iq-imatrix (config: {PredictionOptions:{Model:Poppy_Porpoise-1.4-L3-8B.Q4_K_M.gguf Language: Translate:false N:0 TopP:0xc001beb048 TopK:0xc001beb0d0 Temperature:0xc001beb0d8 Maxtokens:0xc001beb1d8 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc001beb1d0 TypicalP:0xc001beb188 Seed:0xc001beb2f0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:poppy_porpoise-v1.4-l3-8b-iq-imatrix F16:0xc001bead78 Threads:0xc001beafc8 Debug:0xc001beb2c8 Roles:map[] Embeddings:false Backend: TemplateConfig:{Chat:<|begin_of_text|>{{.Input }}
<|start_header_id|>assistant<|end_header_id|>
 ChatMessage:<|start_header_id|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}<|end_header_id|>

{{ if .FunctionCall -}}
Function call:
{{ else if eq .RoleName "tool" -}}
Function response:
{{ end -}}
{{ if .Content -}}
{{.Content -}}
{{ else if .FunctionCall -}}
{{ toJson .FunctionCall -}}
{{ end -}}
<|eot_id|>
 Completion:{{.Input}}
 Edit: Functions:<|start_header_id|>system<|end_header_id|>

You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
<tools>
{{range .Functions}}
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
{{end}}
</tools>
Use the following pydantic model json schema for each tool call you will make:
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Function call:
 UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc001beb180 MirostatTAU:0xc001beb128 Mirostat:0xc001beb120 NGPULayers:0xc001beb2c0 MMap:0xc001bead79 MMlock:0xc001beb2c9 LowVRAM:0xc001beb2c9 Grammar: StopWords:[<|im_end|> <dummy32000> <|eot_id|> <|end_of_text|>] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc001beacb8 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj:Llama-3-Update-2.0-mmproj-model-f16.gguf FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:})
9:31PM DBG Model: wizardcoder-python-34b-v1.0.Q4_K_M (config: {PredictionOptions:{Model:wizardcoder-python-34b-v1.0.Q4_K_M.gguf Language: Translate:false N:0 TopP:0xc001bebd40 TopK:0xc001bebd48 Temperature:0xc001bebdf0 Maxtokens:0xc001bf4090 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc001bf4018 TypicalP:0xc001bf4010 Seed:0xc001bf40e8 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:wizardcoder-python-34b-v1.0.Q4_K_M F16:0xc001bebc98 Threads:0xc001bebc90 Debug:0xc001bf40e0 Roles:map[] Embeddings:false Backend:llama-cpp TemplateConfig:{Chat:{{.Input -}}
<|im_start|>assistant ChatMessage:<|im_start|>{{ .RoleName }}
{{ if .FunctionCall -}}
Function call:
{{ else if eq .RoleName "tool" -}}
Function response:
{{ end -}}
{{ if .Content -}}
{{.Content }}
{{ end -}}
{{ if .FunctionCall -}}
{{toJson .FunctionCall}}
{{ end -}}<|im_end|> Completion: Edit: Functions:<|im_start|>system
You are a function calling AI model. You are provided with functions to execute. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
{{range .Functions}}
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
{{end}}
For each function call return a json object with function name and arguments
<|im_end|>
{{.Input -}}
<|im_start|>assistant UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc001bebe78 MirostatTAU:0xc001bebe70 Mirostat:0xc001bebdf8 NGPULayers:0xc001bebbb0 MMap:0xc001bf40e0 MMlock:0xc001bf40e1 LowVRAM:0xc001bf40e1 Grammar: StopWords:[<|im_end|> <dummy32000> </s>] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc001bebbb8 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:})
9:31PM DBG Extracting backend assets files to /tmp/localai/backend_data
9:31PM DBG processing api keys runtime update
9:31PM DBG processing external_backends.json
9:31PM DBG external backends loaded from external_backends.json
9:31PM INF core/startup process completed!
9:31PM DBG No configuration file found at /tmp/localai/upload/uploadedFiles.json
9:31PM DBG No configuration file found at /tmp/localai/config/assistants.json
9:31PM DBG No configuration file found at /tmp/localai/config/assistantsFile.json
9:31PM INF LocalAI API is listening! Please connect to the endpoint for API documentation. endpoint=http://0.0.0.0:8080
9:31PM INF Success ip=10.0.2.100 latency=2.261943ms method=GET status=200 url=/
9:31PM INF Success ip=10.0.2.100 latency="42.764µs" method=GET status=200 url=/static/assets/highlightjs.css
9:31PM INF Success ip=10.0.2.100 latency="22.697µs" method=GET status=200 url=/static/assets/highlightjs.js
9:31PM INF Success ip=10.0.2.100 latency="22.487µs" method=GET status=200 url=/static/assets/alpine.js
9:31PM INF Success ip=10.0.2.100 latency="21.983µs" method=GET status=200 url=/static/assets/marked.js
9:31PM INF Success ip=10.0.2.100 latency="23.864µs" method=GET status=200 url=/static/assets/purify.js
9:31PM INF Success ip=10.0.2.100 latency="64.227µs" method=GET status=200 url=/static/general.css
9:31PM INF Success ip=10.0.2.100 latency="33.387µs" method=GET status=200 url=/static/assets/font1.css
9:31PM INF Success ip=10.0.2.100 latency="336.664µs" method=GET status=200 url=/static/assets/font2.css
9:31PM INF Success ip=10.0.2.100 latency="21.913µs" method=GET status=200 url=/static/assets/tw-elements.css
9:31PM INF Success ip=10.0.2.100 latency="21.31µs" method=GET status=200 url=/static/assets/tailwindcss.js
9:31PM INF Success ip=10.0.2.100 latency="23.697µs" method=GET status=200 url=/static/assets/fontawesome/css/fontawesome.css
9:31PM INF Success ip=10.0.2.100 latency="20.864µs" method=GET status=200 url=/static/assets/fontawesome/css/brands.css
9:31PM INF Success ip=10.0.2.100 latency="22.636µs" method=GET status=200 url=/static/assets/fontawesome/css/solid.css
9:31PM INF Success ip=10.0.2.100 latency="32.034µs" method=GET status=200 url=/static/assets/htmx.js
9:31PM INF Success ip=10.0.2.100 latency=12.188348ms method=GET status=200 url=/static/assets/UcCO3FwrK3iLTeHuS_fvQtMwCp50KnMw2boKoduKmMEVuFuYMZg.ttf
9:31PM INF Success ip=10.0.2.100 latency="28.593µs" method=GET status=200 url=/static/assets/fontawesome/webfonts/fa-solid-900.woff2
9:31PM INF Success ip=10.0.2.100 latency="498.343µs" method=GET status=200 url=/static/assets/UcCO3FwrK3iLTeHuS_fvQtMwCp50KnMw2boKoduKmMEVuGKYMZg.ttf
9:31PM INF Success ip=10.0.2.100 latency="548.826µs" method=GET status=200 url=/static/assets/UcCO3FwrK3iLTeHuS_fvQtMwCp50KnMw2boKoduKmMEVuLyfMZg.ttf
9:31PM INF Success ip=10.0.2.100 latency="35.877µs" method=GET status=200 url=/favicon.ico
9:31PM INF Success ip=10.0.2.100 latency="632.201µs" method=GET status=200 url=/text2image/
9:31PM INF Success ip=10.0.2.100 latency="49.491µs" method=GET status=200 url=/static/assets/highlightjs.css
9:31PM INF Success ip=10.0.2.100 latency="21.713µs" method=GET status=200 url=/static/assets/highlightjs.js
9:31PM INF Success ip=10.0.2.100 latency="22.297µs" method=GET status=200 url=/static/assets/alpine.js
9:31PM INF Success ip=10.0.2.100 latency="19.237µs" method=GET status=200 url=/static/assets/marked.js
9:31PM INF Success ip=10.0.2.100 latency="13.56µs" method=GET status=200 url=/static/assets/purify.js
9:31PM INF Success ip=10.0.2.100 latency="22.357µs" method=GET status=200 url=/static/general.css
9:31PM INF Success ip=10.0.2.100 latency="33.761µs" method=GET status=200 url=/static/assets/font1.css
9:31PM INF Success ip=10.0.2.100 latency="22.86µs" method=GET status=200 url=/static/assets/font2.css
9:31PM INF Success ip=10.0.2.100 latency="20.947µs" method=GET status=200 url=/static/assets/tw-elements.css
9:31PM INF Success ip=10.0.2.100 latency="20.303µs" method=GET status=200 url=/static/assets/tailwindcss.js
9:31PM INF Success ip=10.0.2.100 latency="47.394µs" method=GET status=200 url=/static/assets/fontawesome/css/fontawesome.css
9:31PM INF Success ip=10.0.2.100 latency="16.667µs" method=GET status=200 url=/static/assets/fontawesome/css/solid.css
9:31PM INF Success ip=10.0.2.100 latency="25.96µs" method=GET status=200 url=/static/assets/fontawesome/css/brands.css
9:31PM INF Success ip=10.0.2.100 latency="23.297µs" method=GET status=200 url=/static/assets/htmx.js
9:31PM INF Success ip=10.0.2.100 latency="15.894µs" method=GET status=200 url=/static/image.js
9:31PM INF Success ip=10.0.2.100 latency="494.31µs" method=GET status=200 url=/text2image/dreamshaper
9:31PM INF Success ip=10.0.2.100 latency="51.881µs" method=GET status=200 url=/static/assets/highlightjs.css
9:31PM INF Success ip=10.0.2.100 latency="26.051µs" method=GET status=200 url=/static/assets/highlightjs.js
9:31PM INF Success ip=10.0.2.100 latency="19.664µs" method=GET status=200 url=/static/assets/alpine.js
9:31PM INF Success ip=10.0.2.100 latency="19.559µs" method=GET status=200 url=/static/assets/marked.js
9:31PM INF Success ip=10.0.2.100 latency="17.463µs" method=GET status=200 url=/static/assets/purify.js
9:31PM INF Success ip=10.0.2.100 latency="18.857µs" method=GET status=200 url=/static/general.css
9:31PM INF Success ip=10.0.2.100 latency="18.587µs" method=GET status=200 url=/static/assets/font1.css
9:31PM INF Success ip=10.0.2.100 latency="17.974µs" method=GET status=200 url=/static/assets/font2.css
9:31PM INF Success ip=10.0.2.100 latency="23.53µs" method=GET status=200 url=/static/assets/tw-elements.css
9:31PM INF Success ip=10.0.2.100 latency="18.354µs" method=GET status=200 url=/static/assets/tailwindcss.js
9:31PM INF Success ip=10.0.2.100 latency="54.517µs" method=GET status=200 url=/static/assets/fontawesome/css/fontawesome.css
9:31PM INF Success ip=10.0.2.100 latency="14.56µs" method=GET status=200 url=/static/assets/fontawesome/css/brands.css
9:31PM INF Success ip=10.0.2.100 latency="14.493µs" method=GET status=200 url=/static/assets/fontawesome/css/solid.css
9:31PM INF Success ip=10.0.2.100 latency="15.149µs" method=GET status=200 url=/static/assets/htmx.js
9:31PM INF Success ip=10.0.2.100 latency="14.987µs" method=GET status=200 url=/static/image.js
9:31PM DBG Request received: {"model":"dreamshaper","language":"","translate":false,"n":1,"top_p":null,"top_k":null,"temperature":null,"max_tokens":null,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"repeat_last_n":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":null,"typical_p":null,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","size":"512x512","prompt":"Supergoat","instruction":"","input":null,"stop":null,"messages":null,"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"grammar_json_name":null,"backend":"","model_base_name":""}
9:31PM DBG Loading model: dreamshaper
9:31PM DBG guessDefaultsFromFile: not a GGUF file
9:31PM DBG Parameter Config: &{PredictionOptions:{Model:DreamShaper_8_pruned.safetensors Language: Translate:false N:0 TopP:0xc001be1588 TopK:0xc001be1640 Temperature:0xc001be1648 Maxtokens:0xc001be1768 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc001be1760 TypicalP:0xc001be1708 Seed:0xc001be1830 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:dreamshaper F16:0xc001be14fa Threads:0xc001be1538 Debug:0xc000569e50 Roles:map[] Embeddings:false Backend:diffusers TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} PromptStrings:[Supergoat] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc001be1700 MirostatTAU:0xc001be1678 Mirostat:0xc001be1670 NGPULayers:0xc001be17d0 MMap:0xc001be17d8 MMlock:0xc001be17d9 LowVRAM:0xc001be17d9 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc001be1530 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:true PipelineType:StableDiffusionPipeline SchedulerType:k_dpmpp_2m EnableParameters:negative_prompt,num_inference_steps CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:25 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:}
9:31PM INF Loading model 'DreamShaper_8_pruned.safetensors' with backend diffusers
9:31PM DBG Loading model in memory from file: /models/DreamShaper_8_pruned.safetensors
9:31PM DBG Loading Model DreamShaper_8_pruned.safetensors with gRPC (file: /models/DreamShaper_8_pruned.safetensors) (backend: diffusers): {backendString:diffusers model:DreamShaper_8_pruned.safetensors threads:10 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0005c26c8 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh openvoice:/build/backend/python/openvoice/run.sh parler-tts:/build/backend/python/parler-tts/run.sh petals:/build/backend/python/petals/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:31PM DBG Loading external backend: /build/backend/python/diffusers/run.sh
9:31PM DBG Loading GRPC Process: /build/backend/python/diffusers/run.sh
9:31PM DBG GRPC Service for DreamShaper_8_pruned.safetensors will be running at: '127.0.0.1:34109'
9:31PM DBG GRPC Service state dir: /tmp/go-processmanager1559952802
9:31PM DBG GRPC Service Started
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stdout Initializing libbackend for build
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stdout virtualenv activated
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stdout activated virtualenv has been ensured
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr /build/backend/python/diffusers/backend_pb2_grpc.py:21: RuntimeWarning: The grpc package installed is at version 1.64.0, but the generated code in backend_pb2_grpc.py depends on grpcio>=1.64.1. Please upgrade your grpc module to grpcio>=1.64.1 or downgrade your generated code using grpcio-tools<=1.64.0. This warning will become an error in 1.65.0, scheduled for release on June 25, 2024.
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   warnings.warn(
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr /build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/utils/hub.py:127: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   warnings.warn(
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr Traceback (most recent call last):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1560, in _get_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     return importlib.import_module("." + module_name, self.__name__)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     return _bootstrap._gcd_import(name[level:], package, level)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py", line 27, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from ...image_processing_utils import BaseImageProcessor, ImageProcessingMixin
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 21, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from .image_transforms import center_crop, normalize, rescale
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/image_transforms.py", line 22, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from .image_utils import (
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/image_utils.py", line 58, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from torchvision.transforms import InterpolationMode
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/torchvision/__init__.py", line 6, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 164, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     def meta_nms(dets, scores, iou_threshold):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/torch/library.py", line 467, in inner
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     handle = entry.abstract_impl.register(func_to_register, source)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/torch/_library/abstract_impl.py", line 30, in register
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr RuntimeError: operator torchvision::nms does not exist
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr 
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr The above exception was the direct cause of the following exception:
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr 
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr Traceback (most recent call last):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 808, in _get_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     return importlib.import_module("." + module_name, self.__name__)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     return _bootstrap._gcd_import(name[level:], package, level)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/loaders/single_file.py", line 24, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from .single_file_utils import (
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/loaders/single_file_utils.py", line 50, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from transformers import AutoImageProcessor
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1551, in __getattr__
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     value = getattr(module, name)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1550, in __getattr__
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     module = self._get_module(self._class_to_module[name])
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1562, in _get_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     raise RuntimeError(
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr RuntimeError: Failed to import transformers.models.auto.image_processing_auto because of the following error (look up to see its traceback):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr operator torchvision::nms does not exist
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr 
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr The above exception was the direct cause of the following exception:
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr 
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr Traceback (most recent call last):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 808, in _get_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     return importlib.import_module("." + module_name, self.__name__)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     return _bootstrap._gcd_import(name[level:], package, level)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py", line 27, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from ...loaders import FromSingleFileMixin, SD3LoraLoaderMixin
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 798, in __getattr__
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     module = self._get_module(self._class_to_module[name])
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 810, in _get_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     raise RuntimeError(
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr RuntimeError: Failed to import diffusers.loaders.single_file because of the following error (look up to see its traceback):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr Failed to import transformers.models.auto.image_processing_auto because of the following error (look up to see its traceback):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr operator torchvision::nms does not exist
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr 
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr The above exception was the direct cause of the following exception:
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr 
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr Traceback (most recent call last):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/backend.py", line 20, in <module>
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     from diffusers import StableDiffusion3Pipeline, StableDiffusionXLPipeline, StableDiffusionDepth2ImgPipeline, DPMSolverMultistepScheduler, StableDiffusionPipeline, DiffusionPipeline, EulerAncestralDiscreteScheduler
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 799, in __getattr__
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     value = getattr(module, name)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 799, in __getattr__
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     value = getattr(module, name)
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 798, in __getattr__
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     module = self._get_module(self._class_to_module[name])
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr   File "/build/backend/python/diffusers/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 810, in _get_module
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr     raise RuntimeError(
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr RuntimeError: Failed to import diffusers.pipelines.stable_diffusion_3.pipeline_stable_diffusion_3 because of the following error (look up to see its traceback):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr Failed to import diffusers.loaders.single_file because of the following error (look up to see its traceback):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr Failed to import transformers.models.auto.image_processing_auto because of the following error (look up to see its traceback):
9:31PM DBG GRPC(DreamShaper_8_pruned.safetensors-127.0.0.1:34109): stderr operator torchvision::nms does not exist
9:32PM ERR failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:34109: connect: connection refused\""
9:32PM DBG GRPC Service NOT ready
9:32PM ERR Server error error="grpc service not ready" ip=10.0.2.100 latency=40.05162577s method=POST status=500 url=/v1/images/generations
@jbash jbash added bug Something isn't working unconfirmed labels Jul 6, 2024
@mudler
Copy link
Owner

mudler commented Jul 9, 2024

some more context in #1592 as this have been already discussed

@cryptk cryptk mentioned this issue Jul 15, 2024
Merged
1 task
mudler added a commit that referenced this issue Aug 7, 2024
@mudler
fix(python): move accelerate and GPU-specific libs to build-type
db003ef 
Some of the dependencies in `requirements.txt`, even if generic, pulls
down the line CUDA libraries.

This changes moves mostly all GPU-specific libs to the build-type, and
tries a safer approach. In `requirements.txt` now are listed only
"first-level" dependencies, for instance, grpc, but libs-dependencies
are moved down to the respective build-type `requirements.txt` to avoid
any mixin.

This should fix #2737 and #1592.

Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler mudler mentioned this issue Aug 7, 2024
Merged
mudler added a commit that referenced this issue Aug 7, 2024
@mudler
fix(python): move accelerate and GPU-specific libs to build-type
84a7fb2 
Some of the dependencies in `requirements.txt`, even if generic, pulls
down the line CUDA libraries.

This changes moves mostly all GPU-specific libs to the build-type, and
tries a safer approach. In `requirements.txt` now are listed only
"first-level" dependencies, for instance, grpc, but libs-dependencies
are moved down to the respective build-type `requirements.txt` to avoid
any mixin.

This should fix #2737 and #1592.

Signed-off-by: Ettore Di Giacinto <[email protected]>
@mudler mudler closed this as completed in 61b5602 Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working unconfirmed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(python): move accelerate and GPU-specific libs to build-type mudler/LocalAI
2 participants
@mudler @jbash