Flux GGUF (Replicate, Document, Add to Model)

[sfxworks]

LocalAI version:
quay.io/go-skynet/local-ai:v2.20.1-aio-gpu-hipblas

Environment, CPU architecture, OS, and Version:
k8s

Describe the bug
I'm trying to load up the gguf version of flux.dev from https://huggingface.co/city96/FLUX.1-dev-gguf, which, according to some chats, uses stable-diffusion.cpp.

So I tried making the yaml for it, having it use the stablediffusion backend,

backend: stablediffusion
name: flux.1-dev-gguf
parameters:
  model: flux1-dev-F16.gguf

and I get this output

7:57PM DBG GRPC(flux1-dev-F16.gguf-127.0.0.1:36783): stderr /tmp/localai/backend_data/backend-assets/grpc/stablediffusion: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory

Specifically, I tried countering this by going into the container and installing lipomp, but the error still persists

I've tried some other variants, like with diffusers, specifying the flux pipeline, only to fail.

To Reproduce

Use the yaml above on hiplabs,
go to the image generation page
notice the failure in logs

Expected behavior

Logs

7:57PM DBG Request received: {"model":"flux.1-dev-gguf","language":"","translate":false,"n":1,"top_p":null,"top_k":null,"temperature":null,"max_tokens":null,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"repeat_last_n":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":null,"typical_p":null,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","size":"512x512","prompt":"black hole in the sky with a beach","instruction":"","input":null,"stop":null,"messages":null,"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"backend":"","model_base_name":""}
7:57PM DBG Loading model: flux.1-dev-gguf
7:57PM DBG Model file loaded: flux1-dev-F16.gguf architecture=flux bosTokenID=-1 eosTokenID=-1 modelName=
7:57PM DBG guessDefaultsFromFile: family not identified
7:57PM DBG Parameter Config: &{PredictionOptions:{Model:flux1-dev-F16.gguf Language: Translate:false N:0 TopP:0xc000aa3ee8 TopK:0xc000aa3ef0 Temperature:0xc000aa3ef8 Maxtokens:0xc000aa3f28 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000aa3f20 TypicalP:0xc000aa3f18 Seed:0xc000aa3f40 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:flux.1-dev-gguf F16:0xc000aa3ee0 Threads:0xc000aa3ed8 Debug:0xc0026d1140 Roles:map[] Embeddings:0xc000aa3f39 Backend:stablediffusion TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil>} PromptStrings:[black hole in the sky with a beach] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder: SchemaType:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionNameKey: FunctionArgumentsKey:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000aa3f10 MirostatTAU:0xc000aa3f08 Mirostat:0xc000aa3f00 NGPULayers:0xc000aa3f30 MMap:0xc000aa3f38 MMlock:0xc000aa3f39 LowVRAM:0xc000aa3f39 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000aa3ed0 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:}
7:57PM INF Loading model 'flux1-dev-F16.gguf' with backend stablediffusion
7:57PM DBG Stopping all backends except 'flux1-dev-F16.gguf'
7:57PM DBG Loading model in memory from file: /build/models/flux1-dev-F16.gguf
7:57PM DBG Loading Model flux1-dev-F16.gguf with gRPC (file: /build/models/flux1-dev-F16.gguf) (backend: stablediffusion): {backendString:stablediffusion model:flux1-dev-F16.gguf threads:3 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0026de908 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh openvoice:/build/backend/python/openvoice/run.sh parler-tts:/build/backend/python/parler-tts/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:true parallelRequests:true}
7:57PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion
7:57PM DBG GRPC Service for flux1-dev-F16.gguf will be running at: '127.0.0.1:36783'
7:57PM DBG GRPC Service state dir: /tmp/go-processmanager889091854
7:57PM DBG GRPC Service Started
7:57PM DBG GRPC(flux1-dev-F16.gguf-127.0.0.1:36783): stderr /tmp/localai/backend_data/backend-assets/grpc/stablediffusion: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory