Can't start LocalAI (with REBUILD) on Xeon X5570 - Unwanted AVX dependency?

[chris-hatton]

LocalAI version:
Using Docker image:
localai/localai:latest-aio-gpu-hipblas

Environment, CPU architecture, OS, and Version:

• Ubuntu 22.04
• Xeon X5570 Specs (No AVX at all)
• GPU Radeon 6700XT (gfx1031 but I've seen this to be gfx1030 compatible on a previous ROCm test)
• installed ROCm driver per AMD instructions (version 6.2.60200)

Describe the bug

• I can't get LocalAI to start up, using the following Docker Compose stack file (inspired by ROCm Acceleration instructions):

version: '3.9'
services:
  api:
    image: localai/localai:latest-aio-gpu-hipblas
    restart: no
    ports:
      - 8082:8080
    volumes:
      - /raid/AI/Models:/build/models:cached
    environment:
      - "DEBUG=true"
      - "REBUILD=true"
      - "CMAKE_ARGS=-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_AVX=OFF -DGGML_FMA=OFF -DLLAMA_F16C=OFF -DLLAMA_AVX512=OFF -DLLAMA_AVX2=OFF -DLLAMA_AVX=OFF -DLLAMA_FMA=OFF"
      - "BUILD_TYPE=hipblas"
      - "GPU_TARGETS=gfx1030"
      - "BUILD_PARALLELISM=16"
    - devices:
      - /dev/dri
      - /dev/kfd
networks:
  apollonet:
    external: true
    name: apollonet

To Reproduce

• Try to start the Docker container with the above configuration.
• The Docker container goes through a lengthy build process for several hours
• It looks like the rebuild completes(?) or gets a significant way through, then the Container quite with error code 132
• Exit 132 is SIGILL (132-128=4) which means illegal instruction. I suspect AVX is still being used by something. My CPU doesn't support AVX at all, but I am expecting to run all inference on the GPU so achieving acceleration on CPU should be unimportant(?)

Expected behavior
LocalAI starts and serves an interface on the exposed host port 8082, ready to run inference on the AMD card.

Logs
The last lines of the startup/rebuild log, before the exit, are:

cp llama.cpp/build/bin/grpc-server .
make[2]: Leaving directory '/build/backend/cpp/llama-fallback'
make[1]: Entering directory '/build'
/usr/bin/upx backend/cpp/llama-fallback/grpc-server
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  85962576 ->  17753404   20.65%   linux/amd64   grpc-server
Packed 1 file.
make[1]: Leaving directory '/build'
cp -rfv backend/cpp/llama-fallback/grpc-server backend-assets/grpc/llama-cpp-fallback
'backend/cpp/llama-fallback/grpc-server' -> 'backend-assets/grpc/llama-cpp-fallback'
I local-ai build info:
I BUILD_TYPE: hipblas
I GO_TAGS: 
I LD_FLAGS: -s -w -X "github.com/mudler/LocalAI/internal.Version=v2.19.3" -X "github.com/mudler/LocalAI/internal.Commit=86f8d5b50acd8fe88af4f537be0d42472772b928"
I UPX: /usr/bin/upx
CGO_LDFLAGS="-O3 --rtlib=compiler-rt -unwindlib=libgcc -lhipblas -lrocblas --hip-link -L/opt/rocm/lib/llvm/lib" go build -ldflags "-s -w -X "github.com/mudler/LocalAI/internal.Version=v2.19.3" -X "github.com/mudler/LocalAI/internal.Commit=86f8d5b50acd8fe88af4f537be0d42472772b928"" -tags "" -o local-ai ./

[mudler]

@chris-hatton looks something behind the scenes is happening - care to share dmesg when that happens?

Also, why using REBUILD? The image should have already binaries for the CPU flagset variants, including non-AVX - did you tried to disable it? your GPU seems also covered by defaults GPU_TARGETS