Recommended latest image is too large for most clusters

[erikkrieg]

While trying out the helm chart I found that the recommended image (from the example values file) is far too large to pull on our cluster. Pulling the image locally I'm seeing that it is just shy of 70.0 Gi.

I think that if the docs make some suggestions for different image tags it would make installing the chart easier.

I'm going forward with v2.0.0-ffmpeg-core.

[ronaldpetty]

Agree. It would also just be good to have a small default (does minimum) just to make sure everything can be installed. Maybe put a warning on size (not able to see it on quay site).

quay.io/go-skynet/local-ai latest e2677207ccd4 16 hours ago 49GB

This download was k8s on EC2, thats a long time.

Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  48m    default-scheduler  Successfully assigned default/release-name-local-ai-5d779484c8-hg722 to k3d-deploykf-server-0
  Normal  Pulling    48m    kubelet            Pulling image "busybox"
  Normal  Pulled     48m    kubelet            Successfully pulled image "busybox" in 837.25772ms (837.272028ms including waiting)
  Normal  Created    48m    kubelet            Created container download-model
  Normal  Started    48m    kubelet            Started container download-model
  Normal  Pulling    46m    kubelet            Pulling image "quay.io/go-skynet/local-ai:latest"
  Normal  Pulled     2m13s  kubelet            Successfully pulled image "quay.io/go-skynet/local-ai:latest" in 43m51.539622447s (43m51.539631847s including waiting)
  Normal  Created    2m13s  kubelet            Created container release-name-local-ai
  Normal  Started    2m12s  kubelet            Started container release-name-local-ai

[ssurovich]

Great point - On a standard K8s node, this is going to really fill up the overlay and when that node hits >=80% you will start getting evictions. Most orgs allocate 100-200GB on the overlay (from what I have seen) - so this taking up 1/4 of the storage may cause issues to other workloads. One thing you could do is to add a node with a taint and then use a toleration to limit that node to running just localai (and other resources for it). I know that can be easier said than done...

[mudler]

docs about the different image tags is here: https://localai.io/basics/getting_started/#container-images - I do agree generally, however note that after few rounds of enhancements the images now are much smaller, compressed are in order of ~17GB:

[ssurovich]

Nice progresss on optimizing - I do get why they are larger images in general - and using a PVC for the models is really the way to help any stress on the conainer host filesystem.