Ollama models with Frigate 0.15+

[DrSpaldo]

I wanted to start a discussion for those that are currently using the 0.15 dev builds of Frigate and have tried Ollama models, finding the good & bad of each. This will become more active I hope once the 0.15 version and beyond are released officially. I went down the rabbit hole a little while ago with Ollama when the dev builds included access, however, had difficulty installing it and getting the models running correctly...

I followed this guide a few days ago; which was alright but I still had difficulty getting the Ollama server to connect with the webui. Then I tried just using the method from Open WebUI in particular the bundled version This worked quite well.

This lead me to start trying different models, ideally one that I can use for Home Assistant as well as for Frigate. The llava-llama3 model was "alright" for images, but terrible for chat responses/Home Assistant. The model listed in the other guide from @maxi1134 finalend/llama-3.1-storm:8b-q8_0 was really good for the chat, almost like the current ChatGPT, but doesn't do any images at all.

So.... I was wondering what others using Ollama were experiencing and what models they had been testing out? Hopefully we can use this as a discussion for exploring locally hosted AI Ollama.

[maxi1134]

I currently use llava:13b-v1.6-vicuna-q8_0 for my image recognition needs.
I run it in aside from the LLama 3.1.

[DrSpaldo]

I currently use llava:13b-v1.6-vicuna-q8_0 for my image recognition needs. I run it in aside from the LLama 3.1.

Thanks for the reply @maxi1134, so you don't run one model to do both? you run two different models? I was just worried about ram usage, etc.

I was just looking at x/llama3.2-vision but haven't got it working yet

[maxi1134]

Thanks for the reply @maxi1134, so you don't run one model to do both? you run two different models? I was just worried about ram usage, etc.

I run both, as well as a Whisper model, for my plex subtitle generation.

I have a 3090, so VRAM is a somewhat softer limit compared to something with only 8 or 12 GB of VRAM.

[NickM-27]

This may not be helpful but I can say that I personally have not found much use out of genai in Frigate. Especially after moving to the Jina models, it can read text (making text on vans, shirts, etc. searchable) and is very specific. Perhaps you may find it similar and it would be easier to just get a genai chat model for HA and not worry about Frigate.

[NickM-27]

I am not using any genai in Frigate, the semantic search embedding model meets every need that I would have

[DrSpaldo]

I agree with Nick on this. I simply pipe my Frigate to my LLava through HA.

@maxi1134 How are you doing that?

[DrSpaldo]

I am not using any genai in Frigate, the semantic search embedding model meets every need that I would have

Does it work for number plates?

[NickM-27]

you can search number plates, you won't be able to extract the text, but we are already working on ALPR being built in to Frigate for the next version (0.16)

[DrSpaldo]

you can search number plates, you won't be able to extract the text, but we are already working on ALPR being built in to Frigate for the next version (0.16)

@NickM-27 nice, that would be a great addition

[hawkeye217]

After a couple of beta releases of 0.15, we plan to post a survey asking what use cases users have found with the genai feature. Like Nick, I thought it was pretty nice at first, but once we integrated the Jina CLIP model, using an "image caption" approach for searches has proved to meet all my needs and exceed my expectations.

[DrSpaldo]

After a couple of beta releases of 0.15, we plan to post a survey asking what use cases users have found with the genai feature. Like Nick, I thought it was pretty nice at first, but once we integrated the Jina CLIP model, using an "image caption" approach for searches has proved to meet all my needs and exceed my expectations.

I haven’t looked this at all. So if I just disable my genai / open ai, it will automatically use the Jina method and add text to the entries?

[hawkeye217]

No, text/descriptions are not generated. Have a look at the beta docs: https://deploy-preview-13787--frigate-docs.netlify.app/configuration/semantic_search#jina-ai-clip

[DrSpaldo]

No, text/descriptions are not generated. Have a look at the beta docs: https://deploy-preview-13787--frigate-docs.netlify.app/configuration/semantic_search#jina-ai-clip

Alright, so Jina doesn’t add the text box but adds info/tags to the image/event itself and you can search that?

[NickM-27]

Yes, it creates an internal set of text for each image. You search and then the most relevant results show up. For example,

[dz4news]

I have not been using dev 0.15 for very long.. but I tend to agree that genai is fun at first, but I am not seeing much use for it. The semantic search is doing everything I need.

[divemasterjm]

fully agree with Nick, i've been testing different models, in my case no gpu wich means unusable, only with llava-phi3 model i was able to get inference less than 50s, i will stay with jinja and waiting for face and alpr on next version witch is very important for me, hope to have soon some dev to try.

[jamesahendry]

I've spun up the OpenWebUI and installed the llava:7b model. Am I to assume you just connect on port 3000 in the Frigate config or 11434? I've gone to the Explorer and tried to re-generate a description of an image and get nothing. I would have throught the CPU would go throught he roof on the Ollama instance as I'm not running a GPU - similar to when I query the model on Open WebUI. Any pointers appreciated.

genai:
enabled: true
provider: ollama
base_url: http://10.0.45.44:3000
model: llava:7b

[hawkeye217]

You'll need the API port, not the web port. For my installation of Ollama on MacOS, that is 11434.

[jamesahendry]

cheers @hawkeye217 - As you can see when I got to 11434 I get a response from the API through the browser. I'll give it another go. I take it that it should regenerate a description?

[leccelecce]

I was finding on a 10th Gen Intel that Ollama couldn't generate anything in under 60 seconds, so is killed by an HTTP timeout in the client library before any response is received. The UI doesn't show the error. There doesn't seem be appetite to work around the timeout, the answer is to use a GPU.

[SlavikCA]

I tried to use GenAI with Ollama, but getting a lot of errors about parallel requests:

[GIN] 2024/10/30 - 17:44:01 | 200 |  7.501288581s |      172.20.0.1 | POST     "/api/generate"
time=2024-10-30T17:47:04.841Z level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet"
encode_image_with_clip: image embedding created: 576 tokens

encode_image_with_clip: image encoded in   138.32 ms by CLIP (    0.24 ms per image patch)
encode_image_with_clip: image embedding created: 576 tokens

encode_image_with_clip: image encoded in   163.75 ms by CLIP (    0.28 ms per image patch)
encode_image_with_clip: image embedding created: 576 tokens

encode_image_with_clip: image encoded in   161.60 ms by CLIP (    0.28 ms per image patch)
time=2024-10-30T17:47:51.640Z level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet"
[GIN] 2024/10/30 - 17:48:04 | 500 |          1m0s |      172.20.0.1 | POST     "/api/generate"
[GIN] 2024/10/30 - 17:48:05 | 500 |  13.60499169s |      172.20.0.1 | POST     "/api/generate"
time=2024-10-30T17:48:10.807Z level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet"
[GIN] 2024/10/30 - 17:48:11 | 500 |  500.831765ms |      172.20.0.1 | POST     "/api/generate"
time=2024-10-30T17:48:59.225Z level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet"
[GIN] 2024/10/30 - 17:48:59 | 500 |  507.278763ms |      172.20.0.1 | POST     "/api/generate"
time=2024-10-30T17:49:56.113Z level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet"
[GIN] 2024/10/30 - 17:49:56 | 500 |   519.94347ms |      172.20.0.1 | POST     "/api/generate"
time=2024-10-30T17:54:39.925Z level=WARN source=sched.go:137 msg="multimodal models don't support parallel requests yet"
[GIN] 2024/10/30 - 17:54:40 | 500 |  487.067531ms |      172.20.0.1 | POST     "/api/generate"

Both Frigate and Ollama running on the host with nVidia Quadro P5000 with 16GB VRAM

What's strange is that looking at nvtop, GPU usage for ollama is low (few percent), but CPU usage is 100%. May be that's normal for this scenario.

Still can't see anything generated.

[SlavikCA]

Looking at the Frigate docs, it links to Ollama docs here:
https://github.com/ollama/ollama/blob/main/docs/faq.md#how-does-ollama-handle-concurrent-requests

And when I tried, I found that's not applicable for multimodal models. Ollama supports concurrent requests to normal LLM models, but not to the multimodal.

See here:

• OLLAMA_NUM_PARALLEL and multi-modal models lead to failed processing images error ollama/ollama#4165
• https://github.com/ollama/ollama/blob/b754f5a6a36d6f3068e938af525d8b056c7f9bce/server/sched.go#L133

My config:

genai:
  enabled: true
  provider: ollama
  base_url: http://192.168.0.88:11434/
  model: llava:7b

So, practically that means, that Frigate doesn't work with Ollama, because Ollama almost immediately gets to deadlock state with many requests

[hawkeye217]

If you're trying to run generative AI on the same object from a couple of different cameras at once, then yes, it may not work well for you as the multimodal model is your limitation. Frigate just passes requests to the Ollama API.

In testing and development, I ran an Ollama server on my M1 Mac and only had it processing one camera. I didn't notice any major issues.

However, I've since disabled GenAI as I don't find it useful for my needs. The Semantic Search models cover everything I personally need out of Frigate.

Be sure to read the tip writeup on how the Semantic Search works - perhaps it will help with your use case. #14654

[SlavikCA]

Thank you, that's great writeup.

Since Ollama doesn't support parallel requests for multimodal models, I think it would be great, if Frigate can be configured to NOT submit new request to Ollama until any existing request is in progress.

[hawkeye217]

That could potentially solve one issue, but would cause another one where users would then be asking "why did it generate a description for this one tracked object but not for this other one?" There would be no way to know that from the UI or from the logs (at least easily).

We're continuing to evaluate the usefulness of genai for Frigate 0.15, and we'll be posting a survey later in the beta period to see how users are taking advantage of the feature. If there are compelling use cases that we haven't considered, it may make sense to enhance the Ollama pipeline in Frigate further.

[hawkeye217]

FYI, we've updated the Ollama genai docs with some recommendations on setting environment variables for Ollama to better handle requests from Frigate. Give it a test and let us know if it's working better for you.

[bagobones]

I am trying to get genai running but I am not seeing ANYTHING in frigate so maybe I should validate my config

I have a global config item of the following by default I have it set to enabled false so only select cameras have it enabled:

genai:
  enabled: false
  provider: ollama
  base_url: http://[myinternal host]:11434
  model: llava:7b

Then on my doorbell I have it set to true

cameras:
  doorbell:
    ui:
      order: 0
    genai:
      enabled: true
      objects:
        - person
      required_zones:
        - front_porch
        - front_walk

I have tried triggering events by waking past the doorbell but I don't see anything gen AI related in the logs at all and no activity on my ollama host.

I have open web UI installed on the same host as frigate and I as able to test the connection from it to the ollama host (my gaming PC with GPU) and that works, I also made sure to run ollama with the model persisting to avoid the timeout and unloading.

I see lots of comments from the devs about the usefulness so here is my use case.

• I want to generate nice short verbal descriptions of people at the door, so I know if it is just someone dropping off a package or if I have someone standing and waiting that for some reason didn't ring the doorbell
• I want to generate alerts in home assistant if possible, like seeing the garbage truck so I know when to bring in the bins.

[bagobones]

FYI with genai enabled globally but disabled on a per camera, the generate button appears and a message about it taking a while will pop up, however if it is disabled on the camera only a log message is generated, no UI feedback that it failed.

The appearance of the button should likely be dictated by the camera not by the global enable setting.

[hawkeye217]

The appearance of the button should likely be dictated by the camera not by the global enable setting.

Yes, this is what I will fix in an upcoming PR.

[bagobones]

Yep, it's right at the top.

Ya I clearly missed that but had it enabled anyway..

I would suggest breaking it out in the yellow warning or info bubble style

[bagobones]

Question.. We have camera and object prompts, would it be possible to have ZONE prompts to help the AI have context?

IE I might have a different prompt for the road / sidewalk than for my front porch? Not sure how that would work for objects crossing zones in an event however.

[hawkeye217]

FYI with genai enabled globally but disabled on a per camera, the generate button appears and a message about it taking a while will pop up, however if it is disabled on the camera only a log message is generated, no UI feedback that it failed.

The appearance of the button should likely be dictated by the camera not by the global enable setting.

Improved in #14850

[bagobones]

If I wanted to generate a notification that includes the AI description once it is complete is there a specific mqtt topic / message type I would be looking for?

I presume an event / alert fires first then the description is generated after?

I am hoping I can do something like the following.

1. When a person is detected in my porch zone I will fire an alert as I do already "Person detected at the front door"
2. When the description comes in "A tall male holding an amazon package" etc

[hawkeye217]

You can watch the undocumented /events_update topic for the event_id and the description.

[bagobones]

You can watch the undocumented /events_update topic for the event_id and the description.

Thanks I will take a look at that

Future alerts I am thinking of are around "bear is eating from your compost bins" or "racoons are trying to open your bins" LOL

[Salvora]

Just wanted to ask, can you perform a face recognition with semantic or genai after frigate detects a person?

[NickM-27]

No, face recognition will be supported separately. GenAI does not know who the people on your camera are

[Salvora]

oh, that is good to know. Is the face recognition going to be integrated into frigate totally? Or with some additional modules? Is it on the roadmap of 0.15 or the post 0.15 versions of frigate?

[NickM-27]

Inside of Frigate, not 0.15

[leccelecce]

@NickM-27

I have been playing around with Ollama, in particular playing around with different prompts to give more context e.g. explaining where the camera is (above front door facing the street in a residential neighbourhood), asking it to describe more elements of what it sees and in a more structured format.

One point I've noticed is that when restarting Frigate so that prompt changes can take effect, then going back to a particular event and re-generating to get a before/after, is that the response suggests that only one image has been used, even when specifying using thumbnails.

Looking at this code, am I correct in thinking that self.tracked_events[event_id]] is in-memory and not persisted, so after any restart will be empty, and so regenerating will just fall back to a single image from base64.b64decode(event.thumbnail)?

frigate/frigate/embeddings/maintainer.py

Line 347 in 7fdf42a

[thumbnail for data in self.tracked_events[event_id]]

This makes it slightly more difficult to experiment by regenerating, but short of storing a full collection of thumbnails not sure it's easily improved upon?

[NickM-27]

yes that is correct

[bagobones]

Is there a debug logging specific to the gen AI that can be turned on?

I am not seeing it actually add descriptions to most events.. I am able to use my config via Openweb UI just fine so I suspect something is up in frigate.

Over all I am also contemplating abandoning the frigate config due to it being object level and concurrency being a potential cause of issues since I mainly want to use it for notifications I am considering just using the HA LLM plugin and attaching it to my automations there.

[bagobones]

it used to do this, the problem is that if the model isn't downloaded yet then it would lead to a timeout error because it took too long to download the model.

Hence why I suggest a parameter. "OLLAMA_KEEP_ALIVE=-1" on the ollama side paired with an Init_model_on_startup=true from frigate should avoid request failures.

[NickM-27]

that doesn't have anything to do with the timeout when downloading a model for the first time

[bagobones]

that doesn't have anything to do with the timeout when downloading a model for the first time

Ok then

A. call the ollama API to list models and if the one the user has specified doesn't exist have frigate throw an error.

B. call the ollama API to list models , if the model isn't listed, call Pull a Model, then send the blank request.

[hawkeye217]

We already do A and the docs already state that the user must download the model first.

[bagobones]

We already do A and the docs already state that the user must download the model first.

ok then I still stick with my suggestion of sending a blank request to warm it up (IF it exists)

[bagobones]

Prompts and models discussion.

I have been experimenting with the models available in the ollama vison catalog and I think I am really down to only llava-llama3 and llama3.2-vision working the best at the moment, with me leaning toward llava-llama3. This is mostly because they seem to give better context to images, fewer made up details or wrong details. They also seem to understand prompt better.

I have been testing with snapshots as those are the only consistent way to test prompts outside frigate since not all models support image sequences and frigate doesn't persist them.

I am largely focused on getting descriptions that are short and useful for text to speech announcements to enhance notifications but I am not quite there yet due to model inconsistency I am still getting some novels written about the image.

These prompts both assume a snapshot where the image is not cropped in on the subject, I may go back to trying clips but that only works with some models.

person: You are reviewing images from a security camera. Your task is to give a short but complete description of the person or persons closest to the camera, without describing the surroundings. Your response will be announced so it needs to be short and clear. If the image appears to be taken at night do not describe the color or brightness. Do not describe the image or when you think it was taken. Do not describe where the person is. If the person is holding an item that you can clearly see in detail, provide a short visual description.

Example output:

"The person closest to the camera appears to be wearing a light-colored puffer jacket and pants with a hood pulled over their head. The image shows a blurry face, making it difficult to discern any facial features or details."

"A man wearing a white hoodie and sweatpants is walking away from the camera in the center of the frame. He appears to be in motion."

This one is good but too long:

"The individual closest to the camera appears to be a middle-aged woman with long gray hair and glasses. She is wearing a dark jacket with white trim on the zipper. The woman has a necklace featuring what appears to be a small white stone or pearl.

Upon closer inspection, it seems that she may also be holding an object in her right hand; however, due to the angle of the image, this cannot be determined for certain."

This one included the time stamp from the camera which I would like to filter out:

"The main object closest to the camera is a woman with brown hair wearing a red shirt and blue jeans. She appears to be standing near a wall. The image also contains some text at the top left corner which reads "11/26/2018 08:03:29 PM Fri"."

car: Your task is to give a short but complete physical description of the car closest to the camera. Your response will be announced so it needs to be short and clear. Do not mention anything about the location of the car or its surroundings.

My trigger zone is intentionally a ways up the driveway to ensure the car is in clear view before triggering.

Correct but too simple:

"The car closest to the camera is silver with a black tire."

Close but not quite rigth

"The car in the image is a small, red hatchback with a white license plate. It has black handle grips and tinted windows. The car appears to be in good condition and well-maintained."

Probably one of the most accurate but also a bit long:

"The car in the image is white with black stripes on the hood. It has a silver grille and headlights that are turned on. The car also features a large spoiler at the back, which adds to its sporty design. The car's color contrasts sharply with the gray sidewalk it is parked on."

[leccelecce]

I've tried your same prompt for person.

The things which surprises me is that if I regenerate a description twenty times, I get (roughly) 15 similar ones, 1-2 wordier ones, and 3 complete hallucinations.

I don't know what's causing the inconsistency but it's negating the usefulness right now.

[bagobones]

If you are using snapshots and the same model they should be consistent for the EXACT same snapshot, but if they are just similar I have seen the same thing. Which is why I posted what I had as I think it still needs work and I am interested in if people are getting better results from different models.

I have also be considering just extracting a capped number of characters for my HA announcements to avoid the wordy responses.

I played with sample snapshots IN Open Web UI a fair bit to get it to this point.. I think with some more work we can get a "more" consistent prompt. A lot of the "DO NOT" in my prompt is to suppress extra info it likes to randomly provide.

[leccelecce]

I've got a weird issue, I can't get the genai or embeddings code to generate any debug logs.

I can configure other packages like frigate.comms.mqtt or frigate.recordings.maintainer to debug, but when I do the same for the genai code I never see any debug messages.

Am I going mad? I can't see any obvious reason the embeddings/genai packages wouldn't work.

logger:
  default: info
  logs:
    frigate.embeddings.maintainer: debug
    frigate.genai: debug
    frigate.genai.ollama: debug
    frigate.comms.mqtt: debug
    frigate.record.cleanup: debug

[NickM-27]

I have created a survey that we want to use to get feedback from users that have been using the GenAI feature for at least 1 week. The feedback will be very helpful in understanding users use cases and how the feature can be improved. The survey is anonymous:

https://docs.google.com/forms/d/e/1FAIpQLSehVvK6oVKzikuKzwlIGz2eiaUwhYQq24ZWy9Iqe5NWSrI2Aw/viewform

[DrSpaldo]

I have created a survey that we want to use to get feedback from users that have been using the GenAI feature for at least 1 week. The feedback will be very helpful in understanding users use cases and how the feature can be improved. The survey is anonymous:

https://docs.google.com/forms/d/e/1FAIpQLSehVvK6oVKzikuKzwlIGz2eiaUwhYQq24ZWy9Iqe5NWSrI2Aw/viewform

I think the survey misses an opportunity to expand a little. For me, I agree that the GenAI feature is limited, sometimes helpful and sometimes not. But, I do like that I can see what has been tagged. It would be good to have the "tags" that have been added for the image from semantic search visible to see.

[NickM-27]

I think some clarification is needed here. Semantic search does not add tags, there is no way to surface this to the user as it does not exist.

If you are instead referring to the genai descriptions specifically, then we'd either need genai to output some kind of structured output or we would need some sophisticated parsing to be able to pull out characteristics in the object description and have them show up as tags.

[arcyleung]

Hats off to the devs @NickM-27 for making it an official feature, I actually ended up rolling my own inference pipeline + semantic search on top of Frigate a few months back to do the same thing. Though that uses plain Transformers to batch process images as opposed to realtime events. I'm glad there's a polished usable version now :)

As for the sentiment that genai descriptions aren't of much use, I agree that it is indeed limited when it comes to describing static images.

However, I think there's more potential instead to feed the LLM model more metadata (when the object entered/ exited the frame etc.) and combining it with the camera orientation/ spatial aware idea from here, to link across multiple detections and come up with explanations for a set of events, rather than individual detections.

In short, detection events can be grouped first by spatial and/or temporal proximity, then sent to the LLM to try and reason about it.

Captures across multiple events can be useful to obtain possible narratives and reconstruct scenarios, and might offer better insight (when you want to find a misplaced object etc.)

{ "model": "llava-llama3", "prompt":"Describe a possible storyline from a security camera: 1 frame taken from front driveway at 4:45pm, and 2 frames taken from backyard at 5:01pm, 5:05pm. List prominent persons and objects, what they might be doing, and where objects may have been placed after this series of events. Be as specific as possible.", "stream": false, "images": [...] } produces: `"response": ""1. 4:45 pm - Front driveway\n- A person, who is wearing a black jacket and pants, is seen walking up the front steps of the house.\n- The house has a garage door visible in the background.\n\n2. 5:01 pm - Backyard frame 1\n- The same person from the previous scene is now shoveling snow in front of a shed in the backyard.\n- There are two shovels present in this scene, one being used by the person and another placed on the ground near the shed.\n\n3. 5:05 pm - Backyard frame 2\n- The same person is still shoveling snow in the backyard, as seen from a different angle of the backyard.\n- A tree stump can be spotted in this scene.\n\nAfter these events, it appears that the person has been clearing the snow from the front and backyard. The snow removed from the front steps may have been piled near the garage door. The shed and tree stump seem to have been left as they were. The second shovel might have been used temporarily or was replaced with a new one during the process of shoveling snow. It's not clear whether any other objects were moved or placed in these scenes, but given that it's winter time, there could be more snow removal happening outside the frame of the security camera footage.""`

I suppose the endgame is some sort of LLM system with object permanence, which is a more power capability on top of facial rec.