The state of long term memory solutions

[LucaSilva-r]

Hello everyone, lately I've been experimenting with openllm vtuber 1.2 and after fighting with the dockerfile I managed to set up everything. Then I decided to try using the Letta (memgpt) agent to help the AI remember conversations, handle mcp etc...

My setup is pretty basic. OpenLLM-VTuber, Letta, vLLM/ollama all on docker. I began searching around the internet for a model to use and quickly I found out that to use Letta the model must supprot tool/function calling.

This is kind of a problem if you want to use vLLM as their solution looks, from what I understand has way less supported models out of the box by the various models and setting it up it's a bit of a gamble.

Ollama on the other hand in their library already have quite long list of models that support this feature.

The main issue begins once you actually try to use Letta because more often than not, even without passing throug openllm-vtuber, the only result you will get after sending a message to the llm is that there was an error, as the AI didn't make a tool/function call or it tried calling a non existant one.

This error may happen more or less depending on the model you use but eventually you will get one. This is quite unfortunate as openllm-vtuber will not retry sending the message in case of this error so more often than not you just won't get a reply.

I tried many many models and the only one that seemed reliable enough was mistral-small-24b, the issue with that is that even with 48gb of VRAM and a quadro a6000 the model can be quite slow, so it's not the best for a voice chat like application.

I wanted to know if anyone also tried going through this endeavour and what your results where or if you have any tips.

This is the function I think OpenLLM-VTuber needs the most as it's what would really make it feel like the AI more personal.

Thanks for reading this whole novel about my frustrations.