[Feature]: More database engines for vector storage

[euller-pereira-brisa]

What feature would you like?

The possibility to configure other database engines for vector storage

What problem would this solve?

SurrealDB is not a bad choice at all, but different teams have pre-existing stacks. A Redis instance can be used as an in-memory vector database as well, SQLite has some extensions that offer vector capabilities, and PostgreSQL has a very robust vector extension. With this feature, it would make adoption easier.

Additional context

No response

[pi22by7]

I agree. Currently the vector database is completely in-memory and has to be rebuilt on every run, which is a issue on its own. This and the other issue you opened align well with the plans of this project.

You can start the implementation, meanwhile I will ponder on whether we should just have one option for ease or present an array of options.

[euller-pereira-brisa]

Sincerely, I don't even know how to begin to solve this issue. Now that I had more time to think, there's more than choosing the vector engine. There's also the question of choosing an embedding model. For an enterprise environment, companies could use embedding models that they have already contracted a plan, and in a personal environment, a developer may want to use a different model.
The other issue is easier to solve, since I just need to decouple SQLite from the implementation.

[pi22by7]

Give me a lil time to think about this, im usually busy with work on weekdays, so I will let you know in a day or two about what we can do about this.