[Platform][Ollama] FallbackModelCatalog for custom platform

[Guikingone]

Q	A
Bug fix?	yes
New feature?	no
Docs?	no
Issues	Fix none (for now)
License	MIT

Hi 👋🏻

It looks like when we're dealing with platforms that can list / detain multiple models at once, the default catalog is "a fixed one", when trying to add a FallbackModelCatalog, the platform fails to retrieve the model as most of them use a custom Model class, this PR aims to introduce / fix the issue with an extra argument.

I know that the solution is not perfect, my main concern is the problem we're facing as we have custom models classes, that's the only way I found to prevent the error:

This approach allows to use any model stored in the platform (especially as the ones listed in the default catalog are not up-to-date).

[Guikingone]

Another approach would be to define a catalog for each platform when needed (Ollama requires a custom one, it's probably the only one with ElevenLabs) and inject it as the default implementation for the platform.

[OskarStark]

I think it's better to open up the FallbackModelCatalog to be able to receive additional models configured from the outside (like for the real ModelCatalog's), check these first and if not go with the Model::class as default. WDYT?

[Guikingone]

Not sure about the implementation but I get the idea of accepting additional models 🤔

My main concern is about the capability to add those models at runtime, my issue here is that if I'm using models stored in Ollama but not in the catalog (so I can't configure them / add them in the catalog directly), the catalog is returning a Model rather than Ollama, of course, it triggers an error later when trying to see if the model is supported 😅

[Guikingone]

Another approach would be to build the catalog from the Ollama API directly, something similar to what we're doing in the OllamaClient, the API already know if the model is supported, IMHO, listing models in the class is a bad idea at the end 🤔

[OskarStark]

Yes, but not all models are available and in the end you cannot be up to date in the end always, so we need to have the option to be able to provide custom models. Take Azure for example, you need to deploy some models, so no model catalog for azure will look the same for a user...

[Guikingone]

What if we're building the catalog directly from the API for specific platform?

What I mean is that in the catalog, we can query the API (when available) and build the model list from it then if a list of models is provided in the constructor, we can merge both at the end of the method, WDYT? 🤔

[OskarStark]

I am not convinced about doing this on runtime, it can be costly, we would need to introduce caches etc. WDYT @chr-hertel

[Guikingone]

I pushed a POC, the idea is to move the code that we already have in the OllamaClient to the OllamaCatalog, this way, the model is built once then you can query the client as many times as needed.

[OskarStark]

now you would need to implement the option handling etc here as well

[Guikingone]

Hum, maybe we can do parent::getModel() to retrieve an object with its options then transform it into the correct model 🤔

[Guikingone]

Something like:

    public function getModel(string $modelName): Model
    {
        $model = parent::getModel($modelName);

        $response = $this->httpClient->request('POST', \sprintf('%s/api/show', $this->host), [
            'json' => [
                'model' => $model->getName(),
            ],
        ]);

        $payload = $response->toArray();

        if ([] === $payload['capabilities'] ?? []) {
            throw new InvalidArgumentException('The model information could not be retrieved from the Ollama API. Your Ollama server might be too old. Try upgrade it.');
        }

        return new Ollama($modelName, $payload['capabilities']);
    }

[chr-hertel]

I am not convinced about doing this on runtime, it can be costly, we would need to introduce caches etc. WDYT @chr-hertel

I don't think it's an issue here. The first call is a fraction of the time compared to the main call later, and we're talking local execution only anyways.

but i agree that at least the OllamaCatalog should have a local cache var so the single instance of that object doesn't need to do it twice - an optional psr cache would be sugar on top but also introduces new questions like cachekey&invalidation after download of a new model etc

[Guikingone]

@chr-hertel What about using the Cache component? Is there any real issue with it? I mean, we can make it optional (along with a TTL), if no cache is provided, the calls are performed each time and if provided, we store it for 10 minutes (or an hour?) 🤔

[chr-hertel]

same like PSR cache. sugar on top & introducing new concerns. not needed from my side for a first PR

[OskarStark]

For reference:

• [Platform] Rename DynamicModelCatalog to FallbackModelCatalog #777

I already updated the PR header and title with the new term FallbackModelCatalog.

I will come up with a proposal shortly