[Feature Request] Support for multiple Ollama servers simultaneously from a single Agent

[stevefoote]

Is there any plan for the Embabel framework to support multiple Ollama servers simultaneously (from the same Agent)? In our production environment, we have multiple Ollama servers used for different qualities of service. Yes, we are aware that a single Ollama server can host multiple models. But we spread our inferencing load across several Ollama servers with varying infrastructure capacity to support a broad spectrum of users. It would be very beneficial to be able to have a single agent (based upon the Embabel framework) accessing more than just one Ollama server at a time.

[alexheifetz]

@igordayen what is the status on this one?

[igordayen]

hi @stevefoote . may i inquire on multi-instance deployment usability. suppose there is config:

spring:
  ai:
    ollama:
      instances:
        summarizer:
          base-url: http://localhost:11434
          model: gemma3
          tags: [fast]
        retriever:
          base-url: http://localhost:11434
          model: mistral, qwen3
          tags: [accurate]

1. access models per instance, i.e. seggregate models space into "fast" / "accurate", or:
2. aggregate modules cross instances, no additional level to step into instance model space is required. best practice for pattern [2] would be to avoid models duplication cross instances.

Could you please which scenario would be more practical. thank you

[stevefoote]

hi @stevefoote . may i inquire on multi-instance deployment usability. suppose there is config:

spring:
  ai:
    ollama:
      instances:
        summarizer:
          base-url: http://localhost:11434
          model: gemma3
          tags: [fast]
        retriever:
          base-url: http://localhost:11434
          model: mistral, qwen3
          tags: [accurate]

1. access models per instance, i.e. seggregate models space into "fast" / "accurate", or:
2. aggregate modules cross instances, no additional level to step into instance model space is required. best practice for pattern [2] would be to avoid models duplication cross instances.

Could you please which scenario would be more practical. thank you

Hi @igordayen - thanks very much for your clarification question. Scenario #1 please.

[igordayen]

@stevefoote
i raised PR: #998
you are welcome to preview / pre-test from branch (before merge):
multi-ollama-instances.
Thank you.

[alexheifetz]

PR #998 will be included as part of 0.2.0 Tag Build
@stevefoote please validate and provide feedback.

[stevefoote]

PR #998 will be included as part of 0.2.0 Tag Build @stevefoote please validate and provide feedback.

@igordayen Thank you very much for this new feature. This opens up a world of possibilities for Embabel-based agents to leverage multiple Ollama-hosted private LLMs simultaneously, each tailored for specific purposes.