Any way to set larger context length when using Ollama models?

[rylativity]

I have successfully set up autogen to use Ollama and create agents and teams. I am currently using Ollama as the inference engine behind a MagneticOneGroupChat team and it is working well.

However, when I look at the Ollama model resource usage, it is clear (based on RAM/vRAM usage) that it is only using the default context length of 2048, even though the relevant context extends far beyond that context length limit as the MagneticOne team continues to work and create additional output. There is no error thrown, but I have seen a couple instances where the team attempt to perform tasks they have already completed further up in the team "chat", which (when combined with the evidence from RAM/vRAM usage) leads me to believe that any context beyond 2048 tokens is being silently truncated.

Is there a way to set an explicit, increased context length when instantiating an OpenAIChatCompletionClient, Agent, or Team? (Relevant documentation for using Ollama with Autogen here)

[kb-]

You can create new models on Ollama, with a larger context. https://ollama.readthedocs.io/en/modelfile/#format

Create a Modelfile

FROM llama3.1:8b-instruct-q4_K_M

PARAMETER num_ctx 8096

Then create your model, derived from a model you have
ollama create llama3.1-8b-instruct-q4_K_M-8k --file Modelfile

You can then use llama3.1-8b-instruct-q4_K_M-8k like other Ollama models.

[rylativity]

Thanks for your response, @kb-. I want to avoid defining a new Ollama model if possible, so I apologize for not being more specific.

I'd like to be able to set the context length (and ideally other options) at query/inference time. Ollama allows passing a num_ctx key & value in the options object sent in a request body to the /generate and /chat endpoints (example of num_ctx in request shown here).

Open WebUI, for example, takes advantage of this and allows you to set num_ctx on a query-by-query basis without defining a new model in Ollama for every possible context length you might want to use.

I'm wondering if there is a way to pass query-time generation options through to Ollama from Autogen by setting a value when instantiating a CompletionClient or Agent or when running a task.

[ekzhu]

You can include extra arguments in extra_create_args in the create method.

https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.openai.html#autogen_ext.models.openai.BaseOpenAIChatCompletionClient.create

[rylativity]

@ekzhu I'm seeing two issues there. The first issue is that when I try to pass "num_ctx" like this...

await model_client.create([UserMessage(content="hi", source="user")], extra_create_args={"num_ctx":30000})

or like this...

await model_client.create([UserMessage(content="hi", source="user")], extra_create_args={"options":{"num_ctx":30000}})

I get an error saying ValueError: Extra create args are invalid: {'num_ctx'} or ValueError: Extra create args are invalid: {'options'}.

The second issue is that even if num_ctx and other options could be passed to the .create() method, it's not clear how I could get those options to an Agent or Team, since I don't explicitly call .create() when running a task through an Agent or Team.

If you were able to successfully pass num_ctx to an Autogen Agent or Team (or even just directly to the .create() method), will you please share the code that worked for you?

[ekzhu]

@rylativity I haven't worked on Ollama extensively but it seems that the OpenAIChatCompletionClient may not be the right choice if you hae custom arguments that are not in the OpenAI's specs.

We have an OllamaClient now, it is still in the main branch. See the dev doc here: https://microsoft.github.io/autogen/dev/reference/python/autogen_ext.models.ollama.html

Could you try it out? If it works for you please let us know here. If it doesn't work, please open an issue regarding the Ollama client.

[rylativity]

@ekzhu thanks for pointing out the autogen Ollama client.

I had a chance to start digging into it a bit, and there appears to be an issue with how the autogen OllamaChatCompletionClient handles the host argument value that is used for passing an Ollama host URL through to the Ollama python client constructor. While the host argument value is correctly passed to the Ollama python client constructor, it is then also passed through to every call to the Ollama client's .chat() method, which raises a TypeError as the Ollama client is not expecting that host kwarg to be passed to the .chat() method:

TypeError: AsyncClient.chat() got an unexpected keyword argument 'host'

I will try to open an issue and a PR this weekend to resolve the issue.