MagenticOrchestrator doesn't handle ToolResultSummaryMessage

[fbpazos]

What happened?

In a Magentic Group Settting, if one of the Assitant Agents uses a tool it gives the following error, note this is under a FALSE reflect_on_tool variable.

Making it true, wont happen, though more tokens will be consumed and it will have a worst output and the philosophy of a tool as an answer is not followed...

Error processing publish message for group_chat_manager/15695b1b-40be-4c41-a16b-9c3b7fd919bd
Traceback (most recent call last):
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_core/_single_threaded_agent_runtime.py", line 409, in _on_message
return await agent.on_message(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_core/_base_agent.py", line 113, in on_message
return await self.on_message_impl(message, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_agentchat/teams/_group_chat/_sequential_routed_agent.py", line 48, in on_message_impl
return await super().on_message_impl(message, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_core/_routed_agent.py", line 485, in on_message_impl
return await h(self, message, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_core/_routed_agent.py", line 268, in wrapper
return_value = await func(self, message, ctx) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_agentchat/teams/_group_chat/_magentic_one/_magentic_one_orchestrator.py", line 187, in handle_agent_response
await self._orchestrate_step(ctx.cancellation_token)
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_agentchat/teams/_group_chat/_magentic_one/_magentic_one_orchestrator.py", line 274, in _orchestrate_step
context = self._thread_to_context()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python/3.11.11/lib/python3.11/site-packages/autogen_agentchat/teams/_group_chat/_magentic_one/_magentic_one_orchestrator.py", line 443, in _thread_to_context
assert isinstance(m, TextMessage) or isinstance(m, MultiModalMessage)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

What did you expect to happen?

Executing the tool and the returned information of the tool use is as the main message.

Aka
agent_address_locator
[FunctionExecutionResult(content='(25.769472, -80.132992)', call_id='call_Xm315jQIJ4VVH8STusHJSWEO')]

agent_address_locator
(25.769472, -80.132992) ----> THIS SHOULD BE TAKEN AS THE MESSAGE OF THE AGENT

How can we reproduce it (as minimally and precisely as possible)?

Example agent on the team
agent_address_locator = AssistantAgent(
name="agent_address_locator",
description="This agent can find the coordintes of a given address in the form of latitude and longitude ",
model_client=az_model_client_fct(0),
tools=[GeoUtils.get_adress_coordinates],
reflect_on_tool_use=False,
system_message=""SECRET!"
"""
)

AutoGen version

0.4.1

Which package was this bug in

AgentChat

Model used

gpt-4o

Python version

3.11

Operating system

Ubuntu

Any additional info you think would be helpful for fixing this bug

To fix it in a not fancy way, add

EDIT: /autogen_agentchat/teams/_group_chat/_magentic_one/_magentic_one_orchestrator.py
def _thread_to_context(self) -> List[LLMMessage]:
"""Convert the message thread to a context for the model."""
context: List[LLMMessage] = []
for m in self._message_thread:
if isinstance(m, ToolCallRequestEvent | ToolCallExecutionEvent):
# Ignore tool call messages.
continue
elif isinstance(m, StopMessage | HandoffMessage):
context.append(UserMessage(content=m.content, source=m.source))
elif m.source == self._name:
assert isinstance(m, TextMessage | ToolCallSummaryMessage)
context.append(AssistantMessage(content=m.content, source=m.source))
else:
assert isinstance(m, TextMessage) or isinstance(m, MultiModalMessage)
context.append(UserMessage(content=m.content, source=m.source))
return context

Either to
...
else:
assert isinstance(m, TextMessage) or isinstance(m, MultiModalMessage) or isinstance(m,ToolCallSummaryMessage)
context.append(UserMessage(content=m.content, source=m.source))

OR
...
elif m.source == self._name or isinstance(m, TextMessage | ToolCallSummaryMessage):
assert isinstance(m, TextMessage | ToolCallSummaryMessage)
context.append(AssistantMessage(content=m.content, source=m.source))

[ekzhu]

@fbpazos would you like to submit a PR for the fix?

assert isinstance(m, TextMessage) or isinstance(m, MultiModalMessage) or isinstance(m,ToolCallSummaryMessage)
context.append(UserMessage(content=m.content, source=m.source))

This makes sense. It should be treated as a regular text message.

[fbpazos]

Yes I would like to submit

[jackgerrits]

We should keep this open until it is fixed. @fbpazos please feel free to open a PR fixing this and just put closes #5059 in the description and this will automatically be closed when it is merged. Thank you!

[fbpazos]

Alright, thanks @jackgerrits
PR: #5156