Autogen 0.4.5 | Human in the loop with UI - Advise on architecture

[skye0402]

I'm letting a user interact through telegram with Autogen 0.4.5. So far so good. Now I want to add the human-in-the-loop pattern.

In principle there are 2 reasons why the agent team (I use Swarm or SelectorGroupChat) would hand back to the user:

1. The task is completed with a result (this works)
2. The agent has a question to the user

For that I implemented a UserProxyAgent:

        self.user_proxy = UserProxyAgent(
            name="user_proxy",
            description="A human user of our application. To be contacted or queried for information and feedback as needed.",
            input_func=input_by_user
        )     

When it gets called it calls the input_by_user and that would collect the user feedback.

My questions:

1. Of course I want to know what the question is that the agent has! How can I capture this? I believe it should be somewhere in the agent state but not sure how to route it to the input_func.
2. Sending the question or request for user feedback to the user should be possible (once I have point ① clarified). However, the answer of the user is decoupled as it's telegram messenger. I need to feed it back to the agent, but how? I think I can write logic that would state "this user response is related to the agent's question TRUE/FALSE" but even then, how to feed it back in? We're not speaking of a console input here.

I believe this should be a common challenge as soon as a UI is involved? Any hints?

[ekzhu]

For Swarm group chat, you can use HandoffTermination to terminate the run and restart it with your user input as the next task. You can read about it in this example: https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/swarm.html#customer-support-example.

You can also use HandoffTermination for SelectorGroupChat, it also requires the agent to be able to use the handoff feature: https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/tutorial/human-in-the-loop.html#using-termination-conditions.

The use of HandoffTermination means you don't use UserProxyAgent in a group chat, you stop the group chat, give the control back to the applicaiton, and restart it with a new message. The state of the group chat is maintained in memory, or persisted using save_state and load_state.

---

If for some reason, you must use UserProxyAgent with a UI (e.g., your model doesn't support function calling), then your application must be asynchronous. In the input callback of UserProxyAgent, you need to send a message to the frontend, wait on a specific asyncio.Event, when the user message arrives back in a separate coroutine or your fast api handler, you set the asyncio.Event, and obtain the message to be used as response.

If the above sound too complicated to you -- I agree, and I think there should be a more elegant way to address this scenario for SelectorGroupChat and RoundRobinGroupChat when function calling model is not available.

One idea is to have the group chat just terminate when user proxy is selected as the speaker. It could be done by having the group chat exposes a SpeakerSelectedEvent and add a new termination condition, NextSpeakerMatchTermination, that terminates the group chat when a SpeakerSelectedEvent emitts with a matching spaker.

What do you think?

cc @husseinmozannar @victordibia @jackgerrits , consider this instead of using user proxy's input function, which is a bit foot-gun in terms of UI intergration.

[ekzhu]

Great to hear!

I have updated selector group chat documentation: #5456.

The default selector_prompt is quite problematic... took some time to prompt-engineer this to make it work well.

Glad you found a solution. I think generally if there is a portion of the workflow with deterministic steps, it's better to separate it out from the SelectorGroupChat into its own step.

[richardgg93]

I have a question regarding the HandoffTermination. When using Swarm I want the Agents to either provide a final answer or request feedback. I'm currently capturing Handoffs to "user" as a termination, so when that happens I stop the chat run, save team state, and restart it from state when a new user input arrives. Solution works well but I cannot distinguish final answers from those that are requesting human actions. I have tried adding TextTermination as well but Agents tend to directly Handoff to "user" instead of sending TERMINATE.

Question is: How can I do that? And how can I integrate HandoffTermination in Swarm together with especial Handoff to "user"?
Documentation is not very clear on this.

[ekzhu]

I think you might need to prompt the agent to generate a final answer first before handing off to user, or prompt the agent to generate the final answer in a specific format that you can parse later from the message history.

A more reliable way is to have the agent to use a model client with structured output enabled so you can extract the answer from a field directly. There is a bug in v0.4.6 that prevents this but #5507 will resolve that in the next release.

Would be good to post your code in a separate discussion so we can make suggestions and try it out

[richardgg93]

Thanks for the answer!! The problem is that I'm basically depending on the prompting, which gives me less confidence than forcing the tools usage. For example I have an agent which I am prompting to always handoff to other agent or the user but many times it doesn't work despite the prompts and it doesn't produce a Handoff. Thus it gets stuck in a loop instead of sending the answer to the user because in the Swarm implementation the agent remains the same if there isn't a Handoff. It would be ideal if the Swarm implementation could force somehow the agents to handoff to other agents (force tool calling maybe?) or terminate the chat with a final answer. It may be that I havent seen the right documentation for this but parsing the output "Terminate" or similar approaches dont look very reliable to me.

[ekzhu]

Some good ideas here. If you are not constrained by the AgentChat interface, it would be nice to prototype your idea with the Core API. See the Core API's handoff pattern: https://microsoft.github.io/autogen/stable/user-guide/core-user-guide/design-patterns/handoffs.html.

Yes, it would be great to have a solution for this. I create a separate issue for us to discuss there and a solution. #5611

[ekzhu]

@skye0402 I created a sample to show AgentChat + FastAPI, including a user proxy with a custom input function to allow for user input: #5433

[skye0402]

@ekzhu Hi Eric - apologies for no-reply, just found time again today to look into it. The HandoffTermination you mentioned is a great approach and I will do that. I will also check the example you provided. It's indeed no problem to stop the team and wait for a reply from the user.