Python: Add support for Azure AI Agent Service (#10414)

### Motivation and Context

Azure AI Agents are a managed offering, and are built on top of the
OpenAI assistant v2 APIs. Until now we have not had support for them in
SK Python. This PR adds support for Azure AI Agents in SK Python. The
current abstractions allow one to run managed Azure AI Agents using SK
constructs as is demonstrated via the getting started samples and/or the
`azure_ai_agent` concept code samples. The developer must first follow
the required Azure AI Agent deployment steps, and bring their connection
string as well as their model deployment name.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Add support for Azure AI Agent Service in SK
- Add sample code
- Add unit tests
- Restructures the `getting_started_with_agents` samples to move the
respective agent code into folders like `chat_completion`,
`azure_ai_agent`, etc. Updates the README as well.
- Closes #10187 

**TODO**: add more test coverage either in this PR, or a subsequent one.

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄

python/pyproject.toml

@@ -51,6 +51,7 @@ dependencies = [
 [project.optional-dependencies]
 azure = [
     "azure-ai-inference >= 1.0.0b6",
+    "azure-ai-projects >= 1.0.0b5",
     "azure-core-tracing-opentelemetry >= 1.0.0b11",
     "azure-search-documents >= 11.6.0b4",
     "azure-identity ~= 1.13",

python/samples/concepts/agents/azure_ai_agent/.env.example

@@ -0,0 +1,6 @@
+AZURE_AI_AGENT_PROJECT_CONNECTION_STRING = "<example-connection-string>"
+AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "<example-model-deployment-name>"
+AZURE_AI_AGENT_ENDPOINT = "<example-endpoint>"
+AZURE_AI_AGENT_SUBSCRIPTION_ID = "<example-subscription-id>"
+AZURE_AI_AGENT_RESOURCE_GROUP_NAME = "<example-resource-group-name>"
+AZURE_AI_AGENT_PROJECT_NAME = "<example-project-name>"

python/samples/concepts/agents/azure_ai_agent/README.md

@@ -0,0 +1,3 @@
+## Azure AI Agents
+
+For details on using Azure AI Agents within Semantic Kernel, please see the [README](../../../getting_started_with_agents/azure_ai_agent/README.md) located in the `getting_started_with_agents/azure_ai_agent` directory.

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_file_manipulation.py

@@ -0,0 +1,93 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+import os
+
+from azure.ai.projects.aio import AIProjectClient
+from azure.ai.projects.models import CodeInterpreterTool, FilePurpose
+from azure.identity.aio import DefaultAzureCredential
+
+from semantic_kernel.agents.azure_ai import AzureAIAgent, AzureAIAgentSettings
+from semantic_kernel.contents.annotation_content import AnnotationContent
+from semantic_kernel.contents.chat_message_content import ChatMessageContent
+from semantic_kernel.contents.utils.author_role import AuthorRole
+
+###################################################################
+# The following sample demonstrates how to create a simple,       #
+# Azure AI agent that uses the code interpreter tool to answer    #
+# a coding question.                                              #
+###################################################################
+
+
+async def main() -> None:
+    ai_agent_settings = AzureAIAgentSettings.create()
+
+    async with (
+        DefaultAzureCredential() as creds,
+        AIProjectClient.from_connection_string(
+            credential=creds,
+            conn_str=ai_agent_settings.project_connection_string.get_secret_value(),
+        ) as client,
+    ):
+        csv_file_path = os.path.join(
+            os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))),
+            "resources",
+            "agent_assistant_file_manipulation",
+            "sales.csv",
+        )
+
+        file = await client.agents.upload_file_and_poll(file_path=csv_file_path, purpose=FilePurpose.AGENTS)
+
+        code_interpreter = CodeInterpreterTool(file_ids=[file.id])
+
+        # Create agent definition
+        agent_definition = await client.agents.create_agent(
+            model=ai_agent_settings.model_deployment_name,
+            tools=code_interpreter.definitions,
+            tool_resources=code_interpreter.resources,
+        )
+
+        # Create the AzureAI Agent
+        agent = AzureAIAgent(
+            client=client,
+            definition=agent_definition,
+        )
+
+        # Create a new thread
+        thread = await client.agents.create_thread()
+
+        user_inputs = [
+            "Which segment had the most sales?",
+            "List the top 5 countries that generated the most profit.",
+            "Create a tab delimited file report of profit by each country per month.",
+        ]
+
+        try:
+            for user_input in user_inputs:
+                # Add the user input as a chat message
+                await agent.add_chat_message(
+                    thread_id=thread.id,
+                    message=ChatMessageContent(role=AuthorRole.USER, content=user_input),
+                )
+                print(f"# User: '{user_input}'")
+                # Invoke the agent for the specified thread
+                async for content in agent.invoke(thread_id=thread.id):
+                    if content.role != AuthorRole.TOOL:
+                        print(f"# Agent: {content.content}")
+                        if len(content.items) > 0:
+                            for item in content.items:
+                                if isinstance(item, AnnotationContent):
+                                    print(f"\n`{item.quote}` => {item.file_id}")
+                                    response_content = await client.agents.get_file_content(file_id=item.file_id)
+                                    content_bytes = bytearray()
+                                    async for chunk in response_content:
+                                        content_bytes.extend(chunk)
+                                    tab_delimited_text = content_bytes.decode("utf-8")
+                                    print(tab_delimited_text)
+        finally:
+            await client.agents.delete_thread(thread.id)
+            await client.agents.delete_agent(agent.id)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_streaming.py

@@ -0,0 +1,94 @@
+# Copyright (c) Microsoft. All rights reserved.
+
+import asyncio
+from typing import Annotated
+
+from azure.ai.projects.aio import AIProjectClient
+from azure.identity.aio import DefaultAzureCredential
+
+from semantic_kernel.agents.azure_ai import AzureAIAgent, AzureAIAgentSettings
+from semantic_kernel.contents.chat_message_content import ChatMessageContent
+from semantic_kernel.contents.utils.author_role import AuthorRole
+from semantic_kernel.functions.kernel_function_decorator import kernel_function
+
+
+# Define a sample plugin for the sample
+class MenuPlugin:
+    """A sample Menu Plugin used for the concept sample."""
+
+    @kernel_function(description="Provides a list of specials from the menu.")
+    def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]:
+        return """
+        Special Soup: Clam Chowder
+        Special Salad: Cobb Salad
+        Special Drink: Chai Tea
+        """
+
+    @kernel_function(description="Provides the price of the requested menu item.")
+    def get_item_price(
+        self, menu_item: Annotated[str, "The name of the menu item."]
+    ) -> Annotated[str, "Returns the price of the menu item."]:
+        return "$9.99"
+
+
+async def main() -> None:
+    ai_agent_settings = AzureAIAgentSettings.create()
+
+    async with (
+        DefaultAzureCredential() as creds,
+        AIProjectClient.from_connection_string(
+            credential=creds,
+            conn_str=ai_agent_settings.project_connection_string.get_secret_value(),
+        ) as client,
+    ):
+        AGENT_NAME = "Host"
+        AGENT_INSTRUCTIONS = "Answer questions about the menu."
+
+        # Create agent definition
+        agent_definition = await client.agents.create_agent(
+            model=ai_agent_settings.model_deployment_name,
+            name=AGENT_NAME,
+            instructions=AGENT_INSTRUCTIONS,
+        )
+
+        # Create the AzureAI Agent
+        agent = AzureAIAgent(
+            client=client,
+            definition=agent_definition,
+        )
+
+        # Add the sample plugin to the kernel
+        agent.kernel.add_plugin(MenuPlugin(), plugin_name="menu")
+
+        # Create a new thread
+        thread = await client.agents.create_thread()
+
+        user_inputs = [
+            "Hello",
+            "What is the special soup?",
+            "How much does that cost?",
+            "Thank you",
+        ]
+
+        try:
+            for user_input in user_inputs:
+                # Add the user input as a chat message
+                await agent.add_chat_message(
+                    thread_id=thread.id, message=ChatMessageContent(role=AuthorRole.USER, content=user_input)
+                )
+                print(f"# User: '{user_input}'")
+                first_chunk = True
+                async for content in agent.invoke_stream(thread_id=thread.id):
+                    if content.role != AuthorRole.TOOL:
+                        if first_chunk:
+                            print(f"# {content.role}: ", end="", flush=True)
+                            first_chunk = False
+                        print(content.content, end="", flush=True)
+                print()
+        finally:
+            await client.agents.delete_thread(thread.id)
+            await client.agents.delete_agent(agent.id)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

python/samples/getting_started_with_agents/README.md

@@ -16,20 +16,36 @@ This project contains a step by step guide to get started with _Semantic Kernel
 
 The getting started with agents examples include:
 
+## Chat Completion
+
+Example|Description
+---|---
+[step1_agent](../getting_started_with_agents/chat_completion/step1_agent.py)|How to create and use an agent.
+[step2_plugins](../getting_started_with_agents/chat_completion/step2_plugins.py)|How to associate plugins with an agent.
+[step3_chat](../getting_started_with_agents/chat_completion/step3_chat.py)|How to create a conversation between agents.
+[step4_kernel_function_strategies](../getting_started_with_agents/chat_completion/step4_kernel_function_strategies.py)|How to utilize a `KernelFunction` as a chat strategy.
+[step5_json_result](../getting_started_with_agents/chat_completion/step5_json_result.py)|How to have an agent produce JSON.
+[step6_logging](../getting_started_with_agents/chat_completion/step6_logging.py)|How to enable logging for agents.
+
+## OpenAI Assistant
+
 Example|Description
 ---|---
-[step1_agent](../getting_started_with_agents/step1_agent.py)|How to create and use an agent.
-[step2_plugins](../getting_started_with_agents/step2_plugins.py)|How to associate plugins with an agent.
-[step3_chat](../getting_started_with_agents/step3_chat.py)|How to create a conversation between agents.
-[step4_kernel_function_strategies](../getting_started_with_agents/step4_kernel_function_strategies.py)|How to utilize a `KernelFunction` as a chat strategy.
-[step5_json_result](../getting_started_with_agents/step5_json_result.py)|How to have an agent produce JSON.
-[step6_logging](../getting_started_with_agents/step6_logging.py)|How to enable logging for agents.
-[step7_assistant](../getting_started_with_agents/step7_assistant.py)|How to create and use an OpenAI Assistant agent.
-[step8_assistant_vision](../getting_started_with_agents/step8_assistant_vision.py)|How to provide an image as input to an Open AI Assistant agent.
-[step9_assistant_tool_code_interpreter](../getting_started_with_agents/step9_assistant_tool_code_interpreter.py)|How to use the code-interpreter tool for an Open AI Assistant agent.
-[step10_assistant_tool_file_search](../getting_started_with_agents/step10_assistant_tool_file_search.py)|How to use the file-search tool for an Open AI Assistant agent.
-
-*Note: As we strive for parity with .NET, more getting_started_with_agent samples will be added. The current steps and names may be revised to further align with our .NET counterpart.*
+[step1_assistant](../getting_started_with_agents/openai_assistant/step1_assistant.py)|How to create and use an OpenAI Assistant agent.
+[step2_assistant_vision](../getting_started_with_agents/openai_assistant/step2_assistant_vision.py)|How to provide an image as input to an Open AI Assistant agent.
+[step3_assistant_tool_code_interpreter](../getting_started_with_agents/openai_assistant/step3_assistant_tool_code_interpreter.py)|How to use the code-interpreter tool for an Open AI Assistant agent.
+[step4_assistant_tool_file_search](../getting_started_with_agents/openai_assistant/step4_assistant_tool_file_search.py)|How to use the file-search tool for an Open AI Assistant agent.
+
+## Azure AI Agent
+Example|Description
+---|---
+[step1_azure_ai_agent](../getting_started_with_agents/azure_ai_agent/step1_azure_ai_agent.py)|How to create an Azure AI Agent and invoke a Semantic Kernel plugin.
+[step2_azure_ai_agent_chat](../getting_started_with_agents/azure_ai_agent/step2_azure_ai_agent_chat.py)|How to an agent group chat with Azure AI Agents.
+[step3_azure_ai_agent_code_interpreter](../getting_started_with_agents/azure_ai_agent/step3_azure_ai_agent_code_interpreter.py)|How to use the code-interpreter tool for an Azure AI agent.
+[step4_azure_ai_agent_file_search](../getting_started_with_agents/azure_ai_agent/step4_azure_ai_agent_file_search.py)|How to use the file-search tool for an Azure AI agent.
+[step5_azure_ai_agent_openapi](../getting_started_with_agents/azure_ai_agent/step5_azure_ai_agent_openapi.py)|How to use the Open API tool for an Azure AI  agent.
+
+_Note: For details on configuring an Azure AI Agent, please see [here](../getting_started_with_agents/azure_ai_agent/README.md)._
 
 ## Configuring the Kernel
 

python/samples/getting_started_with_agents/azure_ai_agent/.env.example

@@ -0,0 +1,6 @@
+AZURE_AI_AGENT_PROJECT_CONNECTION_STRING = "<example-connection-string>"
+AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "<example-model-deployment-name>"
+AZURE_AI_AGENT_ENDPOINT = "<example-endpoint>"
+AZURE_AI_AGENT_SUBSCRIPTION_ID = "<example-subscription-id>"
+AZURE_AI_AGENT_RESOURCE_GROUP_NAME = "<example-resource-group-name>"
+AZURE_AI_AGENT_PROJECT_NAME = "<example-project-name>"

python/samples/getting_started_with_agents/azure_ai_agent/README.md

@@ -0,0 +1,65 @@
+## Azure AI Agents
+
+The following getting started samples show how to use Azure AI Agents with Semantic Kernel.
+
+To set up the required resources, follow the "Quickstart: Create a new agent" guide [here](https://learn.microsoft.com/en-us/azure/ai-services/agents/quickstart?pivots=programming-language-python-azure).
+
+You will need to install the optional Semantic Kernel `azure` dependencies if you haven't already via:
+
+```bash
+pip install semantic-kernel[azure]
+```
+
+Before running an Azure AI Agent, modify your .env file to include:
+
+```bash
+AZURE_AI_AGENT_PROJECT_CONNECTION_STRING = "<example-connection-string>"
+AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "<example-model-deployment-name>"
+```
+
+or
+
+```bash
+AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "<example-model-deployment-name>"
+AZURE_AI_AGENT_ENDPOINT = "<example-endpoint>"
+AZURE_AI_AGENT_SUBSCRIPTION_ID = "<example-subscription-id>"
+AZURE_AI_AGENT_RESOURCE_GROUP_NAME = "<example-resource-group-name>"
+AZURE_AI_AGENT_PROJECT_NAME = "<example-project-name>"
+```
+
+The .env should be placed in the root directory.
+
+### Configuring the AI Project Client
+
+This can be done in one of two ways:
+
+```python
+ai_agent_settings = AzureAIAgentSettings.create()
+
+async with (
+    DefaultAzureCredential() as creds,
+    AIProjectClient.from_connection_string(
+        credential=creds,
+        conn_str=ai_agent_settings.project_connection_string.get_secret_value(),
+    ) as client,
+):
+# code
+```
+
+or 
+
+```python
+ai_agent_settings = AzureAIAgentSettings.create()
+
+async with (
+    DefaultAzureCredential() as creds,
+    AIProjectClient(
+        credential=creds,
+        endpoint=ai_agent_settings.endpoint,
+        subscription_id=ai_agent_settings.subscription_id,
+        resource_group_name=ai_agent_settings.resource_group_name,
+        project_name=ai_agent_settings.project_name
+    ) as client,
+):
+# code
+```