.Net: Structured Data Plugin - Query and CRUD Operations (#10858)

### Motivation and Context

- Resolves Partially #10099
- Resolves #10733
- Resolves #10734

Developers need a clear example of how to integrate database operations
with Semantic Kernel for AI-powered data interactions. The
StructuredDataConnector demo provides a practical implementation showing
how to leverage SK's plugin architecture with Entity Framework Core for
structured data operations.

### Description
- Implemented `StructuredDataConnector` demo showcasing database-AI
integration
- Added `StructuredDataService<TContext>` for handling basic CRUD
operations
- Created `StructuredDataPluginFactory` for easy plugin creation
- Included example using OpenAI chat completion with function calling
- Added interactive mode for testing database queries through natural
language

The demo provides a reference implementation for developers wanting to
integrate database operations with SK's AI capabilities.

Author: rogerbarreto

docs/decisions/0067-structured-data-connector.md

@@ -0,0 +1,82 @@
+---
+status: proposed
+contact: rogerbarreto
+date: 2025-03-07
+deciders: rogerbarreto, markwallace, dmytrostruk, westey-m, sergeymenshykh
+---
+
+# Structured Data Plugin Implementation in Semantic Kernel
+
+## Context and Problem Statement
+
+Modern AI applications often need to interact with structured data in databases while leveraging LLM capabilities. As Semantic Kernel's core focuses on AI orchestration, we need a standardized approach to integrate database operations with AI capabilities. This ADR proposes an experimental StructuredDataConnector as an initial solution for database-AI integration, focusing on basic CRUD operations and simple querying.
+
+## Decision Drivers
+
+- Need for initial database integration pattern with SK
+- Requirement for basic composable AI and database operations
+- Alignment with SK's plugin architecture
+- Ability to validate the approach through real-world usage
+- Support for strongly-typed schema validation
+- Consistent JSON formatting for AI interactions
+
+## Key Benefits
+
+1. **Plugin-Based Architecture**
+
+   - Aligns with SK's plugin architecture
+   - Supports extension methods for common operations
+   - Leverages KernelJsonSchema for type safety
+
+2. **Structured Data Operations**
+
+   - CRUD operations with schema validation
+   - JSON-based interactions with proper formatting
+   - Type-safe database operations
+
+3. **Integration Features**
+
+   - Built-in JSON schema generation
+   - Automatic type conversion
+   - Pretty-printed JSON for better AI interactions
+
+## Implementation Details
+
+The implementation includes:
+
+1. Core Components:
+
+   - `StructuredDataService<TContext>`: Base service for database operations
+   - `StructuredDataServiceExtensions`: Extension methods for CRUD operations
+   - `StructuredDataPluginFactory`: Factory for creating SK plugins
+   - Integration with `KernelJsonSchema` for type validation
+
+2. Key Features:
+
+   - Automatic schema generation from entity types
+   - Properly formatted JSON responses
+   - Extension-based architecture for maintainability
+   - Support for Entity Framework Core
+
+3. Usage Example:
+
+```csharp
+var service = new StructuredDataService<ApplicationDbContext>(dbContext);
+var plugin = StructuredDataPluginFactory.CreateStructuredDataPlugin<ApplicationDbContext, MyEntity>(
+    service,
+    operations: StructuredDataOperation.Default);
+```
+
+## Decision Outcome
+
+Chosen option: TBD:
+
+1. Provides standardized database integration
+2. Leverages SK's schema validation capabilities
+3. Supports proper JSON formatting for AI interactions
+4. Maintains type safety through generated schemas
+5. Follows established SK patterns and principles
+
+## More Information
+
+This is an experimental approach that will evolve based on community feedback.

dotnet/Directory.Packages.props

@@ -20,9 +20,11 @@
     <PackageVersion Include="Azure.Identity" Version="1.13.2" />
     <PackageVersion Include="Azure.Monitor.OpenTelemetry.Exporter" Version="1.3.0" />
     <PackageVersion Include="Azure.Search.Documents" Version="11.6.0" />
+    <PackageVersion Include="Community.OData.Linq" Version="2.1.0" />
     <PackageVersion Include="Dapr.Actors" Version="1.14.0" />
     <PackageVersion Include="Dapr.Actors.AspNetCore" Version="1.14.0" />
     <PackageVersion Include="Dapr.AspNetCore" Version="1.14.0" />
+    <PackageVersion Include="EntityFramework" Version="6.5.1" />
     <PackageVersion Include="FastBertTokenizer" Version="1.0.28" />
     <PackageVersion Include="Google.Apis.Auth" Version="1.69.0" />
     <PackageVersion Include="mcpdotnet" Version="1.0.1.3" />

dotnet/SK-dotnet.sln

@@ -485,6 +485,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Agents.Bedrock", "src\Agent
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ModelContextProtocol", "samples\Demos\ModelContextProtocol\ModelContextProtocol.csproj", "{B16AC373-3DA8-4505-9510-110347CD635D}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StructuredDataPlugin", "samples\Demos\StructuredDataPlugin\StructuredDataPlugin.csproj", "{BEDAC050-016A-46F4-9173-339C46DFD3ED}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Plugins.StructuredData.EntityFramework", "src\Plugins\Plugins.StructuredData.EntityFramework\Plugins.StructuredData.EntityFramework.csproj", "{0C81C377-3CDC-46A8-BED1-4B50BDA2202E}"
+EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SqlServerIntegrationTests", "src\VectorDataIntegrationTests\SqlServerIntegrationTests\SqlServerIntegrationTests.csproj", "{A5E6193C-8431-4C6E-B674-682CB41EAA0C}"
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "PineconeIntegrationTests", "src\VectorDataIntegrationTests\PineconeIntegrationTests\PineconeIntegrationTests.csproj", "{E9A74E0C-BC02-4DDD-A487-89847EDF8026}"
@@ -1330,6 +1334,18 @@ Global
 		{B16AC373-3DA8-4505-9510-110347CD635D}.Publish|Any CPU.Build.0 = Debug|Any CPU
 		{B16AC373-3DA8-4505-9510-110347CD635D}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{B16AC373-3DA8-4505-9510-110347CD635D}.Release|Any CPU.Build.0 = Release|Any CPU
+		{BEDAC050-016A-46F4-9173-339C46DFD3ED}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{BEDAC050-016A-46F4-9173-339C46DFD3ED}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{BEDAC050-016A-46F4-9173-339C46DFD3ED}.Publish|Any CPU.ActiveCfg = Debug|Any CPU
+		{BEDAC050-016A-46F4-9173-339C46DFD3ED}.Publish|Any CPU.Build.0 = Debug|Any CPU
+		{BEDAC050-016A-46F4-9173-339C46DFD3ED}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{BEDAC050-016A-46F4-9173-339C46DFD3ED}.Release|Any CPU.Build.0 = Release|Any CPU
+		{0C81C377-3CDC-46A8-BED1-4B50BDA2202E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{0C81C377-3CDC-46A8-BED1-4B50BDA2202E}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{0C81C377-3CDC-46A8-BED1-4B50BDA2202E}.Publish|Any CPU.ActiveCfg = Publish|Any CPU
+		{0C81C377-3CDC-46A8-BED1-4B50BDA2202E}.Publish|Any CPU.Build.0 = Publish|Any CPU
+		{0C81C377-3CDC-46A8-BED1-4B50BDA2202E}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{0C81C377-3CDC-46A8-BED1-4B50BDA2202E}.Release|Any CPU.Build.0 = Release|Any CPU
 		{A5E6193C-8431-4C6E-B674-682CB41EAA0C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{A5E6193C-8431-4C6E-B674-682CB41EAA0C}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{A5E6193C-8431-4C6E-B674-682CB41EAA0C}.Publish|Any CPU.ActiveCfg = Debug|Any CPU
@@ -1523,6 +1539,8 @@ Global
 		{DAD5FC6A-8CA0-43AC-87E1-032DFBD6B02A} = {3F260A77-B6C9-97FD-1304-4B34DA936CF4}
 		{8C658E1E-83C8-4127-B8BF-27A638A45DDD} = {6823CD5E-2ABE-41EB-B865-F86EC13F0CF9}
 		{B16AC373-3DA8-4505-9510-110347CD635D} = {5D4C0700-BBB5-418F-A7B2-F392B9A18263}
+		{BEDAC050-016A-46F4-9173-339C46DFD3ED} = {5D4C0700-BBB5-418F-A7B2-F392B9A18263}
+		{0C81C377-3CDC-46A8-BED1-4B50BDA2202E} = {D6D598DF-C17C-46F4-B2B9-CDE82E2DE132}
 		{A5E6193C-8431-4C6E-B674-682CB41EAA0C} = {4F381919-F1BE-47D8-8558-3187ED04A84F}
 		{E9A74E0C-BC02-4DDD-A487-89847EDF8026} = {4F381919-F1BE-47D8-8558-3187ED04A84F}
 	EndGlobalSection

dotnet/samples/Demos/StructuredDataPlugin/ApplicationDbContext.cs

@@ -0,0 +1,51 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.Data.Entity;
+using Microsoft.Extensions.Configuration;
+
+namespace StructuredDataPlugin;
+
+/// <summary>
+/// Represents a database context for the structured data plugin demo.
+/// Inherits from Entity Framework's DbContext to provide database access and management.
+/// </summary>
+internal sealed class ApplicationDbContext : DbContext
+{
+    /// <summary>
+    /// Initializes a new instance of the <see cref="ApplicationDbContext"/> class using configuration settings.
+    /// </summary>
+    /// <param name="config">The configuration object containing connection string information.</param>
+    /// <remarks>
+    /// The connection string is retrieved from the configuration using the pattern "ConnectionStrings:ApplicationDbContext".
+    /// </remarks>
+    public ApplicationDbContext(IConfiguration config) :
+        base(config[$"ConnectionStrings:{nameof(ApplicationDbContext)}"]!)
+    {
+    }
+
+    /// <summary>
+    /// Initializes a new instance of the <see cref="ApplicationDbContext"/> class using a direct connection string.
+    /// </summary>
+    /// <param name="connectionString">The connection string to the database.</param>
+    public ApplicationDbContext(string connectionString)
+        : base(connectionString)
+    {
+    }
+
+    /// <summary>
+    /// Configures the database model and its relationships.
+    /// </summary>
+    /// <param name="modelBuilder">The builder being used to construct the model for this context.</param>
+    /// <remarks>
+    /// This method:
+    /// - Disables database initialization
+    /// - Maps the Product entity to the "Products" table
+    /// - Calls the base configuration
+    /// </remarks>
+    protected override void OnModelCreating(DbModelBuilder modelBuilder)
+    {
+        Database.SetInitializer<ApplicationDbContext>(null);
+        modelBuilder.Entity<Product>().ToTable("Products");
+        base.OnModelCreating(modelBuilder);
+    }
+}

dotnet/samples/Demos/StructuredDataPlugin/Product.cs

@@ -0,0 +1,39 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using System.ComponentModel;
+using System.ComponentModel.DataAnnotations;
+using System.ComponentModel.DataAnnotations.Schema;
+
+namespace StructuredDataPlugin;
+
+/// <summary>
+/// Represents a product entity in the database.
+/// </summary>
+public sealed class Product
+{
+    /// <summary>
+    /// The unique identifier for the product.
+    /// </summary>
+    [Description("The unique identifier for the product.")]
+    [Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
+    public Guid? Id { get; set; }
+
+    /// <summary>
+    /// The description of the product.
+    /// </summary>
+    [Description("The name of the product.")]
+    public string? Name { get; set; }
+
+    /// <summary>
+    /// The price of the product.
+    /// </summary>
+    [Description("The price of the product in USD.")]
+    public decimal? Price { get; set; }
+
+    /// <summary>
+    /// The date the product was created.
+    /// </summary>
+    [Description("The date the product was created")]
+    [DatabaseGenerated(DatabaseGeneratedOption.Computed)]
+    public DateTime? DateCreated { get; set; }
+}

dotnet/samples/Demos/StructuredDataPlugin/Program.cs

@@ -0,0 +1,93 @@
+// Copyright (c) Microsoft. All rights reserved.
+
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.SemanticKernel;
+using Microsoft.SemanticKernel.Connectors.OpenAI;
+
+namespace StructuredDataPlugin;
+
+internal sealed class Program
+{
+    internal static async Task Main(string[] args)
+    {
+        var serviceCollection = new ServiceCollection()
+            .AddSingleton<IConfiguration>((sp) => new ConfigurationBuilder()
+                .AddJsonFile("appsettings.json", optional: true)
+                .AddEnvironmentVariables()
+                .AddUserSecrets<Program>()
+                .Build())
+            .AddTransient<ApplicationDbContext>()
+            .AddTransient<StructuredDataService<ApplicationDbContext>>();
+
+        var serviceProvider = serviceCollection.BuildServiceProvider();
+        var config = serviceProvider.GetRequiredService<IConfiguration>();
+        using var structuredDataService = serviceProvider.GetRequiredService<StructuredDataService<ApplicationDbContext>>();
+
+        // Create kernel builder and add OpenAI
+        var kernelBuilder = Kernel.CreateBuilder()
+            .AddOpenAIChatCompletion(
+                modelId: "gpt-4o",
+                apiKey: config["OpenAI:ApiKey"]!);
+
+        // Add the database plugin using the factory with default operations
+        var databasePlugin = StructuredDataPluginFactory.CreateStructuredDataPlugin<ApplicationDbContext, Product>(
+            structuredDataService);
+
+        kernelBuilder.Plugins.Add(databasePlugin);
+
+        // Build the kernel and add the plugin
+        var kernel = kernelBuilder.Build();
+
+        // Create settings for function calling
+        var settings = new OpenAIPromptExecutionSettings
+        {
+            FunctionChoiceBehavior = FunctionChoiceBehavior.Auto()
+        };
+
+        // Example 1: Inserting new records
+        Console.WriteLine("Creating a new record...");
+        var insertPrompt = "Insert a new product with name 'Book' and price 29.99";
+        var insertResult = await kernel.InvokePromptAsync(insertPrompt, new(settings));
+        Console.WriteLine($"Insert Result: {insertResult}");
+
+        // Example 2: Querying data
+        Console.WriteLine("\nQuerying specific records...");
+        var queryPrompt = "Find all products under $50";
+        var queryResult = await kernel.InvokePromptAsync(queryPrompt, new(settings));
+        Console.WriteLine($"Query Result: {queryResult}");
+
+        // Example 3: Updating records
+        Console.WriteLine("\nUpdating a record...");
+        var updatePrompt = "Update the price of 'Book' to 39.99 and keep its name";
+        var updateResult = await kernel.InvokePromptAsync(updatePrompt, new(settings));
+        Console.WriteLine($"Update Result: {updateResult}");
+
+        // Example 3: Updating records
+        Console.WriteLine("\nDeleting a record...");
+        var deletePrompt = "Delete the product 'Book'";
+        var deleteResult = await kernel.InvokePromptAsync(deletePrompt, new(settings));
+        Console.WriteLine($"Delete Result: {deleteResult}");
+
+        // Example 4: Interactive chat-like interaction
+        Console.WriteLine("\nStarting interactive mode (type 'exit' to quit)");
+        Console.WriteLine("You can try queries like:");
+        Console.WriteLine("- Find all products under $50");
+        Console.WriteLine("- Insert a new product with name 'Table' and price 19.99");
+        Console.WriteLine("- Update the price of 'Table' to 25.99");
+
+        while (true)
+        {
+            Console.Write("\nEnter your database query: ");
+            var userInput = Console.ReadLine();
+
+            if (string.IsNullOrEmpty(userInput) || userInput.Equals("exit", StringComparison.OrdinalIgnoreCase))
+            {
+                break;
+            }
+
+            var result = await kernel.InvokePromptAsync(userInput, new(settings));
+            Console.WriteLine($"\nResult: {result}");
+        }
+    }
+}