.Net: Bug: ResponseFormat using typeof() yields invalid schema

[brandonh-msft]

Describe the bug
I'm using ReponseFormat on a PromptExecutionSettings object. In my scenario I am setting it to typeof(Model[]) where Model is a record type with a simple structure, and I want to return an array of them as the LLM's response.

When the prompt executes, an error is thrown:

Microsoft.SemanticKernel.HttpOperationException: HTTP 400 (invalid_request_error: invalid_value) Parameter: response_format.json_schema.name  Invalid 'response_format.json_schema.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.  ---> System.ClientModel.ClientResultException: HTTP 400 (invalid_request_error: invalid_value) Parameter: response_format.json_schema.name  Invalid 'response_format.json_schema.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.    at Azure.AI.OpenAI.ClientPipelineExtensions.ProcessMessageAsync(
...

To Reproduce
Steps to reproduce the behavior:

1. Set ResponseFormat to a typeof() expression on an OpenAIPromptExecutionSettings object
2. Pass that settings obj as KernelArguments to an InvokePromptAsync() call

Expected behavior
The type should be schematized and sent along validly to the Azure OpenAI resource

Screenshots
If applicable, add screenshots to help explain your problem.

Platform

• Language: C#
• Source: Microsoft.SemanticKernel 1.36.1
• AI model: Azure OpenAI:GPT-4o
• IDE: Visual Studio
• OS: Windows

[dmytrostruk]

@brandonh-msft The reason why you are getting the error you mentioned is because type Model[] generates a name that is not acceptable by Azure/OpenAI API as JSON schema name. But even if we fix the name generation and send something like Collection_Model, it still won't work, because Azure/OpenAI API doesn't support collection as response type. You will receive following error:

schema must be a JSON Schema of 'type: "object"', got 'type: "array"

So, if you want a collection as response, you can wrap it in class/struct, like we did in following example:

semantic-kernel/dotnet/samples/Concepts/ChatCompletion/OpenAI_StructuredOutputs.cs

Lines 332 to 336 in 3f22587

	/// <summary>Movie result struct that will be used as desired chat completion response format (structured output).</summary>
	private struct MovieResult
	{
	public List<Movie> Movies { get; set; }
	}

And then set it as response format:

semantic-kernel/dotnet/samples/Concepts/ChatCompletion/OpenAI_StructuredOutputs.cs

Lines 211 to 215 in 3f22587

	// Specify response format by setting Type object in prompt execution settings.
	var executionSettings = new OpenAIPromptExecutionSettings
	{
	ResponseFormat = typeof(MovieResult)
	};

Please let me know if that will work for you, thanks!

[brandonh-msft]

yup that's what I ended up doing, thanks.

am hopeful we can avoid creating bad names, though, even if user does provide wrong types? just a thought.

[dmytrostruk]

am hopeful we can avoid creating bad names, though, even if user does provide wrong types? just a thought.

I think we can fix it at some point, in case if collections will be supported by Azure/OpenAI API in the future. As for now, I will deprioritize this task, since the case is not supported.