Skip to content

Commit fb41835

Browse files
committed
Add ISpeechToTextClient support via GrokSpeechToTextClient
Implements ISpeechToTextClient for xAI's Grok models, including: - GrokSpeechToTextClient: unary transcription via POST /v1/stt and streaming transcription via wss://.../v1/stt WebSocket protocol. Handles session handshake, chunked audio upload, interim/final transcript events, word-level timing, and language detection. - GrokSpeechToTextOptions: Grok-specific options for audio format, sample rate, multichannel, diarization, interim results, and endpointing timeout. - AsISpeechToTextClient() extension on GrokClient wires up the client with the correct HTTP handler and WebSocket factory. Fix: voice REST clients (TTS and STT) were accidentally reusing the gRPC channel's BalancerHttpHandler, which throws for plain HTTP/1.1 requests. Added GrokClient.HttpHandler, backed by a separate httpHandlers cache using the same Polly retry pipeline but independent of the gRPC channel. AsITextToSpeechClient and AsISpeechToTextClient now use client.HttpHandler instead of client.ChannelHandler.Handler. The channels dictionary now holds ChannelBase directly rather than a tuple, since the HttpMessageHandler is no longer needed from it. Add TextToSpeech_SpeechToText integration test that streams TTS audio to a temp file and transcribes it back with STT, asserting the roundtrip text matches (punctuation-insensitive via NormalizeTranscription). Update readme with ISpeechToTextClient usage examples alongside the existing TTS documentation.
1 parent 24ab0f6 commit fb41835

10 files changed

Lines changed: 1144 additions & 93 deletions

AGENTS.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
# xAI SDK implementation notes
22

3-
- `GrokClient` is primarily backed by generated gRPC protocol clients, but text to speech uses xAI's documented REST/WebSocket voice endpoints because there are no generated TTS protocol types in `src\xAI.Protocol`.
3+
- `GrokClient` is primarily backed by generated gRPC protocol clients, but voice features use xAI's documented REST/WebSocket endpoints because there are no generated voice protocol types in `src\xAI.Protocol`.
4+
- Voice REST calls use `GrokClient.HttpHandler` (backed by `httpHandlers` cache) — a plain `SocketsHttpHandler`+Polly pipeline separate from the gRPC channel. `ChannelHandler` returns `ChannelBase` only; there is no `.Handler` property on it.
45
- `AsITextToSpeechClient` returns an `ITextToSpeechClient` implementation that uses `POST /v1/tts` for unary audio and `wss://.../v1/tts` for streaming audio.
6+
- `AsISpeechToTextClient` returns an `ISpeechToTextClient` implementation that uses `POST /v1/stt` for file transcription and `wss://.../v1/stt` for raw-audio streaming transcription.
57
- TTS defaults follow xAI docs: voice `eve`, language `en` when omitted by `TextToSpeechOptions`, and MP3 output when no codec is specified.
8+
- STT streaming defaults follow xAI docs: encoding `pcm` and sample rate `16000` when omitted; WebSocket input must be raw encoded audio, not MP3/WAV container bytes.

readme.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,12 @@ var speech = new GrokClient(Environment.GetEnvironmentVariable("XAI_API_KEY")!)
5151

5252
var audio = await speech.GetAudioAsync("Hello! Welcome to xAI text to speech.",
5353
new TextToSpeechOptions { VoiceId = "eve", Language = "en" });
54+
55+
var transcription = new GrokClient(Environment.GetEnvironmentVariable("XAI_API_KEY")!)
56+
.AsISpeechToTextClient();
57+
58+
var text = await transcription.GetTextAsync(File.OpenRead("audio.mp3"),
59+
new SpeechToTextOptions { TextLanguage = "en" });
5460
```
5561

5662
## File Attachments
@@ -402,6 +408,8 @@ Console.WriteLine($"Edited image URL: {editedImage.Uri}");
402408
## Text to Speech
403409

404410
Grok supports text to speech via the `ITextToSpeechClient` abstraction from Microsoft.Extensions.AI.
411+
See the [xAI text to speech docs](https://docs.x.ai/developers/model-capabilities/audio/text-to-speech)
412+
for supported voices, formats, and streaming details.
405413
Use `AsITextToSpeechClient` to get a TTS client:
406414

407415
```csharp
@@ -465,6 +473,87 @@ var options = new GrokTextToSpeechOptions
465473
var response = await speech.GetAudioAsync("Streaming at 24 kHz, 128 kbps.", options);
466474
```
467475

476+
## Speech to Text
477+
478+
Grok supports speech to text via the `ISpeechToTextClient` abstraction from Microsoft.Extensions.AI.
479+
See the [xAI speech to text docs](https://docs.x.ai/developers/model-capabilities/audio/speech-to-text)
480+
for supported languages, audio formats, diarization, multichannel audio, and streaming details.
481+
Use `AsISpeechToTextClient` to get an STT client:
482+
483+
```csharp
484+
var transcription = new GrokClient(Environment.GetEnvironmentVariable("XAI_API_KEY")!)
485+
.AsISpeechToTextClient();
486+
```
487+
488+
### Unary (single response)
489+
490+
Call `GetTextAsync` to transcribe an audio file in a single request. The result contains transcript
491+
text, timing information, and the raw xAI response:
492+
493+
```csharp
494+
await using var audio = File.OpenRead("meeting.mp3");
495+
496+
var response = await transcription.GetTextAsync(audio,
497+
new GrokSpeechToTextOptions
498+
{
499+
TextLanguage = "en",
500+
Format = true,
501+
});
502+
503+
Console.WriteLine(response.Text);
504+
```
505+
506+
Set `Format = true` with `TextLanguage` to enable xAI's inverse text normalization, such as converting
507+
spoken numbers and currencies into written form.
508+
509+
### Streaming
510+
511+
Call `GetStreamingTextAsync` to stream raw audio and receive transcript updates as speech is processed.
512+
The xAI streaming endpoint expects raw encoded audio such as PCM, µ-law, or A-law rather than MP3/WAV
513+
container bytes:
514+
515+
```csharp
516+
await using var audio = File.OpenRead("audio.pcm");
517+
518+
await foreach (var update in transcription.GetStreamingTextAsync(audio,
519+
new GrokSpeechToTextOptions
520+
{
521+
AudioFormat = "pcm",
522+
SpeechSampleRate = 16000,
523+
TextLanguage = "en",
524+
InterimResults = true,
525+
}))
526+
{
527+
if (update.Kind is SpeechToTextResponseUpdateKind.TextUpdating or
528+
SpeechToTextResponseUpdateKind.TextUpdated)
529+
{
530+
Console.WriteLine(update.Text);
531+
}
532+
}
533+
```
534+
535+
### Grok-Specific Options
536+
537+
Use `GrokSpeechToTextOptions` to control xAI transcription behavior beyond the base
538+
`SpeechToTextOptions`:
539+
540+
```csharp
541+
var options = new GrokSpeechToTextOptions
542+
{
543+
TextLanguage = "en",
544+
SpeechSampleRate = 16000,
545+
Format = true, // normalize spoken numbers, currencies, and units
546+
AudioFormat = "pcm", // pcm | mulaw | alaw for raw audio
547+
Diarize = true, // include speaker IDs on words when available
548+
Multichannel = true, // transcribe each channel independently
549+
Channels = 2,
550+
InterimResults = true, // streaming only
551+
Endpointing = 10, // streaming silence duration in milliseconds
552+
};
553+
554+
var response = await transcription.GetTextAsync(File.OpenRead("call.pcm"), options);
555+
```
556+
468557
<!-- #xai -->
469558

470559
# xAI.Protocol

src/xAI.Tests/SanityChecks.cs

Lines changed: 77 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,8 @@
11
using System.Text.Json;
2-
using Devlooped.Extensions.AI;
3-
using DotNetEnv;
4-
using Grpc.Core;
5-
using Grpc.Net.Client.Configuration;
62
using Microsoft.Extensions.AI;
73
using Microsoft.Extensions.DependencyInjection;
84
using xAI.Protocol;
9-
using Xunit.Abstractions;
10-
using Xunit.Sdk;
5+
using static ConfigurationExtensions;
116
using ChatConversation = Devlooped.Extensions.AI.Chat;
127

138
namespace xAI.Tests;
@@ -18,7 +13,7 @@ public class SanityChecks(ITestOutputHelper output)
1813
public async Task NoEmbeddingModels()
1914
{
2015
var services = new ServiceCollection()
21-
.AddxAIProtocol(Environment.GetEnvironmentVariable("CI_XAI_API_KEY")!)
16+
.AddxAIProtocol(Configuration["CI_XAI_API_KEY"]!)
2217
.BuildServiceProvider();
2318

2419
var client = services.GetRequiredService<Models.ModelsClient>();
@@ -33,7 +28,7 @@ public async Task NoEmbeddingModels()
3328
public async Task ListModelsAsync()
3429
{
3530
var services = new ServiceCollection()
36-
.AddxAIProtocol(Environment.GetEnvironmentVariable("CI_XAI_API_KEY")!)
31+
.AddxAIProtocol(Configuration["CI_XAI_API_KEY"]!)
3732
.BuildServiceProvider();
3833

3934
var client = services.GetRequiredService<Models.ModelsClient>();
@@ -50,7 +45,7 @@ public async Task ListModelsAsync()
5045
public async Task ExecuteLocalFunctionWithWebSearch()
5146
{
5247
var services = new ServiceCollection()
53-
.AddxAIProtocol(Environment.GetEnvironmentVariable("CI_XAI_API_KEY")!)
48+
.AddxAIProtocol(Configuration["CI_XAI_API_KEY"]!)
5449
.BuildServiceProvider();
5550

5651
var client = services.GetRequiredService<xAI.Protocol.Chat.ChatClient>();
@@ -161,7 +156,7 @@ public async Task ExecuteLocalFunctionWithWebSearch()
161156
public async Task ClientSideFunction(bool streaming)
162157
{
163158
var getDateCalls = 0;
164-
var grok = new GrokClient(Env.GetString("CI_XAI_API_KEY")!)
159+
var grok = new GrokClient(Configuration["CI_XAI_API_KEY"]!)
165160
.AsIChatClient("grok-4-1-fast")
166161
.AsBuilder()
167162
.UseFunctionInvocation()
@@ -203,7 +198,7 @@ What is today's date? Use the get_date tool.
203198
[InlineData(true)]
204199
public async Task AgenticWebSearch(bool streaming)
205200
{
206-
var grok = new GrokClient(Env.GetString("CI_XAI_API_KEY")!)
201+
var grok = new GrokClient(Configuration["CI_XAI_API_KEY"]!)
207202
.AsIChatClient("grok-4-1-fast");
208203

209204
var options = new GrokChatOptions
@@ -249,7 +244,7 @@ What is the current price of Tesla (TSLA) stock? Use web search (Yahoo Finance o
249244
[InlineData(true)]
250245
public async Task AgenticXSearch(bool streaming)
251246
{
252-
var grok = new GrokClient(Env.GetString("CI_XAI_API_KEY")!)
247+
var grok = new GrokClient(Configuration["CI_XAI_API_KEY"]!)
253248
.AsIChatClient("grok-4-1-fast");
254249

255250
var options = new GrokChatOptions
@@ -288,7 +283,7 @@ What is the top news from Tesla on X? Use the X search tool.
288283
[InlineData(true)]
289284
public async Task AgenticMcpServer(bool streaming)
290285
{
291-
var grok = new GrokClient(Env.GetString("CI_XAI_API_KEY")!)
286+
var grok = new GrokClient(Configuration["CI_XAI_API_KEY"]!)
292287
.AsIChatClient("grok-4-1-fast");
293288

294289
var options = new GrokChatOptions
@@ -299,7 +294,7 @@ public async Task AgenticMcpServer(bool streaming)
299294
[
300295
new HostedMcpServerTool("GitHub", "https://api.githubcopilot.com/mcp/")
301296
{
302-
Headers = new Dictionary < string, string > {["Authorization"] = Env.GetString("GITHUB_TOKEN") ! },
297+
Headers = new Dictionary < string, string > {["Authorization"] = Configuration["GITHUB_TOKEN"] ! },
303298
AllowedTools = ["list_releases", "get_release_by_tag"],
304299
}
305300
]
@@ -340,7 +335,7 @@ What is the latest release version of the {{ThisAssembly.Git.Url}} repository? U
340335
[InlineData(true)]
341336
public async Task AgenticFileSearch(bool streaming)
342337
{
343-
var grok = new GrokClient(Env.GetString("CI_XAI_API_KEY")!)
338+
var grok = new GrokClient(Configuration["CI_XAI_API_KEY"]!)
344339
.AsIChatClient("grok-4-1-fast");
345340

346341
var options = new GrokChatOptions
@@ -406,7 +401,7 @@ Use the collection search tool.
406401
[InlineData(true)]
407402
public async Task AgenticCodeInterpreter(bool streaming)
408403
{
409-
var client = new GrokClient(Env.GetString("CI_XAI_API_KEY")!);
404+
var client = new GrokClient(Configuration["CI_XAI_API_KEY"]!);
410405

411406
var grok = client.AsIChatClient("grok-4-1-fast");
412407

@@ -451,6 +446,72 @@ parseable by a decimal parser.
451446
output.WriteLine($"Code interpreter calls: {codeInterpreterCalls.Count}");
452447
}
453448

449+
[SecretsTheory("CI_XAI_API_KEY")]
450+
[InlineData("rex")]
451+
public async Task TextToSpeech_SpeechToText(string voiceId)
452+
{
453+
using var client = new GrokClient(Configuration["CI_XAI_API_KEY"]!);
454+
using var tts = client.AsITextToSpeechClient();
455+
456+
var expected = "El que cree en mí, en realidad no cree en mí, sino en aquel que me envió.";
457+
var tempFile = System.IO.Path.Combine(System.IO.Path.GetTempPath(), $"xai-tts-{Guid.NewGuid():N}.pcm");
458+
459+
try
460+
{
461+
await using (var fileStream = System.IO.File.Create(tempFile))
462+
{
463+
await foreach (var update in tts.GetStreamingAudioAsync(
464+
expected,
465+
new TextToSpeechOptions
466+
{
467+
VoiceId = voiceId,
468+
Language = "es-ES",
469+
// uses mp3 by default
470+
}))
471+
{
472+
if (update.Kind == TextToSpeechResponseUpdateKind.AudioUpdating)
473+
{
474+
foreach (var content in update.Contents)
475+
{
476+
if (content is DataContent data)
477+
{
478+
await fileStream.WriteAsync(data.Data);
479+
}
480+
}
481+
}
482+
}
483+
}
484+
485+
Assert.True(System.IO.File.Exists(tempFile));
486+
Assert.True(new System.IO.FileInfo(tempFile).Length > 0);
487+
488+
using var stt = client.AsISpeechToTextClient();
489+
await using var audioStream = System.IO.File.OpenRead(tempFile);
490+
491+
// auto-detect format from content
492+
var transcription = await stt.GetTextAsync(audioStream);
493+
494+
Assert.Equal(
495+
NormalizeTranscription(expected),
496+
NormalizeTranscription(transcription.Text),
497+
ignoreCase: true);
498+
}
499+
finally
500+
{
501+
if (System.IO.File.Exists(tempFile))
502+
System.IO.File.Delete(tempFile);
503+
}
504+
}
505+
506+
static string NormalizeTranscription(string? text)
507+
{
508+
var withoutPunctuation = new string((text ?? string.Empty)
509+
.Select(character => char.IsPunctuation(character) ? ' ' : character)
510+
.ToArray());
511+
512+
return string.Join(" ", withoutPunctuation.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries));
513+
}
514+
454515
static async Task<ChatResponse> GetResponseAsync(IChatClient client, ChatConversation chat, GrokChatOptions options, bool streaming)
455516
{
456517
if (!streaming)

0 commit comments

Comments
 (0)