Mock music / non-speech audio generation (Gemini Lyria, ElevenLabs, fal)

[tombeckenham]

Feature request: mock music / audio-generation endpoints

Summary

aimock currently mocks audio only for OpenAI-style TTS at /v1/audio/speech (AudioResponse { audio: string; format?: string } served via speech.ts → handleSpeech). There's no mocking for music / non-speech audio generation, which blocks end-to-end testing of:

• Gemini Lyria music generation (POST /v1beta/models/{model}:generateContent with responseModalities: ['AUDIO'], returning audio bytes in a candidates[].content.parts[].inlineData part).
• Other provider "audio generation" shapes — ElevenLabs music & sound-effects, fal's audio catalog (stable-audio, audio-to-audio, sound-effects, voice-clone, etc.), which each return audio via provider-specific response shapes.

What breaks today

Gemini Lyria path in dist/gemini.js:

1. geminiToCompletionRequest translates any generateContent call — including AUDIO-modality — to an OpenAI-style completion request. It drops responseModalities and only routes to text / tool-call fixtures.
2. The existing AudioResponse shape (audio: string) is wired to handleSpeech (OpenAI /v1/audio/speech). Even if a Gemini-bound fixture declared audio: "<base64>", there's no code path that emits it back as Gemini's inlineData part on generateContent.

Reproducer:

// fixture
{
  "match": { "userMessage": "[audiogen] upbeat piano loop" },
  "response": {
    "audio": { "b64Json": "SGVsbG8gYXVkaW8=", "contentType": "audio/mp3" }
  }
}

// adapter call (Google GenAI SDK pointed at aimock)
await client.models.generateContent({
  model: 'lyria-3-clip-preview',
  contents: [{ role: 'user', parts: [{ text: '[audiogen] upbeat piano loop' }] }],
  config: { responseModalities: ['AUDIO'], responseMimeType: 'audio/mp3' },
})

Today this either returns a text completion (if a text fixture matches) or a "no fixture matched" error — never an audio inlineData part.

Proposal

1. Broaden the fixture AudioResponse (or add a sibling MusicResponse / GeneratedAudioResponse) to cover non-speech audio generation:

   interface AudioResponse {
     audio: string | { b64Json: string; contentType?: string }
     format?: string
   }

   Keeping the string form for back-compat with /v1/audio/speech, the object form for endpoints whose wire shape needs a MIME type (Gemini Lyria, most fal audio models).

2. Route non-speech audio in gemini.ts: when the incoming generateContent request carries responseModalities: ['AUDIO'] (or similar), match against the fixture and emit a synthetic response:

   {
     "candidates": [{
       "content": {
         "role": "model",
         "parts": [{
           "inlineData": { "mimeType": "audio/mp3", "data": "<base64>" }
         }]
       },
       "finishReason": "STOP"
     }]
   }

3. Provider coverage (follow-ups, nice to have): ElevenLabs music/SFX endpoints (/v1/music/compose, /v1/sound-generation) and a generic fal queue/subscribe handler for audio output, each mapping an audio fixture to the provider's native response shape.

Context / use case

We hit this in TanStack/ai #328 / PR #463, adding a single generateAudio activity covering music, sound effects, audio-to-audio, voice-change, voice-clone, and the rest of fal's catalog. Unit tests are in place, but the E2E audio-gen feature currently ships with an empty feature-support set — we can't exercise it end-to-end against aimock 1.14.

Happy to contribute a PR if the proposed AudioResponse shape and the gemini routing hook look right to you.

[jpr5]

Howdy @tombeckenham, can you give this PR a test drive and tell me how it works for you? 🙏

[tombeckenham]

Just created tests with this for the elevenlabs adapter. It's looking good!

[jpr5]

@tombeckenham Final answer?

[tombeckenham]

Yes. Looks good