feat: audio attachment support for audio-capable models (Gemini)

[wolframs]

What

Routes Discord audio/* attachments to audio-capable models, mirroring the existing image pipeline. Gated behind a new per-bot config — include_audio (default off) and max_audio (default 1) — so audio only ever flows to a model the operator has explicitly opted in. The intended target is Gemini 3.5 Flash (trained on music). Not all models accept audio — and music understanding is narrower still — so this is a deliberate per-bot opt-in, never auto-routed.

How

• Ingestion (connector.ts): detect audio/* attachments by content_type or filename extension (Discord's content_type is optional), fetch + in-memory cache, reject oversized uploads before downloading (MAX_AUDIO_BYTES, 12 MB), normalize MIME to a Gemini-accepted type, capture voice-message duration.
• Types: AudioContent block, CachedAudio, DiscordContext.audios, BotConfig.include_audio/max_audio.
• Selection (format-messages.ts): emit AudioContent blocks gated by include_audio, bounded by a combined images+audio inline budget (~18 MB, under Gemini's ~20 MB inline-data ceiling).
• Adapter: AudioContent → membrane audio block.
• Logging (provider.ts): base64 audio is redacted from request logs, same as images.
• Fetch gating (loop.ts): only audio-capable bots fetch audio (maxAudio threading; max_audio: 0 disables).
• Config docs: config/bots/claude.yaml.example documents the new options.

Dependency

Requires Gemini audio support in the membrane middleware — antra-tess/membrane#29. Until this repo's @animalabs/membrane dep is bumped to a release containing that, audio blocks are dropped harmlessly by the provider (and the feature is off by default regardless).

Verification

• End-to-end against the live Gemini API via the membrane change (tested with gemini-2.5-flash; the path is model-agnostic and targets Gemini 3.5 Flash): a two-tone test clip was correctly described.
• Unit coverage for the adapter audio conversion; full suite green (307 tests), tsc clean.
• Two independent codex review passes addressed: audio log redaction, combined media budget, pre-fetch size cap, max_audio: 0 consistency, and an extension-fallback emission gap.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR adds audio attachment support for audio-capable models (Gemini), mirroring the existing image pipeline. Audio is off by default and must be explicitly opted in per-bot via include_audio: true / max_audio, preventing accidental routing to models that cannot handle audio.

• Ingestion (connector.ts): detects audio/* attachments by content_type or filename extension, pre-checks attachment size before downloading (12 MB cap), normalizes MIME to Gemini-accepted types, and caches audio in-memory per session.
• Emission (format-messages.ts): totalBase64Size is promoted to function scope so images and audio share a combined 18 MB inline-data budget, with audio gated by include_audio and bounded by max_audio.
• Adapter + logging (adapter.ts, provider.ts): adds the audio → membrane block conversion and extends the base64-redaction log path to cover audio alongside images.

Confidence Score: 4/5

Safe to merge — audio is off by default and the new paths are well-gated. The only rough edge is the in-memory audio cache having no eviction policy, which is a concern for long-running bots but not an immediate breakage.

The feature is cleanly opt-in, the combined inline-data budget correctly accounts for both images and audio, and base64 redaction in logs was extended to cover the new media type. The audioCache grows without bound over a session's lifetime — each unique audio URL fetched stays in the heap at up to 12 MB raw — but since max_audio defaults to 1 and hits on the same URL are deduplicated, the practical growth rate is low for typical usage. The profiling log also omits the new audioCount field, which is a minor observability gap.

src/discord/connector.ts — audio cache lifetime and the profiling log omission are both here.

Important Files Changed

Filename	Overview
src/discord/connector.ts	Adds audio detection, pre-fetch size cap, MIME normalization, and in-memory caching for audio attachments. The audioCache has no eviction policy (mirrors imageCache). Profiling log omits audioCount.
src/context/stages/format-messages.ts	Promotes totalBase64Size to function scope so the combined images+audio budget (18 MB) can be enforced together. Audio emission is gated behind include_audio, bounded by max_audio, and checked against the combined ceiling. Logic is sound.
src/llm/membrane/adapter.ts	Adds audio case to toMembraneContentBlock, renaming media_type → mediaType and passing through duration. Simple and correct.
src/llm/membrane/provider.ts	Extends the base64-redaction log path to cover audio blocks alongside image, preventing large audio payloads from appearing in logs.
src/agent/loop.ts	Adds maxAudioFetch computation that gates audio fetching behind include_audio; max_audio ?? 1 handles the default correctly and max_audio: 0 disables audio consistently.
src/types.ts	Adds AudioContent, CachedAudio, and BotConfig audio config fields. Straightforward type additions with no issues.
src/llm/membrane/adapter.test.ts	Adds unit tests for audio block conversion (media_type → mediaType rename and duration passthrough). Coverage is appropriate for the new adapter case.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Discord
    participant Connector as connector.ts
    participant AudioCache as audioCache (in-memory)
    participant Builder as context/builder.ts
    participant Format as format-messages.ts
    participant Adapter as membrane/adapter.ts
    participant Gemini

    Discord->>Connector: Message with audio attachment
    Connector->>Connector: isAudioAttachment() — content_type or extension
    Connector->>Connector: "Pre-fetch size check (attachment.size > 12 MB → skip)"
    Connector->>Connector: audioMimeFor() — normalize MIME
    Connector->>AudioCache: cacheAudio(url, mediaType, messageId, duration)
    AudioCache->>Connector: CachedAudio (Buffer + metadata)
    Connector->>Builder: "DiscordContext { audios: CachedAudio[] }"
    Builder->>Format: formatMessages(..., audios)
    Format->>Format: Build audioMap (url → CachedAudio)
    Format->>Format: "Check include_audio + audioEmitted < maxAudio"
    Format->>Format: Check combined inline budget (images + audio ≤ 18 MB)
    Format->>Adapter: AudioContent block (base64 + media_type)
    Adapter->>Gemini: "{ type: audio, source: { type: base64, mediaType, data } }"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Discord
    participant Connector as connector.ts
    participant AudioCache as audioCache (in-memory)
    participant Builder as context/builder.ts
    participant Format as format-messages.ts
    participant Adapter as membrane/adapter.ts
    participant Gemini

    Discord->>Connector: Message with audio attachment
    Connector->>Connector: isAudioAttachment() — content_type or extension
    Connector->>Connector: "Pre-fetch size check (attachment.size > 12 MB → skip)"
    Connector->>Connector: audioMimeFor() — normalize MIME
    Connector->>AudioCache: cacheAudio(url, mediaType, messageId, duration)
    AudioCache->>Connector: CachedAudio (Buffer + metadata)
    Connector->>Builder: "DiscordContext { audios: CachedAudio[] }"
    Builder->>Format: formatMessages(..., audios)
    Format->>Format: Build audioMap (url → CachedAudio)
    Format->>Format: "Check include_audio + audioEmitted < maxAudio"
    Format->>Format: Check combined inline budget (images + audio ≤ 18 MB)
    Format->>Adapter: AudioContent block (base64 + media_type)
    Adapter->>Gemini: "{ type: audio, source: { type: base64, mediaType, data } }"

Loading

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
src/discord/connector.ts:111
**Unbounded in-memory audio cache**

`audioCache` is a `Map` that grows without eviction for the life of the connector instance. Unlike `imageCache`, which has a disk persistence layer and URL map as a backing store, audio lives purely in RAM. Each entry can be up to 12 MB; a busy bot processing many distinct audio URLs across a long session can accumulate significant heap pressure. The image cache has the same pattern, but images are typically re-encountered across turns (cache hits) whereas audio files sent once are rarely re-sent, meaning the hit rate for audio may be lower and the unique-URL accumulation rate higher.

### Issue 2 of 2
src/discord/connector.ts:907-914
**Missing `audioCount` in the profiling and attachment-completion logs**

The `⏱️ PROFILING` log records `imageCount` and `documentCount` but not `audioCount`. When audio is silently dropped or not fetched, this makes the log entry misleading — a bot with `include_audio: true` shows no sign in the profiling line of whether audio was actually collected.

```suggestion
      // Log fetch timings
        logger.info({
        ...timings,
        messageCount: discordMessages.length,
        imageCount: images.length,
        documentCount: documents.length,
        audioCount: audios.length,
        pinnedCount: pinnedConfigs.length,
      }, '⏱️  PROFILING: fetchContext breakdown (ms)')
```

_{Reviews (1): Last reviewed commit: "fix(audio): emit extension-detected audi..." | Re-trigger Greptile}

[greptile-apps]

Unbounded in-memory audio cache

audioCache is a Map that grows without eviction for the life of the connector instance. Unlike imageCache, which has a disk persistence layer and URL map as a backing store, audio lives purely in RAM. Each entry can be up to 12 MB; a busy bot processing many distinct audio URLs across a long session can accumulate significant heap pressure. The image cache has the same pattern, but images are typically re-encountered across turns (cache hits) whereas audio files sent once are rarely re-sent, meaning the hit rate for audio may be lower and the unique-URL accumulation rate higher.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/discord/connector.ts
Line: 111

Comment:
**Unbounded in-memory audio cache**

`audioCache` is a `Map` that grows without eviction for the life of the connector instance. Unlike `imageCache`, which has a disk persistence layer and URL map as a backing store, audio lives purely in RAM. Each entry can be up to 12 MB; a busy bot processing many distinct audio URLs across a long session can accumulate significant heap pressure. The image cache has the same pattern, but images are typically re-encountered across turns (cache hits) whereas audio files sent once are rarely re-sent, meaning the hit rate for audio may be lower and the unique-URL accumulation rate higher.

How can I resolve this? If you propose a fix, please make it concise.

[wolframs]

Thanks Greptile — both addressed in 5053bb8:

1. Unbounded audio cache → audioCache now has an LRU byte budget (AUDIO_CACHE_MAX_BYTES, 64 MB): cache hits refresh recency, and evictAudioCache() drops oldest entries once over budget. This keeps the cross-activation dedup benefit for audio still in the rolling context window while bounding heap. (Note: imageCache shares the original unbounded pattern; left as-is to keep this PR scoped, but it's a reasonable follow-up.)
2. Missing audioCount → added to both the ⏱️ PROFILING and "Attachment processing complete" logs.

Good catches — the audio cache distinction (lower hit rate, no disk backing, larger entries) was a fair call.

[wolframs]

✅ Acceptance-tested (2026-07-04)

Verified live on a Discord server, together with antra-tess/membrane#29:

• Gemini 3.5 Flash (gemini-3.5-flash, native Gemini vendor, inlineData path): described an attached mp3 in detail — genre, instrumentation, vocals, lyric themes, per-timestamp structure.
• MiMo V2.5 (xiaomi/mimo-v2.5 via OpenRouter, input_audio path, provider-pinned to Parasail/Venice/DeepInfra): equally detailed track breakdown.

Found during testing and fixed in 31915fd: Discord reports mp3 attachments as the legacy MIME alias audio/mpeg3 in the wild — now normalized to audio/mp3.

Gemini 3.5 Flash:

MiMo V2.5 (OpenRouter):

🤖 Generated with Claude Code

[wolframs]

Follow-up: size-cap UX + payload sizing docs (fd447a6)

• Oversized audio attachments (> 12 MB per-file cap) were skipped with only a server-side log line, so users got a misleading text-only reply. The bot now reacts to the message with a configurable emote (default 🐘, oversized_audio_emote: '' disables), mirroring the existing 🚫 blocked-message reaction; deduped per message so rolling-window rescans don't re-react.
• claude.yaml.example now documents payload sizing: the per-file cap × max_audio all ships base64-inlined in one request, so max_audio: 1 is the recommended default while the per-file cap sits at ~12–15 MB; raising max_audio means dividing the ~15 MB raw-audio budget across files.

🤖 Generated with Claude Code