fix(bidi): emit gemini usage metadata alongside content events

[mehtarac]

Description

Gemini's Live API sends token usage in a usageMetadata field that sits outside the messageType union. Per the Gemini Live Api reference:

"Server messages may have a usageMetadata field but will otherwise include exactly one of the other fields." So usage can accompany audio, a tool call, or aresumption update.

BidiGeminiLiveModel._convert_gemini_live_event's implementation was such that any message with both content and usage lost the usage. The loss is silent and cumulative:
_BidiAgentLoop._run_model accumulates session token counters solely from BidiUsageEvent, so every dropped event undercounts the session with no error. This restructures the converter to accumulate into a list. Control messages stay exclusive content messages append, and usage is evaluated unconditionally at the end.

The list-accumulating shape also unblocks emitting multiple events per message, needed for the missing BidiResponseStartEvent / BidiResponseCompleteEvent for google gemini live model provider change left to a follow-up.

Related Issues

Documentation PR

Type of Change

Bug fix

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
• My change is focused and reasonably small; I have split unrelated work into separate PRs
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

Issue: The switch from early-return to list accumulation changes emission from exclusive (one event per message) to cumulative. That's the right fix for usageMetadata, but it also changes content semantics: an interrupted message now co-emits with any output_transcription/model_turn text in the same server_content, whereas before the interruption short-circuited everything.

Suggestion: This looks intentional (and test_interruption_emitted_alongside_other_server_content locks it in), so no change needed if it is — just confirming it's deliberate that partial model text is surfaced after an interruption rather than dropped. Worth a one-line note in the PR description since it's a subtle behavior change beyond the usage fix.

[mehtarac]

Yes, intentional. Per the Live API reference, interrupted, outputTranscription, and modelTurn are all fields on the same BidiGenerateContentServerContent object, not mutually exclusive message types — the old early-return dropped whichever fields came after the first match.

[github-actions]

Issue: This test only asserts modality_details and cache_read_input_tokens, but the fixture also populates input_tokens/output_tokens/total_tokens (10/20/30). Since those fields aren't asserted, a regression that corrupts the token counts while this message shape is in play would pass silently.

Suggestion: BidiUsageEvent is a dict subclass, so you can assert the whole event in one equality and cover every field at once:

assert events == [
    BidiUsageEvent(
        input_tokens=10,
        output_tokens=20,
        total_tokens=30,
        modality_details=[{"modality": "audio", "input_tokens": 7, "output_tokens": 9}],
        cache_read_input_tokens=4,
    )
]

The same pattern applies to test_usage_metadata_emitted_alongside_audio (lines 591–593), which asserts the three token fields individually.

[github-actions]

Confirmed resolved — both test_usage_metadata_modality_details and test_usage_metadata_emitted_alongside_audio now assert the full BidiUsageEvent in a single equality, so every field (including the token counts) is covered. Thanks for tightening these. 👍

[github-actions]

Assessment: Comment (approve-leaning)

Solid, focused bug fix. Restructuring _convert_gemini_live_event from exclusive early-returns to list accumulation is the right shape for the problem, and extracting _convert_server_content / _convert_usage_metadata genuinely improves readability. Verified locally: all 27 tests pass, and the two ruff/mypy findings in this file are pre-existing on main, not introduced here.

Review notes

• Correctness: Fix correctly stops usageMetadata from being silently dropped when it rides along with content; the audio-vs-model_turn-text precedence is preserved via has_audio and covered by a test.
• Testing: Good use of shape assertions ([type(e) for e in events]) and clean fixtures. One gap — a couple of usage tests assert token fields individually and leave others unchecked; consolidating to a full BidiUsageEvent equality would catch regressions in the unasserted fields (inline comment).
• Behavior change: Exclusive → cumulative emission is a subtle semantic shift beyond the usage fix (e.g. interruption now co-emits with leftover content); flagged inline to confirm intent / note in the description.
• Minor: _convert_server_content(..., has_audio=...) couples the helper to the parent message's data field. It's documented, so acceptable, but a small standalone predicate or a comment at the call site would make the coupling more obvious.

Nice cleanup overall — the new structure also clearly sets up the follow-up for the missing response start/complete events.

[mehtarac]

@strandly-the-agent do a fresh review of this PR

[github-actions]

Assessment: Approve

Fresh review of the address comments commit — all prior feedback is resolved and the PR looks good to merge.

Verification

• Testing feedback addressed: Both usage tests now assert the full BidiUsageEvent in a single equality instead of individual fields, closing the silent-regression gap. The new issue references in the test docstrings are a nice touch for traceability.
• Behavior change confirmed: Agreed with your reasoning — interrupted, outputTranscription, and modelTurn are non-exclusive fields on the same server_content object, so accumulating them is the correct semantics and the old early-return was the bug.
• Tests pass: All 27 tests pass locally against this branch's source.
• No new lint: The ruff/mypy findings in this file remain pre-existing on main, not introduced here.

Clean, well-tested fix that also sets up the follow-up for the missing response start/complete events. 🚀

[strandly-the-agent]

Fresh review at 3bfda5ec — verdict: comment. No blocker. The usage fix is right, and the restructure quietly recovers two other silent drops (an interrupted turn's inputTranscription, and audio riding with an outputTranscription). Two new co-emission orderings deserve a decision, and one premise deserves confirming.

Verified on this branch (real google.genai.types, not mocks; differential vs base 11ad6366):

• ✅ 27 passed — python -m pytest tests/strands/experimental/bidi/models/test_gemini_live.py -q (google-genai 2.17.0)
• ✅ the fix works: serverContent(audio) + usageMetadata → ['BidiAudioStreamEvent', 'BidiUsageEvent'] at head vs ['BidiAudioStreamEvent'] on base
• ✅ floor is safe: LiveServerContent/UsageMetadata and every field read exist at google-genai==1.32.0
• ✅ shape matches repo convention (openai_realtime.py:562-609 already accumulates a list with usage appended last)
• 🟡 2 new behaviors reproduced end-to-end through the real _BidiAudioOutput / _BidiAgentLoop (inline)
• 🟡 mutation check: emitting audio before the server_content events still passes all 27 tests → event order is unpinned

❓ Question (worth settling before merge)

Is Gemini Live's usageMetadata a per-message delta or a running total? loop.py:340-344 sums every BidiUsageEvent. If it's cumulative, then emitting more usage events — precisely what this PR does — would make session accounting overcount rather than fix an undercount, inverting the premise. I couldn't settle it: the Live API reference documents the fields but never says which; the official capabilities-guide example doesn't sum, it prints each message standalone as "Used {usage.total_token_count} tokens in total" under the comment "the server will periodically send messages that include UsageMetadata"; and several developer-forum reports describe promptTokenCount as cumulative/growing per turn. One captured live session settles it — do you have one handy? Happy to be wrong here.

Coverage gaps (non-blocking) — 3

• No test pairs usage_metadata with a tool_call or with a transcript, though the new docstring's own claim is that usage "can accompany any other field" — and usage landing with a tool-call message is arguably more common than with a session-resumption update.
• test_usage_metadata_modality_details uses AUDIO for both prompt and response details, so only the merge branch (gemini_live.py:369-370) runs; the new-modality append branch (:372-374, e.g. audio-in/text-out) is never exercised.
• if detail.modality and detail.token_count: (:354, :365) silently drops a legitimately reported token_count == 0. Intentional?

Appendix — non-blocking (5)

Two pre-existing bugs, verified byte-identical on base — they belong in issues, not this PR. I tried to file them and my token lacks issues:write here, so they're written up below for someone with permissions (or say the word and I'll draft them as an issue body you can paste).

1. ⚪ modality_details emits "mediamodality.audio", violating its own type contract. gemini_live.py:357,367 do str(detail.modality).lower(); on the wire detail.modality is a MediaModality enum with no __str__ override, so:

   >>> str(t.MediaModality.AUDIO).lower()   -> 'mediamodality.audio'
   >>> t.MediaModality.AUDIO.value.lower()  -> 'audio'
   

   Real-types run against this branch emits {'modality': 'mediamodality.audio', 'input_tokens': 7, 'output_tokens': 0}, but ModalityUsage.modality is Literal["text","audio","image","cached"] (types/events.py:460) — any consumer branching on it silently fails to match. Same at floor 1.32.0 and at 2.17.0. Fix is detail.modality.value.lower(); openai_realtime.py:608 may share the pattern.

2. ⚪ part.thought isn't filtered, so model reasoning can leak into transcripts and history. gemini_live.py:326 takes every part with .text; google-genai's own .text property explicitly skips thought parts. Real-types repro: parts [Part(text="reasoning: they want the capital", thought=True), Part(text="Paris.")] → transcript "reasoning: they want the capital Paris.", where message.text gives "Paris.". Pre-existing, but the change in the second inline comment widens how often the path fires.

3. ⚪ message.data is now evaluated twice per message (:245 and :248). It's a computed property that runs part.model_dump() per part and logs the "non-data parts" warning for mixed parts — at the 1.32 floor there's no once-per-process dedup, so a mixed message warns twice, which sits oddly beside the comment at :323. Measured cost is negligible (3.9 µs/chunk), so this is purely a has_data = bool(message.data) tidy-up.

4. ⚪ test_event_conversion is still one 113-line multi-scenario test — pre-existing, and the PR re-pointed it at the new fixtures without splitting it, even though it introduces exactly that per-scenario pattern next door.

5. ⚪ Pre-existing ruff UP035 on gemini_live.py:18 (AsyncGenerator from typing) — also on base, not yours.

How this review ran

Staged pipeline: routing triage → context build → parallel specialist passes (correctness/safety, adversarial-repro, test-quality) → aggregation with de-dup and a reachability gate → this post. Each pass was an independent fresh-context agent; the adversarial pass ran warm (no nested spawn available) against its own role rubric. I re-verified every load-bearing finding myself with real google.genai types before posting. Two threads from the earlier automated review are deliberately not re-raised: the exclusive→cumulative shift (you answered it) and the whole-object BidiUsageEvent assertions (fixed in 3bfda5ec).

AI review — solid work for a human to approve, not a gate.

[strandly-the-agent]

🟡 This fixture makes the assertion pass on a value the real API never produces.

modality is set to the plain string "AUDIO", so production's str(detail.modality).lower() (gemini_live.py:357) coincidentally yields "audio" and the test goes green. On the wire detail.modality is a MediaModality enum, and str(MediaModality.AUDIO).lower() is "mediamodality.audio" — which violates ModalityUsage.modality: Literal["text","audio","image","cached"] (types/events.py:460). So the one new test covering modality details certifies the wrong string and can't catch the real bug.

The production bug is pre-existing (see the appendix on the summary comment), so fixing it isn't your obligation here — but the fixture is new, and it's a two-character fix in a line this PR already moves:

Suggested change

	prompt_detail.modality = "AUDIO"
	prompt_detail.modality = genai_types.MediaModality.AUDIO

With the real enum, "modality": "audio" only passes once _convert_usage_metadata uses detail.modality.value.lower(). Same for response_detail at :643. If you'd rather keep this PR scoped to the usage fix, that's fine — just leave the fixture out of it rather than pinning a string the wire can't produce.

[strandly-the-agent]

🟡 Dangling issue link (×3), and this docstring narrates the diff.

https://github.com/strands-agents/harness-sdk/issues/3745 doesn't resolve — I checked via the API, and it's neither an issue nor a PR (the newest issue in the repo is ~3721). Same link at :577 and :675, and the PR's own "Related Issues" section is empty. docs/TESTING.md:82 draws the line as: a regression test for a discovered bug links the issue it guards; a test written as part of feature development carries no issue reference. These ship in the same PR as the fix, so unless there's a real issue to point at, dropping the links is the rule-conformant move.

Separately, "this branch previously returned early, discarding usage outright" narrates how the code changed, which root AGENTS.md explicitly forbids ("Never narrate how the code changed or what it used to be") — that reasoning belongs in the PR description.

Suggested change

	Guards https://github.com/strands-agents/harness-sdk/issues/3745 — this branch previously
	"""Session resumption tracks the handle and still emits co-attached usage metadata.
	usageMetadata sits outside the messageType union, so it can accompany any other field,
	including a session-resumption update.
	"""

[strandly-the-agent]

Read the reference — verdict

You were right that the docs settle it. Net: two of my findings withdrawn (replied on both threads — the interrupt/audio ordering and the duplicate-transcript one), and the usage question resolves to something real but narrower than I framed it — and not this PR's bug.

Usage semantics — answered: a per-generation-request snapshot, not a per-message delta. The reference defines usageMetadata as "Usage metadata about the response(s)", promptTokenCount as "Number of tokens in the prompt", and totalTokenCount as "Total token count for the generation request (prompt + response candidates)". Those are properties of the request, not of the message they ride on — so N usageMetadata payloads within one turn carry the same prompt count N times. The capabilities guide adds that the server sends them "periodically", and its own example prints each one standalone rather than summing.

So the bug is loop.py:341-344, not your converter. += over every BidiUsageEvent is only sound if exactly one usage message arrives per generation request, and nothing in the reference guarantees that. Two consequences:

1. The description's rationale — "every dropped event undercounts the session" — doesn't follow from the field semantics. Under snapshot semantics a dropped snapshot isn't an undercount, and summing the surviving ones is an overcount. Worth rewording, since it's the stated justification for the change.
2. Your converter change is still right on its own terms: "Server messages may have a usageMetadata field but will otherwise include exactly one of the other fields" means dropping it is genuine information loss, and a provider should surface what the server sent. But it does emit strictly more usage events than before — so if the cadence really is more than one per turn, this moves the session total further from the truth rather than closer. I'd land a snapshot-aware fix in loop.py (last-wins per generation request) alongside it, or at least file it before this merges.

One design note for whoever picks that up: BidiUsageEvent's implicit contract in loop.py is a delta, which is correct for openai_realtime (one usage per response.done) and is exactly what Gemini doesn't fit. That smells like it belongs at the event-contract level rather than in each provider.

Verdict: approve-leaning. What's left from my review is two test nits — the fixture that pins "audio" by passing a plain string where the wire sends a MediaModality enum, and the three issues/3745 links plus the "previously" narration. Everything else is pre-existing and parked in the appendix.