fix: Vertex/Gemini whitespace crash + surface retry-after hints (0.15.8)#55
Merged
fix: Vertex/Gemini whitespace crash + surface retry-after hints (0.15.8)#55
Conversation
- ContentPart.new/1: Ecto's default :empty_values trimmed whitespace, which crashed ContentPart.text/1 on legitimate Gemini text parts containing only newlines. Override empty_values to [""] so "\n\n\n" is preserved. - Messages.Gemini.parse_content/1: skip whitespace-only text parts defensively, capture finishReason and promptFeedback in metadata, log a warning when content is empty for non-STOP reasons (SAFETY, RECITATION, MAX_TOKENS) so blocked generations stop being silent. Also handle functionCall without args, and unify tool-call ID generation to a single 64-bit-entropy helper. - Add Nous.Errors.RetryInfo: parses google.rpc.RetryInfo from error body and Retry-After header into milliseconds. - ProviderError gains :retry_after_ms; Provider.request/3 now populates :status_code and :retry_after_ms from HTTP error tuples. - HTTP backends (Req, Hackney, stream Req) surface response headers in error tuples so RetryInfo can extract Retry-After.
Req returns headers as %{binary() => [binary()]} unconditionally and
hackney returns them as a list — the is_list / catch-all fallbacks I
added were flagged as pattern_match_cov.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a production crash in Vertex AI / Gemini response parsing and adds first-class support for server-suggested retry-after hints across all providers.
The crash
Vertex/Gemini occasionally return
textparts whose content is only whitespace (e.g."\n\n\n") — typically between tool calls or as filler when the model is partially blocked. Ecto's default:empty_valuesforcast/3treatswhitespace-only strings as empty, so
Nous.Message.ContentPart's changeset dropped thecontentfield entirely and then raised:…taking down the whole
Nous.LLM.run_with_tools/6call from inside background jobs.What this PR does
Fixes
Nous.Message.ContentPartoverrides:empty_valuesto[""]so legitimate whitespace content is preserved.Nous.Messages.Gemini.parse_content/1defensively skips whitespace-only text parts (matching the existing guard inNous.StreamNormalizer.Gemini).Nous.Messages.Gemini.parse_content/1no longer silently drops nullaryfunctionCallparts (those withoutargs); pattern now requires onlynameand falls back to%{}.Adds — retry-after surfacing
New
Nous.Errors.RetryInfoparsesgoogle.rpc.RetryInfofrom Vertex/Gemini error bodies and the standardRetry-Afterheader into milliseconds. Returnsnilwhen the server gives no hint — itself meaningful for Google APIs,since long-term/daily quota exhaustion deliberately omits
RetryInfoto discourage retry loops.Nous.Errors.ProviderErrorgains:retry_after_msalongside the existing:status_code. Both fields are populated automatically byNous.Provider.request/3andrequest_stream/3when the HTTP layer returns an error tuple.Callers can now branch on rate limits without parsing provider-specific bodies:
Adds — Gemini diagnostics
Nous.Messages.Gemini.from_response/1capturesfinishReasonandpromptFeedbackintomessage.metadataand emits aLogger.warningwhen content is empty for non-STOP reasons (SAFETY,RECITATION,MAX_TOKENS, etc.) or aprompt block. Previously these signals were discarded, so blocked generations manifested as silent empty messages.
Changes — HTTP error shape
Nous.HTTP.Backend.Req,Nous.HTTP.Backend.Hackney, andNous.HTTP.StreamBackend.Reqnow return{:error, %{status, body, headers}}instead of%{status, body}, surfacing response headers soRetry-Aftercan be parsed.headersis a list of{name, value}tuples. Existing pattern matches on%{status: _, body: _}continue to work — map matching is non-exhaustive.Changes — tool-call IDs
parse_content/1used"gemini_#{:rand.uniform(10_000)}"(~50% birthday-paradox collision at ~118 calls) whileparse_parts/1used"call_#{:rand.uniform(1_000_000)}". Bothnow share a
generate_tool_call_id/0helper using 64 bits of:crypto.strong_rand_bytes/1, base64url-encoded.Test plan
mix test— 1674 tests, 0 failures (24 new tests).Nous.Errors.RetryInfoTest— 16 tests covering body parsing (google.rpc.RetryInfo, fractional durations, missing/malformed entries), header parsing (Retry-Afterinteger, case-insensitive, non-integer/HTTP-daterejection, negative values), precedence (body wins over header), and edge cases (non-map input, empty maps).
Nous.HTTP.BackendTestagainst both Req and Hackney via Bypass: verifiesRetry-Afterheader round-trips throughRetryInfo, and that a Vertex-shaped 429 body withRetryInfois parsed end-to-end.Nous.ProviderTest:request/3correctly populates:status_codefrom HTTP error tuples,:retry_after_msfrom both body and header sources, and leaves bothnilfor transport errors.Nous.MessagesGeminiTest: whitespace-only text parts are skipped (the original Vertex"\n\n\n"payload),finishReasonandpromptFeedbackland in metadata,functionCallwithoutargsis parsed.Nous.Message.ContentPartTest:ContentPart.text("\n\n\n")no longer raises;niland""content are still rejected.mix compile --warnings-as-errors --forceclean.Migration notes
%{status: _, body: _}is unchanged. Code matching%{status: _, body: _} = errand reaching forMap.keys(err)will see an extra:headerskey.ProviderError.retry_after_msis new; existing pattern matches on the struct continue to compile because all fields default tonil.