feat(bedrock): default cache_config to auto - #3689
Draft
arielnabavian wants to merge 9 commits into
Draft
Conversation
added 9 commits
August 5, 2026 23:40
CacheConfig(strategy="auto") previously only wrote a cache breakpoint on the last user message, leaving the (usually largest and most static) system prefix uncached. For workloads that share a system prompt across calls with varying user messages, every call wrote to cache and none read — a net cost regression versus no caching. Append a cachePoint to the system prompt when caching resolves to "anthropic" for the model and the caller hasn't already placed one at the end. Both SDKs mirror the change. TS adds a systemTTL knob to BedrockCacheConfig to match toolsTTL/messagesTTL. Fixes strands-agents#3144.
LiteLLM wraps many provider APIs behind one interface, so an SDK-level cache_config has to route to whatever the underlying provider actually supports: - Anthropic-family (anthropic/, bedrock/anthropic., vertex_ai/claude, vertex_ai_beta/claude) accept cache_control markers. Auto mode injects a cachePoint on the last stable system block and the last user message; the existing translator turns them into cache_control: ephemeral markers before dispatch. - OpenAI-compatible providers (openai/, deepseek/, xai/, azure/) cache automatically server-side. cache_config is accepted for portability and is a documented no-op. - Unrecognized underlying providers warn once per resolution and skip injection. Cache breakpoints on earlier turns are stripped before injection so a long conversation stays within Anthropic's 4-breakpoint request budget. Related: strands-agents#2970.
OpenAI's Chat Completions and Responses APIs cache prompts automatically server-side once they exceed the vendor threshold, so an SDK-level cache_config is accepted for cross-provider portability but is a documented no-op — the SDK does not inject anything into the request. Two vendor knobs that DO matter are now exposed as first-class config: - prompt_cache_key: routes similar requests to the same cache node under load, improving cache-hit rates when many agents share a stable system prefix. - prompt_cache_retention: "in_memory" (default, up to 1h absolute) or "24h" (extended retention on eligible models). Both flow verbatim into the request body on Chat Completions and Responses. Both SDKs mirror the surface (promptCacheKey / promptCacheRetention in TypeScript). Related: strands-agents#2970.
Meta's Llama API caches prompt prefixes automatically server-side, so an SDK-level cache_config is accepted for cross-provider portability but is a documented no-op — the SDK does not inject anything into the request. Two useful vendor knobs are now exposed: - prompt_cache_key: groups similar requests when caching. Useful when many agents share a stable system prefix. - prompt_cache_retention: Responses API only, "in_memory" (default) or "24h" on models that support extended retention. Both flow verbatim into the request body. TypeScript SDK does not vend a Meta-Llama provider — this change is Python-only. Related: strands-agents#2970.
Mistral's chat completions cache requests that share a stable prompt_cache_key and a common prefix; cached tokens bill at 10% of standard input. Unlike Anthropic/Bedrock, Mistral has no per-block cache marker to inject — the whole surface is the caller-supplied identifier. cache_config is accepted for cross-provider portability and is a documented no-op — set prompt_cache_key to actually opt in to caching. prompt_cache_key flows verbatim into the request body. TypeScript SDK does not vend a Mistral provider — this change is Python-only. Related: strands-agents#2970.
GPT-5.6 on Bedrock Mantle (openai.gpt-5-6-sol/terra/luna) supports a prompt_cache_options field to switch between "implicit" (default, server-auto) and "explicit" (caller-marked cache breakpoints on specific content blocks) caching modes. Adds prompt_cache_options as a first-class Responses-only config field. It flows verbatim into the request body. Explicit-mode breakpoint marking on specific content blocks is not exposed as an SDK surface yet — callers passing "explicit" today must know to route breakpoints through raw content blocks. Related: strands-agents#2970.
Vercel's AI SDK does not vend a cross-provider caching abstraction — each provider package accepts caching hints under its own providerOptions.<provider> slot with a different shape (cacheControl on Anthropic, cachePoint on Bedrock, promptCacheKey on OpenAI, cachedContent on Google). This change accepts cacheConfig on VercelModel for cross-SDK portability with a documented effect matrix: - openai.* / openai-responses.*: server-auto; no-op - google.*: implicit caching on Gemini 2.5+ paid tier; no-op - anthropic.* / amazon-bedrock.*: warns once per model that per-block markers must be set via providerOptions on individual message parts; the adapter cannot inject them automatically cacheConfig is stripped from the downstream call settings so it does not leak into the vendor call. Related: strands-agents#2970.
Gemini 2.5+ models on the paid tier already cache prompt prefixes automatically (implicit caching), so an SDK-level cache_config is accepted for cross-provider portability but is a documented no-op — the SDK injects nothing into the request. Gemini also offers *explicit* caching via a separate CachedContent resource, which has a create/reference/delete lifecycle and bills for storage while the cache is alive. That is a distinct feature with real cost implications; we deliberately do not wire it up on the default path — callers who need it should use the underlying google-genai SDK directly. This scope decision is documented on the field. Both SDKs mirror the surface (cacheConfig on GoogleModelConfig, cache_config on GeminiConfig). Related: strands-agents#2970.
Flips the default: BedrockModel now constructs with CacheConfig(strategy="auto") instead of no caching. For models that support Anthropic-style caching (Claude family, us./eu./global.anthropic.* cross-region profiles), this places cache points on the system prompt and last user message on every request; tools caching still requires the explicit cache_tools knob. For non-Anthropic models on Bedrock (Amazon Nova, DeepSeek, Meta Llama, Mistral, Amazon Titan, etc.) the runtime silently skips injection — _shouldEnableCaching / _cache_strategy returns null — and logs a "model does not support automatic caching" warning at request time. No request payload changes for those models. Callers who want to opt out pass cache_config=None (Python) or cacheConfig: undefined (TypeScript) explicitly. Behavior change is technically breaking, but a cache miss falls back to normal input pricing so the worst case is neutral, and the best case is 90% off input on repeated prefixes. Ship note recommended. Related: strands-agents#2970. The AnthropicModel default flip is intentionally deferred until upstream PR strands-agents#3571 lands (adds cache_config to AnthropicModel); this stack composes on top.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
yonib05
added
complexity/high
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flips the default: BedrockModel now constructs with
CacheConfig(strategy="auto")instead of no caching. For models that support Anthropic-style caching (Claude family,us./eu./global.anthropic.*cross-region profiles), this places cache points on the system prompt and last user message on every request; tools caching still requires the explicitcache_toolsknob.For non-Anthropic models on Bedrock (Amazon Nova, DeepSeek, Meta Llama, Mistral, Amazon Titan, etc.) the runtime silently skips injection —
_shouldEnableCaching/_cache_strategyreturns null — and logs a "model does not support automatic caching" warning at request time. No request payload changes for those models.Callers who want to opt out pass
cache_config=None(Python) orcacheConfig: undefined(TypeScript) explicitly.Compatibility
Behavior change is technically breaking, but a cache miss falls back to normal input pricing so the worst case is neutral, and the best case is ~90% off input on repeated prefixes. Recommend a release-note callout.
The AnthropicModel default flip is intentionally deferred until upstream #3571 lands (adds
cache_configto AnthropicModel); this stack composes on top.Stack
10/10 — the finale. Closes #2970.
Test plan
Agent(model=BedrockModel())andAgent()(default model) both cache with no config