feat(openai-responses): plumb prompt_cache_options for GPT-5.6 - #3686
Draft
arielnabavian wants to merge 6 commits into
Draft
feat(openai-responses): plumb prompt_cache_options for GPT-5.6#3686arielnabavian wants to merge 6 commits into
arielnabavian wants to merge 6 commits into
Conversation
added 6 commits
August 5, 2026 23:40
CacheConfig(strategy="auto") previously only wrote a cache breakpoint on the last user message, leaving the (usually largest and most static) system prefix uncached. For workloads that share a system prompt across calls with varying user messages, every call wrote to cache and none read — a net cost regression versus no caching. Append a cachePoint to the system prompt when caching resolves to "anthropic" for the model and the caller hasn't already placed one at the end. Both SDKs mirror the change. TS adds a systemTTL knob to BedrockCacheConfig to match toolsTTL/messagesTTL. Fixes strands-agents#3144.
LiteLLM wraps many provider APIs behind one interface, so an SDK-level cache_config has to route to whatever the underlying provider actually supports: - Anthropic-family (anthropic/, bedrock/anthropic., vertex_ai/claude, vertex_ai_beta/claude) accept cache_control markers. Auto mode injects a cachePoint on the last stable system block and the last user message; the existing translator turns them into cache_control: ephemeral markers before dispatch. - OpenAI-compatible providers (openai/, deepseek/, xai/, azure/) cache automatically server-side. cache_config is accepted for portability and is a documented no-op. - Unrecognized underlying providers warn once per resolution and skip injection. Cache breakpoints on earlier turns are stripped before injection so a long conversation stays within Anthropic's 4-breakpoint request budget. Related: strands-agents#2970.
OpenAI's Chat Completions and Responses APIs cache prompts automatically server-side once they exceed the vendor threshold, so an SDK-level cache_config is accepted for cross-provider portability but is a documented no-op — the SDK does not inject anything into the request. Two vendor knobs that DO matter are now exposed as first-class config: - prompt_cache_key: routes similar requests to the same cache node under load, improving cache-hit rates when many agents share a stable system prefix. - prompt_cache_retention: "in_memory" (default, up to 1h absolute) or "24h" (extended retention on eligible models). Both flow verbatim into the request body on Chat Completions and Responses. Both SDKs mirror the surface (promptCacheKey / promptCacheRetention in TypeScript). Related: strands-agents#2970.
Meta's Llama API caches prompt prefixes automatically server-side, so an SDK-level cache_config is accepted for cross-provider portability but is a documented no-op — the SDK does not inject anything into the request. Two useful vendor knobs are now exposed: - prompt_cache_key: groups similar requests when caching. Useful when many agents share a stable system prefix. - prompt_cache_retention: Responses API only, "in_memory" (default) or "24h" on models that support extended retention. Both flow verbatim into the request body. TypeScript SDK does not vend a Meta-Llama provider — this change is Python-only. Related: strands-agents#2970.
Mistral's chat completions cache requests that share a stable prompt_cache_key and a common prefix; cached tokens bill at 10% of standard input. Unlike Anthropic/Bedrock, Mistral has no per-block cache marker to inject — the whole surface is the caller-supplied identifier. cache_config is accepted for cross-provider portability and is a documented no-op — set prompt_cache_key to actually opt in to caching. prompt_cache_key flows verbatim into the request body. TypeScript SDK does not vend a Mistral provider — this change is Python-only. Related: strands-agents#2970.
GPT-5.6 on Bedrock Mantle (openai.gpt-5-6-sol/terra/luna) supports a prompt_cache_options field to switch between "implicit" (default, server-auto) and "explicit" (caller-marked cache breakpoints on specific content blocks) caching modes. Adds prompt_cache_options as a first-class Responses-only config field. It flows verbatim into the request body. Explicit-mode breakpoint marking on specific content blocks is not exposed as an SDK surface yet — callers passing "explicit" today must know to route breakpoints through raw content blocks. Related: strands-agents#2970.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
github-actions
Bot
added
area-model
yonib05
added
complexity/high
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GPT-5.6 on Bedrock Mantle (
openai.gpt-5-6-sol/terra/luna) supports aprompt_cache_optionsfield to switch between"implicit"(default, server-auto) and"explicit"(caller-marked cache breakpoints on specific content blocks) caching modes.Adds
prompt_cache_optionsas a first-class Responses-only config field. It flows verbatim into the request body. Explicit-mode breakpoint marking on specific content blocks is not exposed as an SDK surface yet — callers passing"explicit"today must know to route breakpoints through raw content blocks.Stack
7/9. Related: #2970.
Test plan
openai.gpt-5-6-sol/terra/lunavia Bedrock Mantle