Repository navigation
Conversation
- Probe Ollama, llama.cpp, LM Studio and vLLM runtime context windows, including reverse-proxy subpaths - Add context_window and context_profile=local: clamp budgets, compaction triggers, tool-output limits and always-visible tools - Count tool schemas in the soft input budget and apply a 10% token estimate margin for local tokenizers - Prefer max_tokens for local requests across agent, attack-chain, FOFA, workflow and HITL paths, and honor channel extra_request_fields - Recognize local context-overflow errors and retry with aggressive compaction
The warning was computed after the runtime config had already been clamped, so configured and effective values were identical and the clamp warning was lost.
- Surface llama.cpp parallel slots via context_slots and a warning instead of treating n_ctx as the per-request window - Pass max_completion_tokens into detection previews and warn when the output cap is clamped - Clamp the effective budget to the window and use the assumed local window when clamping output - Overwrite a stale window on explicit auto-detect so the preview matches the saved value - Keep the budget-warning test repeatable and document failover/budget semantics
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimizes how the agent uses local OpenAI-compatible models whose effective context is usually 32K-256K, while keeping cloud behavior unchanged unless the new opt-in profile is enabled.
internal/modelmetaprobes Ollama/api/ps+/api/show, llama.cpp/props, LM Studio/api/v0/models, then vLLM/OpenAI-compatible/v1/models. Reverse-proxy subpaths (e.g.https://host/ollama/v1) are supported, and runtime-vs-model-maximum uncertainty is surfaced as warnings.context_windowandcontext_profile: local. Local clamps the shared budget to 60% of the window (max 128K), keeps output reserve bounded, and tightens summarization triggers, tool-output truncation/clearing, user-message/ledger caps and plan step results. It only tightens explicit middleware settings.max_tokensinstead ofmax_completion_tokensacross the agent, attack-chain, FOFA, workflow-draft and HITL paths; channel-levelreasoning.extra_request_fieldsis now merged into those auxiliary payloads.exceeds the available context size), LM Studio, vLLM (max_model_len) and related messages so aggressive compaction retry can run.ai.channels.num_ctx, KV cache/prefix caching, thinking models and minimum viable windows;config.example.yaml, CHANGELOG and all three i18n files updated.Verification
go test ./...passes (full suite),go vet ./...passes,gofmtclean.max_tokensfor both model factories and summarization, overflow error classification, channel validation, and detect-context response preview.context_window/context_profile; invalid profile and negative window were rejected with 400; settings page rendered the detected result and saved the channel.Notes
context_profileis opt-in;autopreserves existing behavior./v1endpoint ignoresnum_ctx; the docs explain the requiredOLLAMA_CONTEXT_LENGTH/Modelfile setup and why oversized prompts drop the system prompt first.