Skip to content

feat(agent): strip trailing assistant prefill for proxy provider compat#258

Closed
xuandung38 wants to merge 3 commits intonextlevelbuilder:mainfrom
xuandung38:feat/strip-assistant-prefill
Closed

feat(agent): strip trailing assistant prefill for proxy provider compat#258
xuandung38 wants to merge 3 commits intonextlevelbuilder:mainfrom
xuandung38:feat/strip-assistant-prefill

Conversation

@xuandung38
Copy link
Contributor

@xuandung38 xuandung38 commented Mar 18, 2026

Summary

Some LLM providers and models (via LiteLLM, OpenRouter, etc.) do not support assistant message prefill — requests ending with an assistant-role message get rejected with HTTP 400:

This model does not support assistant message prefill

This happens when the system injects trailing assistant messages to guide model behavior or establish context continuity.

Changes

  • Backend: Add per-agent strip_assistant_prefill toggle (stored in other_config)

    • ParseStripAssistantPrefill() on AgentData reads the flag from other_config
    • New stripAssistantPrefill field on Loop / LoopConfig structs
    • Wired through resolver.go → loop initialization
    • Before constructing ChatRequest, if enabled, removes the last message when it has role: "assistant"
  • Frontend: Checkbox in LLM Configuration section of the agent General tab

    • i18n support for EN, VI, ZH
    • State managed alongside other LLM config (provider, model, context window)
  • Config: Added .gemini/ and .opencode/ to .gitignore (AI tool config dirs)

Files Changed

File Change
internal/agent/loop_types.go Add stripAssistantPrefill field + wiring in LoopConfig
internal/agent/resolver.go Wire StripAssistantPrefill from agent config
ui/web/.../agent-general-tab.tsx Add state + save logic for the toggle
ui/web/.../llm-config-section.tsx Add checkbox UI component
.gitignore Ignore AI tool config directories

When to Use

Enable strip_assistant_prefill on agents that use proxy LLM providers which reject assistant-role prefill messages. This is a per-agent setting — only enable it for agents connected to providers that exhibit this issue.

Test Plan

  • Create agent with proxy provider (e.g., LiteLLM → GPT-4o)
  • Enable "Strip Assistant Prefill" in LLM Config section
  • Verify agent responds without HTTP 400 errors
  • Verify agents with native Anthropic provider still work normally (no regression)
  • Toggle saves correctly and persists across page reload

…LM call

Some LLM providers and models do not support assistant message prefill
— requests ending with an assistant-role message get rejected with
HTTP 400 ("This model does not support assistant message prefill").
This happens when the system injects assistant messages to guide model
behavior or establish context.

Add per-agent strip_assistant_prefill toggle (stored in other_config)
that removes the trailing assistant message before constructing the
ChatRequest. The option is configurable in the LLM Configuration
section of the agent General tab.

- Add ParseStripAssistantPrefill() to AgentData (reads from other_config)
- Add stripAssistantPrefill field to Loop/LoopConfig
- Wire through resolver → loop
- Add checkbox UI in LLM Config section with i18n (en/vi/zh)
Add .gemini/, .claude/, .opencode/ to .gitignore to prevent
committing user-specific AI tool configurations.
…prefill

# Conflicts:
#	internal/agent/loop_types.go
#	ui/web/src/pages/agents/agent-detail/agent-general-tab.tsx
#	ui/web/src/pages/agents/agent-detail/general-sections/llm-config-section.tsx
@viettranx
Copy link
Contributor

Thank you for identifying this issue with proxy providers rejecting assistant prefill messages! Great catch.

After tracing the full message flow, we found the root cause: team task reminders (lead + member) were injected after the user message, leaving a trailing assistant message that proxy providers (LiteLLM, OpenRouter) reject with HTTP 400.

Instead of a per-agent config toggle, we addressed this at the source:

  1. Reordered team reminders — injected before the user message so the user's message is always last (commit 11673da)
  2. Added a safety net in the OpenAI provider to strip any trailing assistant message that might slip through in the future

This approach requires zero configuration — all agents using proxy providers automatically benefit without needing to toggle a setting.

Resolved in main via 11673da.

@viettranx viettranx closed this Mar 20, 2026
@xuandung38
Copy link
Contributor Author

The approach makes sense, thanks sir.

@xuandung38 xuandung38 deleted the feat/strip-assistant-prefill branch March 20, 2026 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants