feat(inference): multi-provider LLM upstreams (Anthropic, Ollama) + pluggable guardrail pipeline (OpenAI Moderation)

[John Seong (sandole)]

Summary

This is a first slice of the multi-cloud provider + guardrails roadmap theme. It lets an InferencePolicy pick the inference provider and declare a guardrail pipeline, and teaches the router to actually honor both:

spec:
  provider: anthropic          # or: ollama, azure-openai (default), bedrock (schema only, router says 501)
  guardrails:
    - provider: openai-moderation
      applyTo: both            # input | output | both

I started with Anthropic and Ollama because they're the two cheapest to do properly: Anthropic gets native Messages pass-through on /anthropic/v1/messages (streaming, tool use and multi-modal all survive, which the existing translation route couldn't offer), and Ollama is OpenAI-compatible so chat completions just work, buffered and SSE, with token metering intact.

The part I'd most like eyes on is the streaming guardrail. Scanning a stream after the fact is theater, so the SSE guard holds chunks back and only releases a window once a scan has covered it (default 1000 chars per window, GUARDRAIL_STREAM_SCAN_CHARS to tune). Flagged streams get cut with a structured error frame. The trade-off is chunkier delivery when a pipeline is active; policies without guardrails don't pay anything.

Everything is fail-closed on purpose. A policy that names a provider the router can't serve gets a 501 (bedrock) or 503 (missing key/endpoint) instead of quietly falling back to Azure, and a declared guardrail stage that can't run blocks the request rather than becoming an open gate. Provider keys (ANTHROPIC_API_KEY, OPENAI_MODERATION_API_KEY) only ever exist on the router sidecar - the reconciler forwards them from controller env the same way the dev creds already travel, and the router strips/replaces any x-api-key an agent tries to smuggle through (there's an integration test for exactly that).

Helm CRD template regenerated via the drift-test dumper. Endpoints (not secrets) joined CONFIG_HASH_INPUTS. Both Copilot review comments are addressed in 4b48b87c (outcome-accurate SSE error frames; bounded stream scan context).

Related Issues

No existing issue tracks this - the provider-expansion note in docs/architecture.md asks for a feature request per provider, so I'm happy to file one (auth model, Foundry-feature preservation, the works) and link it here if that's the flow you want. Follow-up slices (Bedrock, Vertex, vLLM; Bedrock Guardrails, Model Armor) would extend the same two seams: provider::ProviderKind and the guardrails::Guardrail trait.

Type of Change

• New feature

Checklist

• Code compiles/builds without errors (cargo check --workspace --tests)
• Tests pass (857 controller + 1,094 router, including the new integration suite)
• Linting passes (clippy adds zero warnings from this change; the one collapsible_if in routes/mod.rs also fires on main with this toolchain)
• Documentation updated (docs/api/crd-reference.md, docs/architecture.md, CHANGELOG.md)
• No secrets committed
• Helm chart updated (crd-inferencepolicy.yaml regenerated)

Testing

Automated (all offline):

• Controller: compile round-trips for the new fields, enum wire-tag pins, version-hash change detection, CEL coverage, helm drift.
• Router units: provider resolution truth table, loader back-compat with pre-slice profiles, moderation response parsing (fail-closed on malformed), and the SSE hold-and-release guard against a fake backend - clean stream intact, flagged text withheld and cut, events split across chunk boundaries, scan errors, keepalives, threshold windowing, bounded scan context.
• Wiremock end-to-end (inference-router/tests/multi_provider_guardrails.rs): fake Ollama (URL shape, no credentials sent), fake Anthropic (router key replaces agent-supplied key, anthropic-version injected), fake moderation endpoint (flag/pass/500→fail-closed), and missing-key pipeline construction failure.

Live, running the router binary locally:

• Anthropic Messages against the real api.anthropic.com, non-streaming and SSE, plus the 501 on chat/completions under an Anthropic policy.
• Chat completions (buffered + SSE) against a local Ollama, and a provider hot-swap through the policy reload watcher without a restart.
• Guardrails: fail-closed with no key configured (503 guardrail_misconfigured), input blocked pre-flight (403), and a mid-stream output cut - clean prompt, model emits the flagged word, zero bytes of model text reach the client, just the structured error frame + [DONE]. kars_guardrail_scans_total counted every scan by direction/outcome. The moderation verdict itself came from a local stub (no OpenAI key on this machine); everything around it was real.

Enforcement edge cases the tests + live run cover explicitly:

• The streaming guard holds bytes until a scan covers them even when a chunk boundary falls inside a data: line, and scans non-JSON data: payloads as raw text rather than fast-releasing an unrecognised frame. A 40-char-threshold live run delivered only the error frame on a cut, zero content deltas.
• All three Responses-API recovery paths (cached responses-only, stream-400 fallback, buffered-400 fallback) run the output scan.
• spec.provider is the sole routing selector; the pre-existing modelPreference.primary.provider tag stays informational, so an unchanged CR that only set a model preference keeps its Azure upstream (verified live).
• The sibling inference routes (/v1/completions, /v1/responses, /v1/embeddings, image generation) don't implement provider routing or guardrails, so they fail closed (501 on a non-Azure provider, 403 when guardrails are declared) rather than silently bypass the policy and live-verified 501 on /v1/embeddings and /v1/responses. Text over 16k chars is scanned in successive windows, not truncated.

[Copilot]

Pull request overview

This PR extends Kars’ inference plane to support policy-driven multi-provider routing (Anthropic, Ollama; Bedrock explicitly unimplemented) and introduces a pluggable guardrail pipeline (first backend: OpenAI Moderation) enforced for buffered and SSE streaming responses.

Changes:

• Adds InferencePolicy.spec.provider (typed enum with kebab-case wire tags) and spec.guardrails[] (ordered stages) with CRD/CEL validation, compilation into the router-consumed profile JSON, and updated docs/changelog.
• Implements router-side provider resolution + per-provider proxy behavior (URL shapes + auth schemes), keeping credentials router-side only.
• Implements guardrail pipeline enforcement for input/output, including hold-and-release scanning for SSE streams, plus metrics for scan outcomes.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
inference-router/tests/proxy_fake_upstream.rs	Updates tests to populate new UpstreamConfig fields (provider, api_key).
inference-router/tests/policy_status_endpoint.rs	Extends test router Config literals with new provider/guardrail fields.
inference-router/tests/multi_provider_guardrails.rs	Adds end-to-end wiremock tests for Anthropic/Ollama forwarding and moderation guardrail behavior.
inference-router/tests/failover_walk.rs	Updates failover tests for expanded UpstreamConfig.
inference-router/tests/egress_blocked_endpoint.rs	Extends test Config literals with new provider/guardrail fields.
inference-router/tests/agt_governance_integration.rs	Extends test Config literals with new provider/guardrail fields.
inference-router/src/routes/mod.rs	Adds apply_provider_resolution to retarget upstream per policy/provider precedence.
inference-router/src/routes/chat_completions.rs	Enforces provider selection + guardrails for chat completions, including streaming and recovery paths.
inference-router/src/routes/anthropic_messages.rs	Applies provider resolution + guardrails to the Anthropic Messages route (pass-through + translated paths).
inference-router/src/proxy.rs	Makes proxy forwarding provider-aware (auth header scheme + URL shaping) and adds UpstreamConfig::azure constructor.
inference-router/src/provider.rs	Introduces provider tag parsing/resolution with fail-closed semantics and config-backed targets.
inference-router/src/metrics.rs	Adds kars_guardrail_scans_total{provider,direction,outcome} metric.
inference-router/src/lib.rs	Exports new guardrails and provider modules.
inference-router/src/inference_policy_loader.rs	Loads compiled policy provider + guardrails fields into snapshots for per-request enforcement.
inference-router/src/guardrails.rs	Implements guardrail pipeline (OpenAI Moderation) and SSE hold-and-release stream guarding.
inference-router/src/failover.rs	Updates test helpers for expanded UpstreamConfig.
inference-router/src/config.rs	Adds router config surface for Anthropic/Ollama endpoints and Moderation backend config + secret mount loading.
docs/architecture.md	Updates architecture notes to include Anthropic/Ollama as wired providers.
docs/api/crd-reference.md	Documents spec.provider and spec.guardrails[] and clarifies precedence/semantics.
deploy/helm/kars/templates/crd-inferencepolicy.yaml	Regenerates CRD schema to include provider/guardrails fields + validations.
controller/src/reconciler/mod.rs	Forwards router-only env vars for provider endpoints/keys and moderation backend to sidecars (skipping empty).
controller/src/inference_policy.rs	Adds typed InferenceProvider + guardrail stage types to the CRD model.
controller/src/inference_policy_reconciler.rs	Enforces bundleRef mutual exclusion with provider/guardrails and keeps bundle canonical format unchanged.
controller/src/inference_policy_compile.rs	Emits compiled profile JSON keys provider and guardrails (null when absent) with wire-tag pin tests.
controller/src/crd_validations.rs	Adds CEL validations for bundleRef exclusivity and guardrail stage count bounds (1–8).
controller/src/config_hash.rs	Adds provider/guardrail endpoint env vars (non-secret) to config hash inputs.
CHANGELOG.md	Documents the multi-provider + guardrails slice and associated behavior/constraints.

[John Seong (sandole)]

John Seong (@sandole) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”), and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your contributions to Microsoft open source projects. This Agreement is effective as of the latest signature date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

@microsoft-github-policy-service agree

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.

Suppressed comments (5)

inference-router/src/guardrails.rs:731

• If an SSE data: payload is not valid JSON (or doesn't match the expected dialect), unscanned stays at 0 and the code may treat the chunk as safe to release. That breaks the “no model text reaches the client before a scan has covered it” contract and makes the guard bypassable by non-JSON/variant frames. A safe fallback is to treat unparseable data: payloads as text-bearing and include them in the scan context (or fail closed).

            if let Ok(event) = serde_json::from_str::<serde_json::Value>(payload)
                && let Some(text) = delta_text_from_event(self.dialect, &event)
            {
                self.unscanned += text.chars().count();
                self.accumulated.push_str(&text);
            }

inference-router/src/guardrails.rs:745

• In on_chunk, when unscanned == 0 the code immediately releases all held bytes. This is unsafe when line_carry contains a partial data: line split across chunks: the first chunk can contain unscanned model output bytes but no newline yet, so it gets released before the JSON payload is parsed and scanned. Only release immediately when there is no partial line carry buffered.

            // Fast path: window not full. Chunks carrying no delta
            // text at all (keepalives, role/annotation frames) are
            // safe to release immediately when nothing text-bearing
            // is being held alongside them.
            if self.unscanned == 0 {
                return SseGuardStep::Release(std::mem::take(&mut self.held));
            }

inference-router/src/routes/anthropic_messages.rs:287

• PR description claims x-kars-decision* headers are attached on every block. The Anthropic route returns deny_response(...) for provider-resolution and guardrail failures, but this helper currently doesn’t inject the decision headers (unlike the /v1/chat/completions path). That’s observable API drift for clients relying on those headers for auditing/telemetry.

    // Multi-provider slice: retarget at the policy-selected provider.
    // Fails closed — see `routes::apply_provider_resolution`.
    if let Err(e) = crate::routes::apply_provider_resolution(&state, &mut upstream, &policy) {
        tracing::warn!(
            target: "inference.audit",
            sandbox = %sandbox_name,
            inference_policy_digest = %policy.digest,
            decision = "deny",
            gate = "provider_resolution",
            error = %e,
            "InferencePolicy provider could not be resolved (anthropic route)"
        );
        let status = match e {
            ProviderError::Unimplemented { .. } => StatusCode::NOT_IMPLEMENTED,
            _ => StatusCode::SERVICE_UNAVAILABLE,
        };
        return deny_response(status, &e.to_string(), "api_error");
    }

docs/api/crd-reference.md:495

• Docs currently state spec.bundleRef is an alternative to inline provider / guardrails, but the controller-side reconciler explicitly notes the signed-bundle canonical format does not carry provider/guardrails yet (they’re forced to None when a bundle is used). As written, this suggests operators can move these fields into bundles when they currently cannot.

| `spec.bundleRef` | Signed OCI artifact alternative to inline `tokenBudget` / `contentSafety` / `modelPreference` / `provider` / `guardrails` / `displayName`. `appliesTo` always comes from the CR. |

deploy/helm/kars/templates/crd-inferencepolicy.yaml:198

• The CRD schema text says guardrail stages are materialized “at policy load time”, but the router builds the GuardrailPipeline per request from the policy snapshot (build_guardrail_pipeline in routes/chat_completions.rs). This description looks stale/misleading for operators troubleshooting runtime failures (which happen at request time).

                    A single stage of the router-side guardrail pipeline. The router
                    materialises each stage into a scanner (network client + policy)
                    at policy load time; a stage whose backend is not configured on
                    the router (e.g. missing moderation API key) fails the *request*

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (6)

inference-router/src/proxy.rs:527

• Same as the buffered forward path: this wraps provider-specific credential resolution, not just token acquisition. The current message is misleading for Anthropic/Ollama.

    let credential = credential_for_upstream(&auth, copilot.as_deref(), &upstream)
        .await
        .context("Failed to acquire auth token")?;
    let headers = build_upstream_headers(&request_headers, &auth, &credential, &upstream.endpoint)?;

inference-router/src/routes/chat_completions.rs:387

• The explicit 501 returned when a policy selects the Anthropic provider doesn’t attach the canonical x-kars-decision* headers, unlike other provider/guardrail blocks in this handler. This makes denial telemetry inconsistent for downstream tooling that relies on these headers.

    if upstream.provider == ProviderKind::Anthropic {
        return errors::openai(
            StatusCode::NOT_IMPLEMENTED,
            "InferencePolicy selects provider 'anthropic', which serves the Anthropic \
             Messages API — send Anthropic-shaped requests to /anthropic/v1/messages \
             (chat-completions translation for Anthropic is not implemented)",
            "provider_unimplemented",
        )
        .into_response();

inference-router/src/proxy.rs:300

• This error context is now used for all provider types (including Anthropic API keys and unauthenticated Ollama), so the message “auth token” is misleading. Renaming it to “upstream credential” makes logs/errors accurate across providers.

This issue also appears on line 524 of the same file.

    let credential = credential_for_upstream(auth, copilot, upstream)
        .await
        .context("Failed to acquire auth token")?;

controller/src/crd_validations.rs:283

• The validation message claims “the bundle carries those content fields”, but this repo currently explicitly drops provider/guardrails when bundleRef is used (see merge_bundle_with_selector in this same PR). The message should not imply the bundle contains provider/guardrails yet.

        ValidationRule {
            rule: "!has(self.bundleRef) || (!has(self.tokenBudget) && !has(self.contentSafety) && !has(self.modelPreference) && !has(self.provider) && !has(self.guardrails) && !has(self.displayName))".into(),
            message: Some("spec.bundleRef is mutually exclusive with spec.tokenBudget, spec.contentSafety, spec.modelPreference, spec.provider, spec.guardrails, and spec.displayName; the bundle carries those content fields".into()),
            reason: Some("FieldValueInvalid".into()),
            ..ValidationRule::default()

deploy/helm/kars/templates/crd-inferencepolicy.yaml:331

• This CRD CEL validation message also claims “the bundle carries those content fields”, but bundle-sourced policies currently do not carry provider/guardrails yet (they are forced to defaults). The message should avoid implying otherwise.

            - message: spec.bundleRef is mutually exclusive with spec.tokenBudget, spec.contentSafety, spec.modelPreference, spec.provider, spec.guardrails, and spec.displayName; the bundle carries those content fields
              reason: FieldValueInvalid
              rule: '!has(self.bundleRef) || (!has(self.tokenBudget) && !has(self.contentSafety) && !has(self.modelPreference) && !has(self.provider) && !has(self.guardrails) && !has(self.displayName))'

docs/api/crd-reference.md:495

• The docs currently describe bundleRef as an alternative to inline provider/guardrails, which implies those fields can come from the signed bundle. In this PR they’re explicitly inline-only and mutually exclusive with bundleRef, and bundle-sourced policies drop them to defaults. Updating this line would prevent operator confusion.

| `spec.bundleRef` | Signed OCI artifact alternative to inline `tokenBudget` / `contentSafety` / `modelPreference` / `provider` / `guardrails` / `displayName`. `appliesTo` always comes from the CR. |

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Suppressed comments (3)

inference-router/src/proxy.rs:526

• Same as above: this path resolves provider-specific credentials (including unauthenticated Ollama), so “auth token” is no longer an accurate description of what failed.

    let credential = credential_for_upstream(&auth, copilot.as_deref(), &upstream)
        .await
        .context("Failed to acquire auth token")?;

inference-router/src/guardrails.rs:725

• SseGuardState::ingest_text decodes arbitrary upstream bytes via String::from_utf8_lossy(chunk) and appends into line_carry. If a multi-byte UTF-8 sequence is split across chunk boundaries, from_utf8_lossy can replace it with U+FFFD, so the guardrail scans a different text than what the client ultimately receives. This can weaken the “no unscanned text is released” guarantee for non-ASCII output.

Consider buffering as raw bytes and only decoding complete lines (split on \n) with std::str::from_utf8, failing closed (cut the stream with a guardrail error frame) on invalid UTF-8 / unparseable data: payloads.

    fn ingest_text(&mut self, chunk: &[u8]) {
        self.line_carry.push_str(&String::from_utf8_lossy(chunk));
        // Keep the trailing partial line (no '\n' yet) in the carry.

inference-router/src/proxy.rs:299

• The error context string is now misleading: credential_for_upstream can return an Anthropic API key or None for Ollama, not just an auth token. Using “auth token” in the context makes failures harder to interpret when debugging non-Azure providers.

This issue also appears on line 524 of the same file.

    let credential = credential_for_upstream(auth, copilot, upstream)
        .await
        .context("Failed to acquire auth token")?;

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

controller/src/inference_policy.rs:166

• The doc comment claims modelPreference.primary.provider “takes precedence” over spec.provider, but the router implementation and docs state spec.provider is the only routing selector and modelPreference.*.provider stays informational. This comment is misleading and suggests behavior that does not exist.

    /// process. `modelPreference.primary.provider`, when it names a
    /// recognised tag, takes precedence over this field so a fallback
    /// chain can pin its own route. Mutually exclusive with

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (3)

controller/src/inference_policy.rs:167

• The doc comment says modelPreference.primary.provider takes precedence over spec.provider, but the router code explicitly ignores modelPreference.*.provider for routing (see inference-router/src/routes/mod.rs where apply_model_preference_override ignores primary.provider and provider routing is driven by policy.provider). This comment is misleading for CRD/API consumers and contradicts the fail-closed routing semantics described elsewhere in the PR.

    /// process. `modelPreference.primary.provider`, when it names a
    /// recognised tag, takes precedence over this field so a fallback
    /// chain can pin its own route. Mutually exclusive with

inference-router/src/proxy.rs:118

• The comment implies an inbound SDK-provided auth value is copied through, but build_upstream_headers explicitly strips inbound x-api-key (and other creds) before injecting provider auth. Clarifying that only anthropic-version is preserved (while x-api-key is always replaced) makes the security behavior unambiguous.

        // Anthropic's Messages API authenticates with `x-api-key` and
        // requires an `anthropic-version` header. The inbound SDK
        // value (when present) was already copied through above —
        // only the default is filled in here.

controller/src/reconciler/mod.rs:1964

• The router supports OPENAI_MODERATION_MODEL and GUARDRAIL_STREAM_SCAN_CHARS env overrides (see inference-router/src/config.rs and inference-router/src/guardrails.rs), but the controller only forwards moderation key/endpoint into the router sidecar env. In controller-managed sandboxes this makes those overrides effectively unusable, despite being documented/tunable in the router layer.

        // Multi-provider inference + guardrail backends. Router-only —
        // never on the agent container. Empty values are skipped so
        // Azure-only clusters keep an identical env surface (and
        // config-hash) to previous releases.
        for (name, value) in [
            ("ANTHROPIC_API_KEY", &ctx.anthropic_api_key),
            ("ANTHROPIC_ENDPOINT", &ctx.anthropic_endpoint),
            ("OLLAMA_ENDPOINT", &ctx.ollama_endpoint),
            ("OPENAI_MODERATION_API_KEY", &ctx.openai_moderation_api_key),
            (
                "OPENAI_MODERATION_ENDPOINT",
                &ctx.openai_moderation_endpoint,
            ),
        ] {

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (3)

inference-router/src/guardrails.rs:414

• scan_windows is documented as splitting by characters, but the early-return check uses text.len() (bytes). For non-ASCII text this can enter the slow path unnecessarily and makes the implementation diverge from the stated contract. Use a char-count check for consistency with the rest of the function.

fn scan_windows(text: &str) -> Vec<&str> {
    if text.len() <= MAX_SCAN_CHARS {
        return vec![text];
    }

inference-router/src/routes/chat_completions.rs:249

• The error code used for RouteGap::NonAzureProvider is provider_unimplemented, but the message indicates the provider is implemented and only this route lacks enforcement. Using a distinct code (e.g. provider_route_unsupported) avoids conflating “provider not implemented by this router build” (bedrock) with “implemented elsewhere but unsupported on this endpoint”.

        RouteGap::NonAzureProvider => (
            StatusCode::NOT_IMPLEMENTED,
            "provider_unimplemented",
            format!(

inference-router/src/config.rs:108

• The doc comment for openai_moderation_api_key says it “falls back to OPENAI_API_KEY, then the openai-moderation-api-key secret mount”, but the implementation checks the openai-moderation-api-key mount (via secret_from_env_or_mount) before falling back to OPENAI_API_KEY. Updating the comment avoids confusing operators about precedence.

    /// OpenAI Moderation key — `OPENAI_MODERATION_API_KEY` env (falls
    /// back to `OPENAI_API_KEY`, then the `openai-moderation-api-key`
    /// secret mount). `None` ⇒ `openai-moderation` stages fail closed.

[Pal Lakatos-Toth (pallakatos)]

Thanks a lot for this John Seong (@sandole) - this looks really great - I am going to review it properly this week! :)