feat(inference): multi-provider LLM upstreams (Anthropic, Ollama) + pluggable guardrail pipeline (OpenAI Moderation)

CHANGELOG.md

@@ -7,6 +7,91 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased] — `crd-well-oiled-machine`
 
+### Multi-cloud LLM providers + pluggable guardrails (slice 1: Anthropic, Ollama, OpenAI Moderation)
+
+First slice of the "multi-cloud LLM providers + native guardrails"
+roadmap theme. Credentials stay router-side (secret mount / env on the
+sidecar) — the agent process keeps talking to the localhost proxy and
+never sees a provider key.
+
+**`InferencePolicy` CRD (controller)**
+
+- `spec.provider` — new optional typed enum (`InferenceProvider`:
+  `azure-openai` | `anthropic` | `ollama` | `bedrock`), serialized with
+  the same kebab-case tags `ModelRef.provider` strings have always
+  documented, so existing YAML values stay valid. `bedrock` is
+  schema-accepted for forward-compat; the router answers 501 until the
+  Bedrock client lands (declared intent is never silently rerouted).
+- `spec.guardrails[]` — new optional ordered guardrail pipeline
+  (`{provider: openai-moderation, applyTo: input|output|both}`, 1–8
+  stages via CEL). Both new fields are mutually exclusive with
+  `spec.bundleRef` (CEL + reconciler defense-in-depth) and flow through
+  `compile_to_profile` into the compiled `inference-policy.json`
+  (`"provider"` / `"guardrails"` keys, null when absent).
+- Helm CRD template regenerated (`crd-inferencepolicy.yaml`).
+- Reconciler forwards router-only provider config to every sidecar when
+  present on the controller env: `ANTHROPIC_API_KEY`,
+  `ANTHROPIC_ENDPOINT`, `OLLAMA_ENDPOINT`, `OPENAI_MODERATION_API_KEY`
+  (falls back to `OPENAI_API_KEY`), `OPENAI_MODERATION_ENDPOINT`.
+  Endpoints (not secrets) join `CONFIG_HASH_INPUTS`.
+
+**Inference router — multi-provider upstreams**
+
+- New `provider` module: tag parsing + fail-closed resolution.
+  `spec.provider` is the sole routing selector; the pre-existing
+  `modelPreference.primary.provider` tag stays informational (it drove
+  no routing before this slice), so an unchanged CR that only set a
+  model preference keeps its Azure upstream. Unknown tags warn and fall
+  back to Azure; missing endpoint/credential → 503; unimplemented
+  `bedrock` → 501. Never a silent Azure fallback for a selected
+  non-Azure provider.
+- `UpstreamConfig` carries `provider` + router-held `api_key`;
+  `proxy.rs` gains per-provider URL shapes (Anthropic: path verbatim;
+  Ollama: OpenAI-compat under `/v1/`) and auth schemes (Anthropic:
+  `x-api-key` + default `anthropic-version`; Ollama: no credential).
+  Agent-supplied `x-api-key` headers are stripped as before.
+- `provider: anthropic` serves the Anthropic Messages surface natively
+  (`/anthropic/v1/messages`, `/v1/messages`): streaming, tool use and
+  multi-modal content pass through. OpenAI-shaped `/v1/chat/completions`
+  under an Anthropic policy returns an explicit 501 pointing at the
+  Messages surface (mirrors the GitHub-Models 501 precedent).
+- `provider: ollama` serves `/v1/chat/completions` (buffered + SSE)
+  against `OLLAMA_ENDPOINT` with token metering and budget tracking.
+- Routing + guardrails are wired on `/v1/chat/completions` and the
+  Anthropic Messages routes only. The sibling inference routes
+  (`/v1/completions`, `/v1/responses`, `/v1/embeddings`, image
+  generation) don't implement either and **fail closed** (501 on a
+  non-Azure provider, 403 when guardrails are declared) rather than
+  silently bypass the policy; a plain Azure policy uses them unchanged.
+
+**Inference router — pluggable guardrail pipeline**
+
+- New `guardrails` module: `Guardrail` trait + OpenAI Moderation
+  backend (`POST {endpoint}/v1/moderations`, model
+  `omni-moderation-latest`, override via `OPENAI_MODERATION_MODEL`).
+- Enforcement on the governed routes: request pre-flight (input
+  stages), buffered responses (including all Responses-API recovery
+  paths), and SSE streams via **hold-and-release** windows
+  (`GUARDRAIL_STREAM_SCAN_CHARS`, default 1000, clamped to the 16k scan
+  cap) — no assistant message text is delivered before a scan covers
+  it, including across partial-line chunk boundaries; flagged streams
+  are cut with a structured SSE error frame + `data: [DONE]`. Text over
+  16k chars is scanned in successive windows, not truncated, so nothing
+  can be hidden past the cap. Tool-call arguments and provider
+  extended-"thinking" deltas are not yet scanned (roadmap).
+- Fail-closed contract: declared-but-unbuildable stages reject the
+  request (503 `guardrail_misconfigured`); backend outages block (502
+  `guardrail_unavailable`); unparseable bodies are scanned as raw text
+  rather than skipped. New `kars_guardrail_scans_total` metric
+  (provider / direction / outcome) and `x-kars-decision*` headers on
+  every block.
+
+Tests: compile/round-trip + enum wire-tag pins (controller), provider
+resolution truth table, loader parsing, hold-and-release SSE guard
+(clean / violation / split-event / scan-error), and wiremock
+integration tests against fake Anthropic / Ollama / moderation
+upstreams (`inference-router/tests/multi_provider_guardrails.rs`).
+
 ### Upstream alignment — AGT mesh-identity hardening merged
 
 [microsoft/agent-governance-toolkit#2719](https://github.com/microsoft/agent-governance-toolkit/pull/2719)

controller/src/config_hash.rs

@@ -38,6 +38,11 @@ use std::sync::LazyLock;
 /// change and should be called out in the audit trail.
 pub const CONFIG_HASH_INPUTS: &[&str] = &[
     "KARS_DISABLE_ENTRA_AUTH",
+    // Multi-provider endpoints (never secrets — matches the
+    // AZURE_OPENAI_API_KEY precedent).
+    "ANTHROPIC_ENDPOINT",
+    "OLLAMA_ENDPOINT",
+    "OPENAI_MODERATION_ENDPOINT",
     "AZURE_AUTHORITY_HOST",
     "AZURE_OPENAI_ENDPOINT",
     "AZURE_SUBSCRIPTION_ID",

controller/src/crd_validations.rs

@@ -277,8 +277,16 @@ pub fn inference_policy_validations() -> Vec<ValidationRule> {
     let severities = "['Safe','Low','Medium','High']";
     vec![
         ValidationRule {
-            rule: "!has(self.bundleRef) || (!has(self.tokenBudget) && !has(self.contentSafety) && !has(self.modelPreference) && !has(self.displayName))".into(),
-            message: Some("spec.bundleRef is mutually exclusive with spec.tokenBudget, spec.contentSafety, spec.modelPreference, and spec.displayName; the bundle carries those content fields".into()),
+            rule: "!has(self.bundleRef) || (!has(self.tokenBudget) && !has(self.contentSafety) && !has(self.modelPreference) && !has(self.provider) && !has(self.guardrails) && !has(self.displayName))".into(),
+            message: Some("spec.bundleRef is mutually exclusive with spec.tokenBudget, spec.contentSafety, spec.modelPreference, spec.provider, spec.guardrails, and spec.displayName; the bundle carries those content fields".into()),
+            reason: Some("FieldValueInvalid".into()),
+            ..ValidationRule::default()
+        },
+        // An empty list is an authoring mistake (omit the field for
+        // no pipeline); the cap bounds per-request scan fan-out.
+        ValidationRule {
+            rule: "!has(self.guardrails) || (size(self.guardrails) >= 1 && size(self.guardrails) <= 8)".into(),
+            message: Some("spec.guardrails, when set, must contain 1-8 stages (omit the field for no pipeline)".into()),
             reason: Some("FieldValueInvalid".into()),
             ..ValidationRule::default()
         },

controller/src/inference_policy.rs

@@ -54,6 +54,44 @@ use serde::{Deserialize, Serialize};
 
 use crate::mcp_server::LocalObjectRef;
 
+/// Inference provider selector. Explicit kebab-case renames (not
+/// `rename_all`) keep the wire tags byte-identical to the free-form
+/// `ModelRef.provider` strings this CRD has always documented, so a
+/// CR can move to the typed field without a migration. `bedrock` is
+/// schema-accepted for forward-compat; the router returns 501 until
+/// its client lands.
+#[derive(Serialize, Deserialize, Clone, Copy, Debug, Default, PartialEq, Eq, JsonSchema)]
+pub enum InferenceProvider {
+    /// Azure OpenAI / Azure AI Foundry (default — Phase 1 substrate).
+    #[default]
+    #[serde(rename = "azure-openai")]
+    AzureOpenAI,
+    /// Anthropic Messages API (`api.anthropic.com` or a compatible
+    /// gateway).
+    #[serde(rename = "anthropic")]
+    Anthropic,
+    /// OpenAI-compatible Ollama server (in-cluster or local).
+    #[serde(rename = "ollama")]
+    Ollama,
+    /// AWS Bedrock — schema-level forward-compat; router support is a
+    /// follow-up slice.
+    #[serde(rename = "bedrock")]
+    AWSBedrock,
+}
+
+impl InferenceProvider {
+    /// The kebab-case wire tag serde emits.
+    #[must_use]
+    pub fn as_tag(&self) -> &'static str {
+        match self {
+            Self::AzureOpenAI => "azure-openai",
+            Self::Anthropic => "anthropic",
+            Self::Ollama => "ollama",
+            Self::AWSBedrock => "bedrock",
+        }
+    }
+}
+
 /// `InferencePolicy.spec` — declares per-sandbox inference-time
 /// guardrails: token budgets, Content Safety severity floors, model
 /// preference + fallback chain.
@@ -102,6 +140,20 @@ pub struct InferencePolicySpec {
     /// [`Self::bundle_ref`].
     pub model_preference: Option<ModelPreference>,
 
+    /// Inference provider for the call sites this policy governs, and
+    /// the sole routing selector. Absent ⇒ the env-configured Azure
+    /// OpenAI / Foundry upstream. A non-Azure provider swaps the base
+    /// URL and auth scheme; credentials stay in the router sidecar,
+    /// never in the agent. Mutually exclusive with [`Self::bundle_ref`].
+    pub provider: Option<InferenceProvider>,
+
+    /// Ordered guardrail pipeline stages the router runs around each
+    /// call (input pre-flight + response, buffered and streaming), in
+    /// declaration order — first flag blocks. Absent ⇒ only the
+    /// Phase 1 substrate (Foundry annotations + `contentSafety`).
+    /// Mutually exclusive with [`Self::bundle_ref`].
+    pub guardrails: Option<Vec<GuardrailStage>>,
+
     /// Optional human-readable label. Mutually exclusive with
     /// [`Self::bundle_ref`] — when `bundleRef` is set, the label
     /// comes from the signed bundle.
@@ -228,6 +280,45 @@ pub struct ModelRef {
     pub deployment: String,
 }
 
+/// A single router-side guardrail stage. A stage whose backend isn't
+/// configured (e.g. missing moderation key) fails the request closed,
+/// never skips.
+#[derive(Debug, Serialize, Deserialize, Clone, Copy, PartialEq, Eq, JsonSchema)]
+#[serde(rename_all = "camelCase")]
+pub struct GuardrailStage {
+    /// Guardrail backend.
+    pub provider: GuardrailProvider,
+
+    /// Which direction(s) this stage scans. Absent ⇒ `both`.
+    pub apply_to: Option<GuardrailApplyTo>,
+}
+
+/// Guardrail backend. Bedrock Guardrails / Model Armor are roadmap
+/// follow-ups that extend this enum.
+#[derive(Debug, Serialize, Deserialize, Clone, Copy, PartialEq, Eq, JsonSchema)]
+pub enum GuardrailProvider {
+    /// OpenAI Moderation API; router-side key via
+    /// `OPENAI_MODERATION_API_KEY` (falls back to `OPENAI_API_KEY`).
+    #[serde(rename = "openai-moderation")]
+    OpenAIModeration,
+}
+
+/// Scan direction for a [`GuardrailStage`].
+#[derive(Debug, Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Default, JsonSchema)]
+pub enum GuardrailApplyTo {
+    /// Scan only the request (prompt) text.
+    #[serde(rename = "input")]
+    Input,
+    /// Scan only the response (completion) text — buffered and
+    /// streaming.
+    #[serde(rename = "output")]
+    Output,
+    /// Scan both directions (default).
+    #[default]
+    #[serde(rename = "both")]
+    Both,
+}
+
 /// Status of an `InferencePolicy` reconcile.
 #[derive(Debug, Serialize, Deserialize, Default, Clone, JsonSchema)]
 #[serde(rename_all = "camelCase")]

controller/src/inference_policy_compile.rs

@@ -72,6 +72,8 @@ use crate::inference_policy::InferencePolicySpec;
 ///   "tokenBudget":     { "perRequestTokens": ..., "dailyTokens": ..., "monthlyTokens": ... } | null,
 ///   "contentSafety":   { "hate": ..., "selfHarm": ..., "sexual": ..., "violence": ..., "requirePromptShields": ... } | null,
 ///   "modelPreference": { "primary": {provider, deployment}, "fallback": [...] } | null,
+///   "provider":        "azure-openai" | "anthropic" | "ollama" | "bedrock" | null,
+///   "guardrails":      [ { "provider": "openai-moderation", "applyTo": "input"|"output"|"both"|null }, ... ] | null,
 ///   "displayName":     "..." | null
 /// }
 /// ```
@@ -118,11 +120,29 @@ pub fn compile_to_profile(spec: &InferencePolicySpec) -> Value {
         })
     });
 
+    // Provider + guardrails emit the same kebab-case wire tags the
+    // CRD serde uses, so the router parses one vocabulary.
+    let provider = spec.provider.as_ref().map(|p| json!(p.as_tag()));
+
+    let guardrails = spec.guardrails.as_ref().map(|stages| {
+        json!(
+            stages
+                .iter()
+                .map(|g| json!({
+                    "provider": g.provider,
+                    "applyTo": g.apply_to,
+                }))
+                .collect::<Vec<_>>()
+        )
+    });
+
     json!({
         "appliesTo": applies_to,
         "tokenBudget": token_budget,
         "contentSafety": content_safety,
         "modelPreference": model_preference,
+        "provider": provider,
+        "guardrails": guardrails,
         "displayName": spec.display_name,
     })
 }
@@ -195,7 +215,8 @@ pub fn inference_policy_digest(body: &[u8]) -> String {
 mod tests {
     use super::*;
     use crate::inference_policy::{
-        ContentSafetyFloor, InferenceAppliesTo, InferencePolicySpec, ModelPreference, ModelRef,
+        ContentSafetyFloor, GuardrailApplyTo, GuardrailProvider, GuardrailStage,
+        InferenceAppliesTo, InferencePolicySpec, InferenceProvider, ModelPreference, ModelRef,
         TokenBudget,
     };
 
@@ -230,6 +251,17 @@ mod tests {
                     deployment: "claude-3-5-sonnet".into(),
                 }],
             }),
+            provider: Some(InferenceProvider::Anthropic),
+            guardrails: Some(vec![
+                GuardrailStage {
+                    provider: GuardrailProvider::OpenAIModeration,
+                    apply_to: Some(GuardrailApplyTo::Output),
+                },
+                GuardrailStage {
+                    provider: GuardrailProvider::OpenAIModeration,
+                    apply_to: None,
+                },
+            ]),
             display_name: Some("Prod chat policy".into()),
             bundle_ref: None,
         }
@@ -243,6 +275,8 @@ mod tests {
         assert!(profile.get("tokenBudget").unwrap().is_null());
         assert!(profile.get("contentSafety").unwrap().is_null());
         assert!(profile.get("modelPreference").unwrap().is_null());
+        assert!(profile.get("provider").unwrap().is_null());
+        assert!(profile.get("guardrails").unwrap().is_null());
         assert!(profile.get("appliesTo").unwrap().is_object());
     }
 
@@ -271,9 +305,60 @@ mod tests {
             profile["modelPreference"]["fallback"][0]["provider"],
             "anthropic"
         );
+        assert_eq!(profile["provider"], "anthropic");
+        let stages = profile["guardrails"].as_array().unwrap();
+        assert_eq!(stages.len(), 2);
+        assert_eq!(stages[0]["provider"], "openai-moderation");
+        assert_eq!(stages[0]["applyTo"], "output");
+        assert_eq!(stages[1]["provider"], "openai-moderation");
+        assert!(stages[1]["applyTo"].is_null());
         assert_eq!(profile["displayName"], "Prod chat policy");
     }
 
+    #[test]
+    fn provider_enum_serializes_to_kebab_case_wire_tags() {
+        // Wire-contract pin: the typed `spec.provider` must emit the
+        // same kebab-case tags the free-form `ModelRef.provider`
+        // strings have always documented, so existing YAML values
+        // stay valid when operators migrate to the typed field.
+        for (variant, tag) in [
+            (InferenceProvider::AzureOpenAI, "azure-openai"),
+            (InferenceProvider::Anthropic, "anthropic"),
+            (InferenceProvider::Ollama, "ollama"),
+            (InferenceProvider::AWSBedrock, "bedrock"),
+        ] {
+            assert_eq!(serde_json::to_value(variant).unwrap(), tag);
+            assert_eq!(variant.as_tag(), tag);
+            let parsed: InferenceProvider = serde_json::from_value(serde_json::json!(tag)).unwrap();
+            assert_eq!(parsed, variant);
+        }
+    }
+
+    #[test]
+    fn version_hash_changes_when_provider_changes() {
+        let mut a = full_spec();
+        let b = full_spec();
+        a.provider = Some(InferenceProvider::Ollama);
+        assert_ne!(
+            version_hash(&compile_to_profile(&a)),
+            version_hash(&compile_to_profile(&b))
+        );
+    }
+
+    #[test]
+    fn version_hash_changes_when_guardrails_change() {
+        let mut a = full_spec();
+        let b = full_spec();
+        a.guardrails = Some(vec![GuardrailStage {
+            provider: GuardrailProvider::OpenAIModeration,
+            apply_to: Some(GuardrailApplyTo::Input),
+        }]);
+        assert_ne!(
+            version_hash(&compile_to_profile(&a)),
+            version_hash(&compile_to_profile(&b))
+        );
+    }
+
     #[test]
     fn compile_is_deterministic() {
         let spec = full_spec();