β οΈ EARLY STAGE PROJECT WARNING
This project is currently in its early development stages. Breaking changes are expected and the API may change significantly between releases. Use with caution in production environments and expect potential migration requirements when upgrading versions.
A Kubernetes operator for automating the deployment and management of Inference Gateway instances
A Kubernetes operator for automating the deployment and management of Inference Gateway instances on Kubernetes.
This Kubernetes operator extends the Kubernetes API to create, configure and manage Inference Gateway instances within a Kubernetes cluster. It provides a comprehensive CRD (Custom Resource Definition) that allows you to declaratively manage:
- Gateway Deployment: Automated deployment with configurable replicas, resources, and Horizontal Pod Autoscaling (HPA)
- AI Provider Integration: Support for OpenAI, Anthropic, Ollama, and other AI/ML providers
- Authentication & Authorization: OIDC integration with configurable identity providers
- Model Context Protocol (MCP): Integration with MCP servers for extended AI capabilities
- Agent-to-Agent (A2A): Support for distributed agent communication and orchestration
- GPU Runtimes: Lease externally hosted, GPU-backed inference runtimes (RunPod today) via a pluggable provider interface, exposed as an HTTP endpoint and connection Secret (
examples/gpu) - Observability: Built-in metrics, tracing, and health monitoring
- Network Configuration: Service and Gateway API routing (Gateway + HTTPRoute) with listener-level TLS
The operator follows cloud-native best practices and provides a unified control plane for managing both the gateway infrastructure and its associated AI workloads.
| π Deployment | π Security | π Observability | π Integration |
|---|---|---|---|
| Auto-scaling | OIDC Auth | Prometheus Metrics | MCP Protocol |
| Rolling Updates | TLS/SSL | Distributed Tracing | A2A Agents |
| Health Checks | Secret Management | Status Monitoring | Multiple Providers |
| Resource Management | Network Policies | Logging | Custom Extensions |
π€ Supported AI Providers:
- Anthropic β’ Cloudflare β’ Cohere β’ DeepSeek β’ ElevenLabs β’ Google β’ Groq β’ llama.cpp β’ MiniMax β’ Mirelo β’ Mistral β’ Moonshot β’ NVIDIA β’ Ollama β’ Ollama Cloud β’ OpenAI β’ Webwright β’ Z.ai
βΈοΈ Kubernetes Native:
- CRDs β’ Controller Pattern β’ RBAC β’ Service Mesh Ready
- π Quick Start
- π¦ Installation
- β Verification
- π·οΈ Namespace Scoping
- π Deploy Your First Gateway
- π€ Deploy an Orchestrator
- π Upgrade
- ποΈ Uninstallation
- ποΈ Supported Architectures
- π API Overview
- βοΈ Configuration Examples
- β Frequently Asked Questions
- ποΈ Development
- π Monitoring & Management
- π§ Troubleshooting
- π API Reference
- π€ Contributing
- π License
The Gateway CRD supports the following key configuration areas:
- Replicas: Number of gateway instances (1-100)
- Image: Container image and version
- Environment: Deployment environment (development, staging, production)
- Resources: CPU and memory requests/limits
- Horizontal Pod Autoscaler: Automatic scaling based on CPU, memory, or custom metrics
- Min/Max Replicas: Configurable scaling boundaries
- Multiple Metrics: CPU utilization, memory utilization, custom metrics support
- Stabilization Windows: Fine-tuned scaling behavior control
Powered by OpenTelemetry for industry-standard observability:
- Metrics: Prometheus metrics endpoint with OpenTelemetry SDK
- Request counts, durations, error rates
- Token usage tracking (prompt, completion, total)
- Provider-specific metrics by model
- Custom histogram boundaries for latency analysis
- Tracing: Distributed tracing with OpenTelemetry exporters
- OTLP trace export to Jaeger, Zipkin, or any OTLP-compatible backend
- Request flow visualization across services
- Performance bottleneck identification
- Logs: Structured logging with correlation IDs for distributed tracing
- OIDC: OpenID Connect is the only authentication method the gateway
implements. The operator maps
spec.auth.enabledplus thespec.auth.oidcfieldsissuerUrl,clientId,audiences,clientSecretRefandcaCertRefonto the gateway container. spec.auth.providercurrently has no effect - the Gateway controller never reads it, so settingjwtorbasicstill yields OIDC.
Support for multiple AI/ML providers with flexible configuration. The list is
generated from the canonical inference-gateway/schemas
OpenAPI enum into internal/providers/zz_generated_providers.go,
so it never drifts from the gateway:
providers[].name |
Provider |
|---|---|
anthropic |
Claude API |
cloudflare |
Cloudflare Workers AI |
cohere |
Command and embedding models |
deepseek |
Cost-effective reasoning models |
elevenlabs |
Speech and audio models |
google |
Google AI / Gemini |
groq |
Fast inference with open models |
llamacpp |
llama.cpp server |
minimax |
MiniMax models |
mirelo |
Mirelo AI sound-effect generation |
mistral |
Mistral AI |
moonshot |
Moonshot AI (Kimi) |
nvidia |
NVIDIA NIM |
ollama |
Local model serving |
ollama_cloud |
Ollama Cloud |
openai |
OpenAI API |
webwright |
Webwright AI |
zai |
Z.ai |
The gateway reads <PROVIDER_ID>_API_URL and <PROVIDER_ID>_API_KEY only for
these ids - there is no catch-all custom provider. To reach another
OpenAI-compatible endpoint (vLLM, a proxy), override the URL of an OpenAI-compatible
id such as openai, including its /v1 base path. See
examples/gpu for a worked
example.
providers[].name has no CRD enum: it is validated at runtime and matched
case-insensitively, so OpenAI, openai and OPENAI are equivalent. A
name outside the list above is skipped - it is left out of
status.providerSummary.
- MCP (Model Context Protocol): Integration with MCP servers for tool access
- Service Discovery: Automatic discovery of
MCPCRs via Kubernetes label selectors (Gateway and Orchestrator) - Dynamic Updates: The pod's
MCP_SERVERS/ mountedmcp.yamlis rebuilt and rolled when the discovered set changes
- Service Discovery: Automatic discovery of
- A2A (Agent-to-Agent): Distributed agent communication - the
Gatewayrelays A2A calls whenspec.a2a.enabledis set, theOrchestratordrives multi-turn agent tasks- Service Discovery: Automatic discovery of
AgentCRs via Kubernetes label selectors (Gateway and Orchestrator) - Dynamic Agent Registration: The gateway's
A2A_AGENTSis re-rendered and the pod rolled when the selected set changes, and discovered agents are written to the orchestrator's mountedagents.yaml
- Service Discovery: Automatic discovery of
- Audio: The gateway's Audio API - speech, sound effects and music under
/v1/audio- is served whenspec.audio.enabledis set, and theAudiocolumn ofkubectl get gatewaysshows it- Local speech engine: The built-in
local/qwen3-ttsmodel runs inside the gateway pod. It downloads about 1.4 GB into the pod when it starts and held about 6 GiB of memory per synthesis when measured, two at a time by default, so raisespec.resources.limits.memorywell above the 1Gi default or the pod is killed on the first request
- Local speech engine: The built-in
- Health Checks: Automated health monitoring for external services
- Service: Kubernetes Service configuration (ClusterIP, NodePort, LoadBalancer)
- Routing: Kubernetes Gateway API (
gateway.networking.k8s.io/v1) - operator-managedGateway+HTTPRoute, or attach to a platform-team-managed sharedGatewayviaparentRefs - TLS: Listener-level TLS termination via cert-manager (Gateway API integration)
Gateway, Agent and MCP accept a spec.pod block that shapes the pods of the Deployment each one creates - pod labels and annotations for sidecar injectors (Istio, Linkerd, NetBird), and nodeSelector / tolerations / affinity for scheduling:
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: inference-gateway
spec:
pod:
labels:
sidecar.example.com/inject: "true"
annotations:
linkerd.io/inject: enabled
nodeSelector:
kubernetes.io/hostname: worker-0
tolerations:
- key: gpu
operator: Exists
effect: NoSchedule
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: inference-gatewayThe operator's own app: <name> pod label always wins over spec.pod.labels, so Deployment selectors and Services keep working. Changing or removing a spec.pod field rolls the Deployment.
When there is no injector to do it - a VPN or proxy sidecar such as Tailscale against a Headscale control plane - spec.pod.containers, spec.pod.initContainers and spec.pod.volumes put the sidecar in the pod spec itself:
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: inference-gateway
spec:
pod:
containers:
- name: tailscale
image: tailscale/tailscale:v1.104.4
env:
- name: TS_AUTHKEY
valueFrom:
secretKeyRef:
name: tailnet-key-gateway
key: authkey
securityContext:
privileged: true
volumeMounts:
- name: tailnet-ca
mountPath: /etc/tailnet-ca
readOnly: true
volumes:
- name: tailnet-ca
configMap:
name: tailnet-cacontainers are appended after the operator's own container, which stays first and untouched, and initContainers run before it - or alongside it with restartPolicy: Always for a native sidecar. A container, init container or volume whose name collides with one the operator owns (inference-gateway, agent, mcp, the TLS and ConfigMap volumes it mounts) fails reconciliation with an error naming the field and the name.
-
kubectlversion v1.35.4+ with access to a Kubernetes cluster -
Kubernetes cluster v1.35.4+ (supports both arm64 and amd64 architectures)
-
Kubernetes Gateway API standard-channel CRDs - required even if you never use Gateway API routing. The operator watches
gateway.networking.k8s.io/v1GatewayandHTTPRoute, so without these CRDs the manager fails its cache sync and exits (CrashLoopBackOff):kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml
install.yamldoes not bundle them. v1.5.1 is the version this repo tests against (GATEWAY_API_VERSIONinTaskfile.yaml). Newer standard-channel releases work too. -
agent-sandbox (optional) - only for Orchestrators with
runtime: Sandbox. Install it before the operator starts, or restart the operator afterwards, so it watches Sandboxes. v1.0.8 is the version this repo tests against (AGENT_SANDBOX_VERSIONinTaskfile.yaml):kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v1.0.8/sandbox.yaml
The Inference Gateway Operator supports multiple installation methods. Choose the one that best fits your deployment strategy:
Apply server-side.
spec.pod.containersembeds the Kubernetes container and volume schemas, so each CRD is larger than the 256 KB a client-sidekubectl applycan keep in itslast-applied-configurationannotation. Usekubectl apply --server-side(add--force-conflictswhen upgrading a release installed client-side), andServerSideApply=truein ArgoCD - Flux applies server-side already.
Install the operator and CRDs in one command using the latest release:
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yamlThis command will:
- Create the
inference-gateway-systemnamespace - Install all required Custom Resource Definitions (CRDs)
- Deploy the operator with proper RBAC permissions
- Set up monitoring and metrics collection
For production environments, pin to a specific version:
# Replace <version> with a release tag from https://github.com/inference-gateway/operator/releases
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/download/<version>/install.yamlFor GitOps workflows, use stable manifest URLs:
# ArgoCD Application example
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: inference-gateway-operator
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/inference-gateway/operator
targetRevision: main # tracks the latest release - set a release tag to pin
path: manifests
destination:
server: https://kubernetes.default.svc
namespace: inference-gateway-system
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=trueAt a release tag, manifests/install.yaml references that release's operator image, so pinning targetRevision pins the operator version as well. On main the manifest tracks the most recent release.
Use v0.26.0 or later. Tags before v0.26.0 ship
manifests/install.yamlwithghcr.io/inference-gateway/operator:latest, so pinningtargetRevisionto one of them pins the CRDs but still deploys the current operator image. That image starts the Orchestrator and GPU controllers, whose CRDs those older manifests do not contain, and the manager exits on startup.
For scenarios where you need separate control over the CRD lifecycle (for example, upgrading CRDs independently of the operator):
# Step 1: Install (or upgrade) the CRDs on their own
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/crds.yaml
# Step 2: Install the operator (install.yaml is idempotent and keeps the CRDs in sync)
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yamlBy default, the operator deploys to the inference-gateway-system namespace. To deploy to a custom namespace:
# Download and modify the install.yaml
curl -L https://github.com/inference-gateway/operator/releases/latest/download/install.yaml | \
sed 's/inference-gateway-system/my-custom-namespace/g' | \
kubectl apply --server-side -f -# Clone the repository
git clone https://github.com/inference-gateway/operator.git
cd operator
# Generate manifests for your custom namespace
task manifests-for-namespace NAMESPACE=my-custom-namespace
# Deploy the generated manifests
kubectl apply --server-side -f manifests/my-custom-namespace/install.yamlmanifests/install.yaml hardcodes namespace: inference-gateway-system on every namespaced object
(operator Deployment, ServiceAccount, leader-election Role and RoleBinding, metrics Service) and in
the operator-manager-rolebinding ClusterRoleBinding subjects. A kustomize patch that only renames
the Namespace object therefore leaves the operator in the original namespace. Generate the
manifests instead and point ArgoCD at your own repository:
git clone https://github.com/inference-gateway/operator.git
cd operator
# Rewrites the namespace across the whole manifest
task manifests-for-namespace NAMESPACE=my-custom-namespace
# Commit manifests/my-custom-namespace/ to your GitOps repository# ArgoCD Application pointing at your generated manifests
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: inference-gateway-operator
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/my-org/my-gitops-repo
targetRevision: main
path: manifests/my-custom-namespace
destination:
server: https://kubernetes.default.svc
namespace: my-custom-namespace
syncPolicy:
syncOptions:
- CreateNamespace=true
- ServerSideApply=trueFor development and testing with the latest code:
# Clone the repository
git clone https://github.com/inference-gateway/operator.git
cd operator
# Install CRDs
task install
# Build and deploy operator
task deploy IMG=ghcr.io/inference-gateway/operator:latesttask deploy builds the image with Docker, imports it into the local k3d cluster named dev
(k3d image import <IMG> -c dev) and applies config/environments/dev. It needs Go 1.27.3+,
Docker, and the k3d dev cluster created by task cluster:create - it does not work against an
arbitrary ~/.kube/config context.
Verify the installation:
# Check if the operator is running
kubectl get pods -n inference-gateway-system
# Check if CRDs are installed
kubectl get crd | grep inference-gateway
# View operator logs
kubectl logs -n inference-gateway-system deployment/operator-inference-gateway -fExpected output:
# Pods should show Running status
NAME READY STATUS RESTARTS AGE
operator-inference-gateway-74c9c5f5b-x4d2k 1/1 Running 0 2m
# CRDs should be listed
agents.core.inference-gateway.com 2025-06-21T17:30:00Z
gateways.core.inference-gateway.com 2025-06-21T17:30:00Z
gpus.core.inference-gateway.com 2025-06-21T17:30:00Z
mcps.core.inference-gateway.com 2025-06-21T17:30:00Z
orchestrators.core.inference-gateway.com 2025-06-21T17:30:00ZThe shipped operator Deployment sets WATCH_NAMESPACE_SELECTOR=inference-gateway.com/managed=true, so Gateway, Agent, MCP and GPU resources are only reconciled in namespaces carrying that label. In an unlabeled namespace the operator just logs skipping gateway, namespace does not match WATCH_NAMESPACE_SELECTOR and creates nothing.
Label every namespace you deploy resources into:
kubectl label namespace <namespace> inference-gateway.com/managed=trueThe manifests under examples/ already create their namespaces with this label. Orchestrator resources are not namespace-filtered.
To change the scope, edit WATCH_NAMESPACE_SELECTOR on the operator Deployment - any valid label selector works, and an empty value (or removing the variable) makes the operator watch all namespaces:
# Watch all namespaces
kubectl set env -n inference-gateway-system deployment/operator-inference-gateway WATCH_NAMESPACE_SELECTOR=All Gateway, Agent, MCP and GPU examples below assume their namespace is labeled accordingly.
Create a simple gateway to test the installation:
# Label the target namespace so the operator reconciles it
kubectl label namespace default inference-gateway.com/managed=true
# Create a minimal gateway
cat <<EOF | kubectl apply -f -
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: my-first-gateway
namespace: default
spec:
replicas: 1
environment: development
telemetry:
enabled: true
metrics:
enabled: true
port: 9464
providers:
- name: OpenAI
enabled: true
env:
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: openai-secret
key: api_key
EOFNote: You'll need to create the openai-secret with your API key:
kubectl create secret generic openai-secret \
--from-literal=api_key=your-openai-api-key-hereThe Orchestrator CRD deploys the Inference Gateway CLI's infer daemon: an
LLM-driven loop that listens on chat channels, serves devices over its AG-UI
binding, and fans out to A2A Agents and tools (incl. MCP). Telegram is the
channel currently supported - more channels are planned. Channels, the gateway
and the agent are all optional, but the daemon needs something to run: an
Orchestrator without channels, daemon.binding or tools.schedule (or a
switch for the heartbeat or the A2A server in env) gets Ready=False with
reason InvalidSpec.
When the Telegram channel is enabled, the Deployment is forced to a singleton
(replicas: 1, strategy: Recreate) because Telegram allows only one active
getUpdates consumer per bot token - running two replicas would cause them
to terminate each other with 409 Conflict. For high-availability today, run
multiple Orchestrator resources with different tokens and disjoint
allowedUsers (manual sharding) - webhook mode and shared state would be
required for true scale-out and are out of scope.
# 1. Create the bot credentials secret
kubectl create namespace orchestrators
kubectl create secret generic telegram-bot-credentials -n orchestrators \
--from-literal=token='<TELEGRAM_BOT_TOKEN>' \
--from-literal=allowedUsers='111111111,222222222'
# 2. Apply the Orchestrator resource
cat <<EOF | kubectl apply -f -
apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
name: orchestrator-controlled-by-telegram
namespace: orchestrators
spec:
image: ghcr.io/inference-gateway/cli:latest
channels:
telegram:
enabled: true
tokenSecretRef:
name: telegram-bot-credentials
key: token
allowedUsersSecretRef:
name: telegram-bot-credentials
key: allowedUsers
pollTimeout: 30s
gateway:
url: http://inference-gateway.inference-gateway.svc.cluster.local:8080
agent:
model: deepseek/deepseek-v4-pro
systemPrompt: "You are a helpful assistant running as a Telegram bot."
EOF
# 3. Watch logs (requires CLI built with INFER_LOGGING_STDOUT support)
kubectl logs -n orchestrators deploy/orchestrator-controlled-by-telegram -fA runnable Telegram example with a step-by-step walkthrough is available at
examples/orchestrator/.
spec.gateway takes either url, the address of any gateway, or ref, the name
of a Gateway in the same namespace - one or the other, not both. With ref the
operator addresses that Gateway on its own Service port, so the daemon's
INFER_GATEWAY_URL follows a port change instead of hard-coding one, and it runs
the Gateway with zero replicas while every Orchestrator referencing it is
suspended:
spec:
gateway:
ref: inference-gateway- A Gateway no Orchestrator references keeps its
spec.replicasand its HPA. - A parked Gateway keeps its Deployment, Service and Secrets, and has no HPA while it is parked - an HPA would scale the parked Deployment straight back up. It gets one again when a referencing Orchestrator resumes.
- Resuming an Orchestrator (
spec.sandbox.operatingMode: Running) scales its Gateway back up right away, because the Gateway's controller watches the Orchestrators that reference it. - An Orchestrator whose
refnames no Gateway reportsReady=Falsewith reasonGatewayNotFound.
spec.daemon.binding turns on the daemon's AG-UI binding, the WebSocket the
chat TUI, the browser extension and other daemons sync sessions over. The
binding listens on every interface, so it always serves TLS from
tls.secretName (tls.crt and tls.key, plus ca.crt with
clientAuth: true). Clients authenticate with the token from tokenSecretRef,
a client certificate, or both, and the API rejects a binding with neither. The
controller adds a <name>-binding Service on the binding port (default 52789)
and holds the pod back from Ready until the binding listens.
spec.daemon.binding.service.annotations and .labels are copied verbatim onto
that Service for other controllers to act on (a VPN operator such as Tailscale
with tailscale.com/expose: "true", external-dns, a service mesh). The operator
ships no such keys itself, and metadata those controllers add to the Service
survives a reconcile.
The certificate must name the host clients dial: daemon-binding.orchestrators.svc
in the cluster, or localhost through a port-forward. See
examples/daemon/ for an Orchestrator that only serves the
binding.
spec.runtime: Sandbox runs the daemon in an agent-sandbox Sandbox instead of a
Deployment: a stable pod with two persistent volumes, the daemon's workspace
(/workspace) and its state (~/.infer/projects), that outlive a suspension.
It needs agent-sandbox installed. Without it, the Orchestrator
reports Ready=False with reason SandboxCRDMissing and everything else keeps
running. spec.runtime is fixed at creation, because switching away from a
Sandbox would delete its volumes.
apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
name: sandboxed-daemon
namespace: orchestrators
spec:
image: ghcr.io/inference-gateway/cli:latest
runtime: Sandbox
sandbox:
storage: 5Gi
retention: 24h
runtimeClassName: gvisor
resources:
limits:
cpu: "2"
memory: 4Gi
daemon:
binding:
tokenSecretRef:
name: daemon-binding
key: token
tls:
secretName: daemon-binding-tlsspec.sandbox.operatingMode: Suspendeddeletes the pod and keeps the volumes.Runningresumes it on a fresh pod once the suspension completed. A linked Gateway is scaled down and up with it.- The daemon idles out by exiting 0 (
INFER_DAEMON_IDLE_EXITinspec.env). The operator then parks the Orchestrator itself: it setsoperatingMode: Suspended, and a client resumes the daemon by settingRunning. A daemon that writesagents-done=trueto its termination message has every agent done, which startsspec.sandbox.retention: the Orchestrator is deleted that long after, unless a client resumed it first. spec.sandbox.shutdownTimeexpires the Sandbox at that time, and the operator deletes the Orchestrator with the Sandbox, the volumes and everything else it owns once agent-sandbox reportsSandboxExpired.- A change to the daemon's pod (its image, env, binding or discovered agents) cycles
the pod through
Suspended, because agent-sandbox only applies a new template to the pod it creates next. A pod that fails is recreated by agent-sandbox with a backoff (failedPodPolicy: Recreate). spec.sandbox.storage(default5Gi) sizes both volumes and is fixed at creation.spec.sandbox.volumesmounts PersistentVolumeClaims that already exist in the Orchestrator's namespace into the daemon's container - an app installed once, or content its sessions share. See Mount existing claims.status.podNamenames the Sandbox's pod once it is ready, for clients that port-forward to it.
A Sandbox's own volumes start empty, so an app installed by spec.userData is
downloaded again on every pod. spec.sandbox.volumes mounts claims that are
created and filled elsewhere:
spec:
runtime: Sandbox
sandbox:
volumes:
- name: resolve
claimName: davinci-resolve
mountPath: /opt/resolve
- name: footage
claimName: raw-footage
mountPath: /mnt/footage
readOnly: truereadOnly: truemounts the claim read-only, for content the agents copy into/workspacebefore they change it. The original stays as it is.- The Sandbox never owns a mounted claim. A suspension,
spec.sandbox.retention,spec.sandbox.shutdownTimeand deleting the Orchestrator all leave it and its data in place. - A
mountPathis absolute and cannot be, or lie under, a path the operator mounts:/workspace,/home/infer/.infer/projectsand/var/lib/cloud/seed/nocloud. Two volumes cannot share one. - Changing the list cycles a running pod through
Suspended, like any other change to the daemon's pod. - A claim that does not exist leaves the pod unscheduled. The Orchestrator
reports it as
Ready=Falsewith reasonSandboxNotReadyand the scheduler's message. - A
ReadWriteOnceclaim is shared by pods on one node only. Orchestrators on several nodes need aReadWriteManyorReadOnlyManyclaim. - A pod that mounts a claim sets
fsGroupChangePolicy: OnRootMismatch, so a large volume is not walked on every boot.
For an app alone, an image with the app installed needs none of this - a node pulls it once and caches it.
The pod follows agent-sandbox's
threat model
for untrusted, LLM-driven code: it has no Kubernetes API access
(automountServiceAccountToken: false), runs as a non-root user without
capabilities or privilege escalation under the RuntimeDefault seccomp profile,
and spec.sandbox.runtimeClassName runs it under gVisor or Kata. Set
spec.resources limits too. The operator creates no NetworkPolicy, because the
daemon reaches the inference gateway, MCP servers and A2A agents inside the
cluster. Restrict its traffic with one of your own, for example:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: sandboxed-daemon
namespace: orchestrators
spec:
podSelector:
matchLabels:
app: sandboxed-daemon
policyTypes:
- Ingress
- Egress
ingress:
- ports:
- port: 52789
egress:
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: inference-gateway
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16spec.userData hands the pod a cloud-init
user-data script that runs as root when it boots, before the daemon starts, the
way a cloud VM runs it - for example to install a browser on a computer-use image.
The Secret key is mounted read-only for root at cloud-init's NoCloud seed path,
/var/lib/cloud/seed/nocloud/user-data. The image runs it and then drops to the
daemon's user, as ghcr.io/inference-gateway/computer does.
spec:
image: ghcr.io/inference-gateway/computer:latest
runtime: Sandbox
userData:
secretRef:
name: computer-user-data
key: userdataThe pod then boots as root in a
user namespace
(hostUsers: false), so its root maps to an unprivileged user on the node and keeps
only the capabilities a package install needs (CHOWN, DAC_OVERRIDE, FOWNER,
FSETID, SETUID, SETGID), still without privilege escalation. Nodes need
Kubernetes 1.36 or newer with containerd 2.0+ or CRI-O 1.25+ and Linux 6.3+. The
secretRef has the shape of KubeVirt's cloudInitNoCloud volume, which reads the
key userdata, so the same Secret boots a VM unchanged.
To upgrade the operator to a newer version:
# Upgrade to latest version
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yamlThe operator Deployment uses the Recreate strategy, so an upgrade terminates the running operator
pod before starting the new one. Existing Gateway, Agent, MCP, Orchestrator and GPU workloads keep
serving during that gap - only reconciliation pauses until the new pod is ready.
To completely remove the operator:
# Delete all Gateway instances first
kubectl delete gateway --all --all-namespaces
# Uninstall the operator
kubectl delete -f https://github.com/inference-gateway/operator/releases/latest/download/install.yamlThe operator supports multi-architecture deployments:
- linux/amd64 - Intel/AMD 64-bit processors
- linux/arm64 - ARM 64-bit processors (Apple Silicon, AWS Graviton, etc.)
Container images are automatically selected based on your cluster's node architecture.
Every
Gateway,Agent,MCPandGPUexample below only reconciles if its namespace is labeledinference-gateway.com/managed=true- see Namespace Scoping.
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: simple-gateway
namespace: default
spec:
replicas: 1
environment: development
telemetry:
enabled: true
metrics:
enabled: true
port: 9464
providers:
- name: OpenAI
enabled: true
env:
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: openai-secret
key: api_keyapiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: production-gateway
namespace: inference-gateway
spec:
replicas: 3
image: "ghcr.io/inference-gateway/inference-gateway:0.23.6"
environment: production
auth:
enabled: true
provider: oidc
oidc:
issuerUrl: "https://auth.company.com/realms/ai"
clientId: "inference-gateway"
audiences:
- "api://inference-gateway"
clientSecretRef:
name: auth-secrets
key: client-secret
providers:
- name: OpenAI
enabled: true
env:
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: ai-secrets
key: openai-key
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "2000m"
memory: "2Gi"
gatewayAPI:
enabled: true
gateway:
gatewayClassName: envoy
tls:
enabled: true
issuer: letsencrypt-prod
secretName: ai-gateway-tls
httpRoute:
hostnames:
- "ai-gateway.company.com"Both the Gateway and the Orchestrator discover Agent CRs by label
selector. The Gateway unions them with its static spec.a2a.agents[] and
renders A2A_AGENTS for the gateway pod, the Orchestrator writes them into
~/.infer/agents.yaml inside its pod.
With spec.a2a.enabled, the gateway becomes an A2A server: it publishes one
merged card at GET /.well-known/agent-card.json and relays every POST /a2a
call to the agent the request names.
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: gateway-with-a2a
namespace: default
spec:
a2a:
# Turns the gateway into an A2A server (A2A_ENABLED).
enabled: true
# Static agents are {name, url} objects, kept alongside discovered ones.
# (The Orchestrator takes plain URL strings instead.)
agents:
- name: static-agent
url: "http://static-agent.agents.svc.cluster.local:8080"
# Automatic discovery of Agent CRs by label selector.
serviceDiscovery:
enabled: true
namespace: "agents" # Namespace to search (defaults to the Gateway's own namespace)
selector:
matchLabels:
agent-group: group1
# Canonical public /a2a URL published on the agent card and as the RFC 9728
# `resource` (A2A_RESOURCE_URL). Defaults to
# "<scheme>://<first httpRoute hostname>/a2a".
resourceUrl: "https://ai-gateway.company.com/a2a"
timeouts:
client: "30s" # A2A_CLIENT_TIMEOUT - one non-streaming call, including the card fetch
streamIdle: "5m" # A2A_STREAM_IDLE_TIMEOUT - idle cutoff for a relayed stream, "0" disables
# How often agent cards are re-fetched in the background
# (A2A_CARD_REFRESH_INTERVAL) - "0" disables the refresh.
cardRefreshInterval: "5m"Agent alias rule: each entry is rendered into A2A_AGENTS as
<name>=<url>, and that name prefixes the agent's skill ids on the gateway card
and every task id the gateway hands out. To be used as an alias the name must
match ^[a-z0-9_-]+$ and be unique across static and discovered agents.
Otherwise the entry is rendered as a bare URL and the gateway derives the alias
from the host. Discovered agents are named after metadata.name and use
status.card.url, falling back to the in-cluster Service URL while status is
still empty. Entries are deduped on URL and sorted for determinism.
The Orchestrator configures the same discovery with plain URL strings:
apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
name: orchestrator-with-service-discovery
namespace: default
spec:
image: ghcr.io/inference-gateway/cli:latest
channels:
telegram:
enabled: true
tokenSecretRef:
name: telegram-bot-credentials
key: token
gateway:
url: http://inference-gateway.inference-gateway.svc.cluster.local:8080
agent:
model: deepseek/deepseek-v4-pro
a2a:
enabled: true
# Static agent URLs are kept alongside discovered ones.
agents:
- "http://static-agent.agents.svc.cluster.local:8080"
# Automatic discovery of Agent CRs by label selector.
serviceDiscovery:
enabled: true
namespace: "agents" # Namespace to search (defaults to the Orchestrator's own namespace)
selector:
matchLabels:
agent-group: group1Service Discovery Features:
- Automatic Agent Discovery: Discovers
AgentCRs based on their Kubernetes labels - Dynamic Configuration: The gateway's
A2A_AGENTSis re-rendered and the orchestrator's mountedagents.yamlrewritten when the set changes, rolling the pod - Label-Based Selection: Uses a configurable label selector to identify
AgentCRs - Namespace Scoping: Can search for
AgentCRs in a specific namespace (defaults to the Gateway's or Orchestrator's own namespace)
Agent Requirements:
For an Agent CR to be discovered, it must carry labels matching the selector:
apiVersion: core.inference-gateway.com/v1alpha1
kind: Agent
metadata:
name: my-agent
namespace: agents
labels:
agent-group: group1 # matches spec.a2a.serviceDiscovery.selector
spec:
# Agent configurationWhen spec.mcp.expose is on, the gateway serves OAuth 2.0 Protected Resource
Metadata at GET /.well-known/oauth-protected-resource/mcp, so an MCP client
with no token discovers the IdP from the 401 on /mcp (required by MCP
2026-07-28). The operator-managed HTTPRoute matches the / path prefix, so
that document is routed to the gateway Service wherever /mcp is - no extra
route needed.
spec.mcp.resourceUrl pins the canonical public /mcp URL the gateway puts in
the document's resource and in the challenge's resource_metadata. It is
emitted as MCP_RESOURCE_URL:
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: gateway-with-mcp-auth
namespace: inference-gateway
spec:
mcp:
enabled: true
expose: true
resourceUrl: "https://api.example.com/mcp"
gatewayAPI:
enabled: true
gateway:
tls:
enabled: true
httpRoute:
hostnames:
- api.example.comWhen resourceUrl is omitted the operator defaults it to
<scheme>://<first httpRoute hostname>/mcp (https when
gatewayAPI.gateway.tls.enabled), so the example above works without the
explicit value. Nothing is emitted when routing is disabled or the hostname is
a wildcard, and the gateway then derives the URL from the inbound request
scheme (honouring X-Forwarded-Proto) and Host - set resourceUrl
explicitly whenever the ingress rewrites either, or clients are handed a URL
they cannot reach.
MCP Custom Resources can be discovered automatically by both Gateway and
Orchestrator via the same label-selector pattern used for A2A agents. Opt-in
on the MCP side is via plain metadata.labels - no extra field on MCPSpec.
Gateway-side discovery - discovered MCPs are unioned with the static
spec.mcp.servers[] list, deduped on URL, sorted, and exposed via the gateway
pod's MCP_SERVERS env var as name=url entries. The name (spec.mcp.servers[].name
for static servers, metadata.name for discovered MCP CRs) becomes the tool
namespace, so tools reach the model as mcp_<name>_<tool> - for example
mcp_time_get_current_time rather than a long host-derived alias. To be used as an
alias a name must match ^[a-z0-9_-]+$, be unique across static and discovered
servers, and must not be the reserved alias tools. A name that fails these rules is
rendered as a bare URL and the gateway falls back to deriving the alias from the host.
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
name: gateway-with-mcp-discovery
namespace: inference-gateway
spec:
mcp:
enabled: true
# Static entries still work alongside discovery.
servers:
- name: external-mcp
url: "https://mcp.example.com/mcp"
serviceDiscovery:
enabled: true
namespace: mcp # defaults to the Gateway's own namespace
selector:
matchLabels:
mcp-group: group1Orchestrator-side discovery - discovered MCPs are rendered into an
mcp.yaml ConfigMap mounted at ~/.infer/mcp.yaml inside the orchestrator
pod. A content-hash pod annotation rolls the singleton when the set changes:
apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
name: orchestrator
namespace: orchestrator
spec:
image: ghcr.io/inference-gateway/cli:latest
channels:
telegram:
enabled: true
tokenSecretRef:
name: telegram-bot-credentials
key: token
gateway:
url: http://inference-gateway.inference-gateway.svc.cluster.local:8080
agent:
model: deepseek/deepseek-v4-pro
mcp:
enabled: true
servers: [] # {name, url, headers, includeTools, excludeTools, timeoutSeconds} entries, optional
serviceDiscovery:
enabled: true
namespace: mcp
selector:
matchLabels:
mcp-group: group1MCP CR labeling - tag each MCP you want discovered with a matching label:
apiVersion: core.inference-gateway.com/v1alpha1
kind: MCP
metadata:
name: my-mcp
namespace: mcp
labels:
mcp-group: group1 # picked up by both selectors above
spec:
image: ghcr.io/your-org/your-mcp-server:latest
server:
port: 8080
# Endpoint path the operator appends when building the in-cluster URL.
# Defaults to "/mcp". Override if your server listens elsewhere.
path: "/mcp"MCP servers by URL - spec.mcp.servers lists servers the pod reaches by URL, each with a name,
a url and optional headers, as a Gateway's do. They are written into the pod's mcp.yaml with or
without discovery. Header values are written as given, so a ${VAR} stays a reference the infer CLI in
the pod expands from its environment. Fill it from a Secret through spec.env, and never put a secret
value in a header:
spec:
env:
- name: MCP_EXAMPLE_API_KEY
valueFrom:
secretKeyRef:
name: mcp-example-credentials
key: MCP_EXAMPLE_API_KEY
mcp:
enabled: true
servers:
- name: example
url: https://mcp.example.com/mcp
headers:
x-api-key: ${MCP_EXAMPLE_API_KEY}A server can also narrow and bound what the pod gets from it. includeTools limits the pod to the
listed tools, named as the server lists them, and excludeTools hides the listed ones. When both are
set, includeTools wins. timeoutSeconds bounds one request to the server, and unset the pod's default
applies:
servers:
- name: example
url: https://mcp.example.com/mcp
includeTools:
- search
timeoutSeconds: 10The pod's infer image must be v0.248.0 or newer, which reads url and headers.
Visibility:
kubectl get gateway # MCPS column shows static + discovered MCP servers,
# A2A column (.status.a2aAgentCount) static + discovered A2A agents,
# Audio column (.spec.audio.enabled) whether the Audio API is on
kubectl get orchestrator # MCPS column shows the discovered countFor a runnable MCP server to discover, see
examples/mcp-server/.
Each example shows one feature. kubectl explain gateway.core.inference-gateway.com.spec documents every field.
examples/gateway-with-guardrails/- OPA/Rego guardrails (spec.guardrails)examples/gateway-with-mtls/- server TLS, inbound and outbound mutual TLS (spec.server.tls,spec.client.tls)examples/gateway-with-model-routing/- model aliases with round-robin routing (spec.routing)examples/agent-with-mcp/- an Agent's MCP client (spec.mcp) and advertised URL (spec.card.url)examples/orchestrator-telemetry-receiver/- an Orchestrator OTLP receiver (spec.telemetry.receiver)
The operator includes comprehensive validation:
- Replica limits: 1-100 replicas
- Port ranges: Valid port numbers (1024-65535 for server ports)
- Environment values: Restricted to development, staging, production
- Resource limits: Proper CPU/memory specifications
Check Gateway status and health:
# Check gateway resources
kubectl get gateways -A
# Get detailed gateway status
kubectl describe gateway my-first-gateway
# Check generated resources (the Deployment and Service are named after the
# Gateway - the routing ConfigMap, HTTPRoute, upstream Gateway and HPA are the
# ones labeled app=<gateway-name>)
kubectl get deployment,service my-first-gateway
kubectl get configmaps,httproutes,hpa -l app=my-first-gateway
# View Gateway logs
kubectl logs -l app=my-first-gateway -fstatus reports:
url: the address the gateway is reachable at (HTTPRoute hostname when Gateway API routing is enabled, otherwise the in-cluster service FQDN)providerSummary: comma-separated list of configured providersserviceAccountName: the ServiceAccount the gateway pods run asmcpServers: sorted<name>=<url>entries the pod is configured withmcpServerCount: number of static plus discovered MCP serversa2aAgents: sorted<name>=<url>entries mirroringA2A_AGENTSa2aAgentCount: number of static plus discovered A2A agents
For replica counts and rollout health, check the Deployment directly
(kubectl get deployment my-first-gateway) - the Gateway status does not
mirror them.
It depends on your installation method:
-
One-command installation (Recommended): No! CRDs are included automatically:
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml
-
GitOps/ArgoCD installations: No! Use the
manifests/directory which includes CRDs:source: repoURL: https://github.com/inference-gateway/operator path: manifests # Includes both CRDs and operator
-
Custom namespace installations: No! When you substitute the namespace in
install.yaml, the CRDs are still included:curl -L https://github.com/inference-gateway/operator/releases/latest/download/install.yaml | \ sed 's/inference-gateway-system/my-namespace/g' | kubectl apply --server-side -f -
-
Advanced scenarios: You can install CRDs separately for more control:
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/crds.yaml
The operator supports both arm64 and amd64 architectures:
- Container images are built for both platforms
- Kubernetes automatically selects the correct image for your nodes
- Works on Apple Silicon (M1/M2), AWS Graviton, Intel/AMD processors
Run these commands to verify your installation:
# 1. Check operator pods
kubectl get pods -n inference-gateway-system
# 2. Verify CRDs are installed
kubectl get crd | grep inference-gateway
# 3. Test creating a Gateway resource
kubectl get gateways --all-namespacesYes! The operator defaults to inference-gateway-system but can be deployed to any namespace:
Quick Method:
curl -L https://github.com/inference-gateway/operator/releases/latest/download/install.yaml | \
sed 's/inference-gateway-system/my-namespace/g' | \
kubectl apply --server-side -f -Development Method:
# Generate manifests for custom namespace
task manifests-for-namespace NAMESPACE=my-namespace
kubectl apply --server-side -f manifests/my-namespace/install.yamlGitOps Method: Kustomize namespace patches do not work - the upstream install.yaml hardcodes namespace: inference-gateway-system on every namespaced object and in the operator-manager-rolebinding subjects. Generate the manifests with task manifests-for-namespace NAMESPACE=my-namespace, commit manifests/my-namespace/ to your GitOps repository, and point ArgoCD or Flux at that directory. See Method 5 Option C.
Simply reapply the installation with a newer version:
# Upgrade to latest
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yamlThe operator Deployment uses the Recreate strategy, so the old operator pod is terminated before
the new one starts. Running Gateway instances are unaffected - reconciliation pauses briefly.
Your Gateway resources will remain in the cluster but will no longer be managed. To completely clean up:
# 1. Delete all Gateway instances first
kubectl delete gateway --all --all-namespaces
# 2. Then uninstall the operator
kubectl delete -f https://github.com/inference-gateway/operator/releases/latest/download/install.yamlThe recommended way to get a complete toolchain is via the project's Flox environment - flox activate provides Go, task, kubectl, kubebuilder, kustomize, golangci-lint, k3d, ctlptl, gh, and prettier at the pinned versions used by CI. See CONTRIBUTING.md for alternatives (flox environment or manual install).
If installing manually:
- Task runner (
task) - Go 1.27+
- Docker
- k3d (via
ctlptl) or similar local Kubernetes cluster
# Run tests
task test
# Run linting
task lint
# Generate code and manifests (including install.yaml)
task generate manifests
# Build locally
task build
# Run against local cluster
task runNote: The task manifests command automatically generates:
- CRDs in
config/crd/bases/ - Installation manifests in
manifests/install.yaml - CRD-only manifests in
manifests/crds.yaml
These files are version-controlled and included in releases.
The operator includes comprehensive unit and integration tests:
# Run all tests
task test
# Run e2e tests (requires running cluster)
task test:e2eThe Inference Gateway provides enterprise-grade observability through OpenTelemetry:
Metrics Collection:
# Access Prometheus metrics. The metrics port is exposed on the Service named
# after the Gateway, and only when both telemetry.enabled and
# telemetry.metrics.enabled are true.
curl http://my-gateway:9464/metrics
# Key metrics include:
# - llm_requests_total: Request counts by provider/model
# - llm_tokens_*: Token usage tracking
# - llm_request_duration_seconds: Request latency histograms
# - llm_latency_*: Detailed timing breakdownsDistributed Tracing:
# Configure OTLP trace export - OTLP is the only supported exporter
telemetry:
enabled: true
traces:
exporter:
otlp:
endpoint: "http://otel-collector:4318"
protocol: "http/protobuf"The Gateway ignores protocol and always exports OTLP over HTTP, so
endpoint must point at an OTLP/HTTP receiver such as
http://otel-collector:4318. A gRPC endpoint like :4317 receives OTLP/HTTP
and no spans arrive. (protocol is forwarded only for Agent workloads, as
A2A_OTEL_EXPORTER_OTLP_PROTOCOL.)
TELEMETRY_TRACING_ENABLED and TELEMETRY_TRACING_OTLP_ENDPOINT are only set
on the gateway container when telemetry.enabled is true and
telemetry.traces.exporter.otlp is present. Non-OTLP collector endpoints (for
example Jaeger's :14268/api/traces) are not supported - point the exporter at
an OTLP receiver instead.
Supported Backends:
- Metrics: Prometheus, Grafana, any OpenTelemetry-compatible backend
- Tracing: Jaeger, Zipkin, Lightstep, Honeycomb, Datadog (via OTLP)
- Logs: Structured JSON with trace correlation for any log aggregation system
# Check gateway status
kubectl get gateway my-gateway -o yaml
# Check deployment health
kubectl get deployment my-gateway
# Check service endpoints
kubectl get service my-gateway
# View the inline model-routing config (only created when spec.routing.config is set)
kubectl get configmap my-gateway-routing -o yamlCommon issues and solutions:
- Nothing happens after applying a Gateway/Agent/MCP/GPU: the namespace is missing the
inference-gateway.com/managed=truelabel - see Namespace Scoping - Operator pod in CrashLoopBackOff: the Kubernetes Gateway API standard CRDs are missing - see Prerequisites
- Gateway not starting: Check image pull policy and secrets
- Authentication failures: Verify OIDC configuration and secrets
- Provider connection issues: Check network policies and secret references
- Resource constraints: Review resource requests/limits
The operator supports rolling upgrades:
- Update the Gateway spec with new image version
- Operator automatically performs rolling update
- Monitor status for completion
Delete gateway instances:
kubectl delete gateway --allUninstall operator:
task undeploy
task uninstallFor complete API reference, see the generated CRD documentation or use:
kubectl explain gateway.spec
kubectl explain gateway.spec.providers
kubectl explain gateway.spec.auth
# etc.- Fork the repository
- Create a feature branch
- Make changes with tests
- Run
task lint test - Submit a pull request
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.