Skip to content
inference-gatewayPublic

About

This project provides a Kubernetes Operator for managing the lifecycle of the inference-gateway and its related components. It simplifies deployment, configuration, and scaling of the gateway within Kubernetes clusters, enabling seamless integration of inference workflows.

Topics

Resources

Contributing

Security policy

Stars

2 stars

Watchers

1 watching

Forks

Latest commit

Β 

History

465 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Inference Gateway Operator

⚠️ EARLY STAGE PROJECT WARNING
This project is currently in its early development stages. Breaking changes are expected and the API may change significantly between releases. Use with caution in production environments and expect potential migration requirements when upgrading versions.

A Kubernetes operator for automating the deployment and management of Inference Gateway instances

Go Version License: Apache 2.0 Kubernetes Docker OpenAPI

Latest Release Container Registry Multi-Arch Build Status Tests Lint


A Kubernetes operator for automating the deployment and management of Inference Gateway instances on Kubernetes.

Description

This Kubernetes operator extends the Kubernetes API to create, configure and manage Inference Gateway instances within a Kubernetes cluster. It provides a comprehensive CRD (Custom Resource Definition) that allows you to declaratively manage:

  • Gateway Deployment: Automated deployment with configurable replicas, resources, and Horizontal Pod Autoscaling (HPA)
  • AI Provider Integration: Support for OpenAI, Anthropic, Ollama, and other AI/ML providers
  • Authentication & Authorization: OIDC integration with configurable identity providers
  • Model Context Protocol (MCP): Integration with MCP servers for extended AI capabilities
  • Agent-to-Agent (A2A): Support for distributed agent communication and orchestration
  • GPU Runtimes: Lease externally hosted, GPU-backed inference runtimes (RunPod today) via a pluggable provider interface, exposed as an HTTP endpoint and connection Secret (examples/gpu)
  • Observability: Built-in metrics, tracing, and health monitoring
  • Network Configuration: Service and Gateway API routing (Gateway + HTTPRoute) with listener-level TLS

The operator follows cloud-native best practices and provides a unified control plane for managing both the gateway infrastructure and its associated AI workloads.

✨ Key Features

πŸš€ Deployment πŸ” Security πŸ“Š Observability πŸ”— Integration
Auto-scaling OIDC Auth Prometheus Metrics MCP Protocol
Rolling Updates TLS/SSL Distributed Tracing A2A Agents
Health Checks Secret Management Status Monitoring Multiple Providers
Resource Management Network Policies Logging Custom Extensions

πŸ€– Supported AI Providers:

  • Anthropic β€’ Cloudflare β€’ Cohere β€’ DeepSeek β€’ ElevenLabs β€’ Google β€’ Groq β€’ llama.cpp β€’ MiniMax β€’ Mirelo β€’ Mistral β€’ Moonshot β€’ NVIDIA β€’ Ollama β€’ Ollama Cloud β€’ OpenAI β€’ Webwright β€’ Z.ai

☸️ Kubernetes Native:

  • CRDs β€’ Controller Pattern β€’ RBAC β€’ Service Mesh Ready

πŸ“š Table of Contents

πŸ“‹ API Overview

The Gateway CRD supports the following key configuration areas:

Core Configuration

  • Replicas: Number of gateway instances (1-100)
  • Image: Container image and version
  • Environment: Deployment environment (development, staging, production)
  • Resources: CPU and memory requests/limits

Auto-scaling (HPA)

  • Horizontal Pod Autoscaler: Automatic scaling based on CPU, memory, or custom metrics
  • Min/Max Replicas: Configurable scaling boundaries
  • Multiple Metrics: CPU utilization, memory utilization, custom metrics support
  • Stabilization Windows: Fine-tuned scaling behavior control

Observability (Telemetry)

Powered by OpenTelemetry for industry-standard observability:

  • Metrics: Prometheus metrics endpoint with OpenTelemetry SDK
    • Request counts, durations, error rates
    • Token usage tracking (prompt, completion, total)
    • Provider-specific metrics by model
    • Custom histogram boundaries for latency analysis
  • Tracing: Distributed tracing with OpenTelemetry exporters
    • OTLP trace export to Jaeger, Zipkin, or any OTLP-compatible backend
    • Request flow visualization across services
    • Performance bottleneck identification
  • Logs: Structured logging with correlation IDs for distributed tracing

Authentication

  • OIDC: OpenID Connect is the only authentication method the gateway implements. The operator maps spec.auth.enabled plus the spec.auth.oidc fields issuerUrl, clientId, audiences, clientSecretRef and caCertRef onto the gateway container.
  • spec.auth.provider currently has no effect - the Gateway controller never reads it, so setting jwt or basic still yields OIDC.

AI Providers

Support for multiple AI/ML providers with flexible configuration. The list is generated from the canonical inference-gateway/schemas OpenAPI enum into internal/providers/zz_generated_providers.go, so it never drifts from the gateway:

providers[].name Provider
anthropic Claude API
cloudflare Cloudflare Workers AI
cohere Command and embedding models
deepseek Cost-effective reasoning models
elevenlabs Speech and audio models
google Google AI / Gemini
groq Fast inference with open models
llamacpp llama.cpp server
minimax MiniMax models
mirelo Mirelo AI sound-effect generation
mistral Mistral AI
moonshot Moonshot AI (Kimi)
nvidia NVIDIA NIM
ollama Local model serving
ollama_cloud Ollama Cloud
openai OpenAI API
webwright Webwright AI
zai Z.ai

The gateway reads <PROVIDER_ID>_API_URL and <PROVIDER_ID>_API_KEY only for these ids - there is no catch-all custom provider. To reach another OpenAI-compatible endpoint (vLLM, a proxy), override the URL of an OpenAI-compatible id such as openai, including its /v1 base path. See examples/gpu for a worked example.

providers[].name has no CRD enum: it is validated at runtime and matched case-insensitively, so OpenAI, openai and OPENAI are equivalent. A name outside the list above is skipped - it is left out of status.providerSummary.

Extensions

  • MCP (Model Context Protocol): Integration with MCP servers for tool access
    • Service Discovery: Automatic discovery of MCP CRs via Kubernetes label selectors (Gateway and Orchestrator)
    • Dynamic Updates: The pod's MCP_SERVERS / mounted mcp.yaml is rebuilt and rolled when the discovered set changes
  • A2A (Agent-to-Agent): Distributed agent communication - the Gateway relays A2A calls when spec.a2a.enabled is set, the Orchestrator drives multi-turn agent tasks
    • Service Discovery: Automatic discovery of Agent CRs via Kubernetes label selectors (Gateway and Orchestrator)
    • Dynamic Agent Registration: The gateway's A2A_AGENTS is re-rendered and the pod rolled when the selected set changes, and discovered agents are written to the orchestrator's mounted agents.yaml
  • Audio: The gateway's Audio API - speech, sound effects and music under /v1/audio - is served when spec.audio.enabled is set, and the Audio column of kubectl get gateways shows it
    • Local speech engine: The built-in local/qwen3-tts model runs inside the gateway pod. It downloads about 1.4 GB into the pod when it starts and held about 6 GiB of memory per synthesis when measured, two at a time by default, so raise spec.resources.limits.memory well above the 1Gi default or the pod is killed on the first request
  • Health Checks: Automated health monitoring for external services

Networking

  • Service: Kubernetes Service configuration (ClusterIP, NodePort, LoadBalancer)
  • Routing: Kubernetes Gateway API (gateway.networking.k8s.io/v1) - operator-managed Gateway + HTTPRoute, or attach to a platform-team-managed shared Gateway via parentRefs
  • TLS: Listener-level TLS termination via cert-manager (Gateway API integration)

Pod Customization and Scheduling

Gateway, Agent and MCP accept a spec.pod block that shapes the pods of the Deployment each one creates - pod labels and annotations for sidecar injectors (Istio, Linkerd, NetBird), and nodeSelector / tolerations / affinity for scheduling:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  pod:
    labels:
      sidecar.example.com/inject: "true"
    annotations:
      linkerd.io/inject: enabled
    nodeSelector:
      kubernetes.io/hostname: worker-0
    tolerations:
      - key: gpu
        operator: Exists
        effect: NoSchedule
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: inference-gateway

The operator's own app: <name> pod label always wins over spec.pod.labels, so Deployment selectors and Services keep working. Changing or removing a spec.pod field rolls the Deployment.

When there is no injector to do it - a VPN or proxy sidecar such as Tailscale against a Headscale control plane - spec.pod.containers, spec.pod.initContainers and spec.pod.volumes put the sidecar in the pod spec itself:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  pod:
    containers:
      - name: tailscale
        image: tailscale/tailscale:v1.104.4
        env:
          - name: TS_AUTHKEY
            valueFrom:
              secretKeyRef:
                name: tailnet-key-gateway
                key: authkey
        securityContext:
          privileged: true
        volumeMounts:
          - name: tailnet-ca
            mountPath: /etc/tailnet-ca
            readOnly: true
    volumes:
      - name: tailnet-ca
        configMap:
          name: tailnet-ca

containers are appended after the operator's own container, which stays first and untouched, and initContainers run before it - or alongside it with restartPolicy: Always for a native sidecar. A container, init container or volume whose name collides with one the operator owns (inference-gateway, agent, mcp, the TLS and ConfigMap volumes it mounts) fails reconciliation with an error naming the field and the name.

πŸš€ Quick Start

Prerequisites

  • kubectl version v1.35.4+ with access to a Kubernetes cluster

  • Kubernetes cluster v1.35.4+ (supports both arm64 and amd64 architectures)

  • Kubernetes Gateway API standard-channel CRDs - required even if you never use Gateway API routing. The operator watches gateway.networking.k8s.io/v1 Gateway and HTTPRoute, so without these CRDs the manager fails its cache sync and exits (CrashLoopBackOff):

    kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

    install.yaml does not bundle them. v1.5.1 is the version this repo tests against (GATEWAY_API_VERSION in Taskfile.yaml). Newer standard-channel releases work too.

  • agent-sandbox (optional) - only for Orchestrators with runtime: Sandbox. Install it before the operator starts, or restart the operator afterwards, so it watches Sandboxes. v1.0.8 is the version this repo tests against (AGENT_SANDBOX_VERSION in Taskfile.yaml):

    kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v1.0.8/sandbox.yaml

πŸ“¦ Installation

The Inference Gateway Operator supports multiple installation methods. Choose the one that best fits your deployment strategy:

Apply server-side. spec.pod.containers embeds the Kubernetes container and volume schemas, so each CRD is larger than the 256 KB a client-side kubectl apply can keep in its last-applied-configuration annotation. Use kubectl apply --server-side (add --force-conflicts when upgrading a release installed client-side), and ServerSideApply=true in ArgoCD - Flux applies server-side already.

Method 1: One-Command Installation (Recommended)

Install the operator and CRDs in one command using the latest release:

kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

This command will:

  • Create the inference-gateway-system namespace
  • Install all required Custom Resource Definitions (CRDs)
  • Deploy the operator with proper RBAC permissions
  • Set up monitoring and metrics collection

Method 2: Specific Version Installation

For production environments, pin to a specific version:

# Replace <version> with a release tag from https://github.com/inference-gateway/operator/releases
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/download/<version>/install.yaml

Method 3: GitOps/ArgoCD-Friendly Installation

For GitOps workflows, use stable manifest URLs:

# ArgoCD Application example
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-gateway-operator
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/inference-gateway/operator
    targetRevision: main # tracks the latest release - set a release tag to pin
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: inference-gateway-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

At a release tag, manifests/install.yaml references that release's operator image, so pinning targetRevision pins the operator version as well. On main the manifest tracks the most recent release.

Use v0.26.0 or later. Tags before v0.26.0 ship manifests/install.yaml with ghcr.io/inference-gateway/operator:latest, so pinning targetRevision to one of them pins the CRDs but still deploys the current operator image. That image starts the Orchestrator and GPU controllers, whose CRDs those older manifests do not contain, and the manager exits on startup.

Method 4: Separate CRD Installation (Advanced)

For scenarios where you need separate control over the CRD lifecycle (for example, upgrading CRDs independently of the operator):

# Step 1: Install (or upgrade) the CRDs on their own
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/crds.yaml

# Step 2: Install the operator (install.yaml is idempotent and keeps the CRDs in sync)
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

Method 5: Custom Namespace Installation

By default, the operator deploys to the inference-gateway-system namespace. To deploy to a custom namespace:

Option A: Simple sed replacement

# Download and modify the install.yaml
curl -L https://github.com/inference-gateway/operator/releases/latest/download/install.yaml | \
  sed 's/inference-gateway-system/my-custom-namespace/g' | \
  kubectl apply --server-side -f -

Option B: Using the development workflow

# Clone the repository
git clone https://github.com/inference-gateway/operator.git
cd operator

# Generate manifests for your custom namespace
task manifests-for-namespace NAMESPACE=my-custom-namespace

# Deploy the generated manifests
kubectl apply --server-side -f manifests/my-custom-namespace/install.yaml

Option C: GitOps with custom namespace

manifests/install.yaml hardcodes namespace: inference-gateway-system on every namespaced object (operator Deployment, ServiceAccount, leader-election Role and RoleBinding, metrics Service) and in the operator-manager-rolebinding ClusterRoleBinding subjects. A kustomize patch that only renames the Namespace object therefore leaves the operator in the original namespace. Generate the manifests instead and point ArgoCD at your own repository:

git clone https://github.com/inference-gateway/operator.git
cd operator

# Rewrites the namespace across the whole manifest
task manifests-for-namespace NAMESPACE=my-custom-namespace

# Commit manifests/my-custom-namespace/ to your GitOps repository
# ArgoCD Application pointing at your generated manifests
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-gateway-operator
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/my-gitops-repo
    targetRevision: main
    path: manifests/my-custom-namespace
  destination:
    server: https://kubernetes.default.svc
    namespace: my-custom-namespace
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

Method 6: Development Installation

For development and testing with the latest code:

# Clone the repository
git clone https://github.com/inference-gateway/operator.git
cd operator

# Install CRDs
task install

# Build and deploy operator
task deploy IMG=ghcr.io/inference-gateway/operator:latest

task deploy builds the image with Docker, imports it into the local k3d cluster named dev (k3d image import <IMG> -c dev) and applies config/environments/dev. It needs Go 1.27.3+, Docker, and the k3d dev cluster created by task cluster:create - it does not work against an arbitrary ~/.kube/config context.

βœ… Verification

Verify the installation:

# Check if the operator is running
kubectl get pods -n inference-gateway-system

# Check if CRDs are installed
kubectl get crd | grep inference-gateway

# View operator logs
kubectl logs -n inference-gateway-system deployment/operator-inference-gateway -f

Expected output:

# Pods should show Running status
NAME                                          READY   STATUS    RESTARTS   AGE
operator-inference-gateway-74c9c5f5b-x4d2k    1/1     Running   0          2m

# CRDs should be listed
agents.core.inference-gateway.com          2025-06-21T17:30:00Z
gateways.core.inference-gateway.com        2025-06-21T17:30:00Z
gpus.core.inference-gateway.com            2025-06-21T17:30:00Z
mcps.core.inference-gateway.com            2025-06-21T17:30:00Z
orchestrators.core.inference-gateway.com   2025-06-21T17:30:00Z

🏷️ Namespace Scoping

The shipped operator Deployment sets WATCH_NAMESPACE_SELECTOR=inference-gateway.com/managed=true, so Gateway, Agent, MCP and GPU resources are only reconciled in namespaces carrying that label. In an unlabeled namespace the operator just logs skipping gateway, namespace does not match WATCH_NAMESPACE_SELECTOR and creates nothing.

Label every namespace you deploy resources into:

kubectl label namespace <namespace> inference-gateway.com/managed=true

The manifests under examples/ already create their namespaces with this label. Orchestrator resources are not namespace-filtered.

To change the scope, edit WATCH_NAMESPACE_SELECTOR on the operator Deployment - any valid label selector works, and an empty value (or removing the variable) makes the operator watch all namespaces:

# Watch all namespaces
kubectl set env -n inference-gateway-system deployment/operator-inference-gateway WATCH_NAMESPACE_SELECTOR=

All Gateway, Agent, MCP and GPU examples below assume their namespace is labeled accordingly.

πŸš€ Deploy Your First Gateway

Create a simple gateway to test the installation:

# Label the target namespace so the operator reconciles it
kubectl label namespace default inference-gateway.com/managed=true

# Create a minimal gateway
cat <<EOF | kubectl apply -f -
apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: my-first-gateway
  namespace: default
spec:
  replicas: 1
  environment: development
  telemetry:
    enabled: true
    metrics:
      enabled: true
      port: 9464
  providers:
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: api_key
EOF

Note: You'll need to create the openai-secret with your API key:

kubectl create secret generic openai-secret \
  --from-literal=api_key=your-openai-api-key-here

πŸ€– Deploy an Orchestrator

The Orchestrator CRD deploys the Inference Gateway CLI's infer daemon: an LLM-driven loop that listens on chat channels, serves devices over its AG-UI binding, and fans out to A2A Agents and tools (incl. MCP). Telegram is the channel currently supported - more channels are planned. Channels, the gateway and the agent are all optional, but the daemon needs something to run: an Orchestrator without channels, daemon.binding or tools.schedule (or a switch for the heartbeat or the A2A server in env) gets Ready=False with reason InvalidSpec.

When the Telegram channel is enabled, the Deployment is forced to a singleton (replicas: 1, strategy: Recreate) because Telegram allows only one active getUpdates consumer per bot token - running two replicas would cause them to terminate each other with 409 Conflict. For high-availability today, run multiple Orchestrator resources with different tokens and disjoint allowedUsers (manual sharding) - webhook mode and shared state would be required for true scale-out and are out of scope.

# 1. Create the bot credentials secret
kubectl create namespace orchestrators
kubectl create secret generic telegram-bot-credentials -n orchestrators \
  --from-literal=token='<TELEGRAM_BOT_TOKEN>' \
  --from-literal=allowedUsers='111111111,222222222'

# 2. Apply the Orchestrator resource
cat <<EOF | kubectl apply -f -
apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
  name: orchestrator-controlled-by-telegram
  namespace: orchestrators
spec:
  image: ghcr.io/inference-gateway/cli:latest
  channels:
    telegram:
      enabled: true
      tokenSecretRef:
        name: telegram-bot-credentials
        key: token
      allowedUsersSecretRef:
        name: telegram-bot-credentials
        key: allowedUsers
      pollTimeout: 30s
  gateway:
    url: http://inference-gateway.inference-gateway.svc.cluster.local:8080
  agent:
    model: deepseek/deepseek-v4-pro
    systemPrompt: "You are a helpful assistant running as a Telegram bot."
EOF

# 3. Watch logs (requires CLI built with INFER_LOGGING_STDOUT support)
kubectl logs -n orchestrators deploy/orchestrator-controlled-by-telegram -f

A runnable Telegram example with a step-by-step walkthrough is available at examples/orchestrator/.

Link the daemon to a Gateway

spec.gateway takes either url, the address of any gateway, or ref, the name of a Gateway in the same namespace - one or the other, not both. With ref the operator addresses that Gateway on its own Service port, so the daemon's INFER_GATEWAY_URL follows a port change instead of hard-coding one, and it runs the Gateway with zero replicas while every Orchestrator referencing it is suspended:

spec:
  gateway:
    ref: inference-gateway
  • A Gateway no Orchestrator references keeps its spec.replicas and its HPA.
  • A parked Gateway keeps its Deployment, Service and Secrets, and has no HPA while it is parked - an HPA would scale the parked Deployment straight back up. It gets one again when a referencing Orchestrator resumes.
  • Resuming an Orchestrator (spec.sandbox.operatingMode: Running) scales its Gateway back up right away, because the Gateway's controller watches the Orchestrators that reference it.
  • An Orchestrator whose ref names no Gateway reports Ready=False with reason GatewayNotFound.

Serve devices over the AG-UI binding

spec.daemon.binding turns on the daemon's AG-UI binding, the WebSocket the chat TUI, the browser extension and other daemons sync sessions over. The binding listens on every interface, so it always serves TLS from tls.secretName (tls.crt and tls.key, plus ca.crt with clientAuth: true). Clients authenticate with the token from tokenSecretRef, a client certificate, or both, and the API rejects a binding with neither. The controller adds a <name>-binding Service on the binding port (default 52789) and holds the pod back from Ready until the binding listens. spec.daemon.binding.service.annotations and .labels are copied verbatim onto that Service for other controllers to act on (a VPN operator such as Tailscale with tailscale.com/expose: "true", external-dns, a service mesh). The operator ships no such keys itself, and metadata those controllers add to the Service survives a reconcile.

The certificate must name the host clients dial: daemon-binding.orchestrators.svc in the cluster, or localhost through a port-forward. See examples/daemon/ for an Orchestrator that only serves the binding.

Run the daemon in a Sandbox

spec.runtime: Sandbox runs the daemon in an agent-sandbox Sandbox instead of a Deployment: a stable pod with two persistent volumes, the daemon's workspace (/workspace) and its state (~/.infer/projects), that outlive a suspension. It needs agent-sandbox installed. Without it, the Orchestrator reports Ready=False with reason SandboxCRDMissing and everything else keeps running. spec.runtime is fixed at creation, because switching away from a Sandbox would delete its volumes.

apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
  name: sandboxed-daemon
  namespace: orchestrators
spec:
  image: ghcr.io/inference-gateway/cli:latest
  runtime: Sandbox
  sandbox:
    storage: 5Gi
    retention: 24h
    runtimeClassName: gvisor
  resources:
    limits:
      cpu: "2"
      memory: 4Gi
  daemon:
    binding:
      tokenSecretRef:
        name: daemon-binding
        key: token
      tls:
        secretName: daemon-binding-tls
  • spec.sandbox.operatingMode: Suspended deletes the pod and keeps the volumes. Running resumes it on a fresh pod once the suspension completed. A linked Gateway is scaled down and up with it.
  • The daemon idles out by exiting 0 (INFER_DAEMON_IDLE_EXIT in spec.env). The operator then parks the Orchestrator itself: it sets operatingMode: Suspended, and a client resumes the daemon by setting Running. A daemon that writes agents-done=true to its termination message has every agent done, which starts spec.sandbox.retention: the Orchestrator is deleted that long after, unless a client resumed it first.
  • spec.sandbox.shutdownTime expires the Sandbox at that time, and the operator deletes the Orchestrator with the Sandbox, the volumes and everything else it owns once agent-sandbox reports SandboxExpired.
  • A change to the daemon's pod (its image, env, binding or discovered agents) cycles the pod through Suspended, because agent-sandbox only applies a new template to the pod it creates next. A pod that fails is recreated by agent-sandbox with a backoff (failedPodPolicy: Recreate).
  • spec.sandbox.storage (default 5Gi) sizes both volumes and is fixed at creation.
  • spec.sandbox.volumes mounts PersistentVolumeClaims that already exist in the Orchestrator's namespace into the daemon's container - an app installed once, or content its sessions share. See Mount existing claims.
  • status.podName names the Sandbox's pod once it is ready, for clients that port-forward to it.

Mount existing claims

A Sandbox's own volumes start empty, so an app installed by spec.userData is downloaded again on every pod. spec.sandbox.volumes mounts claims that are created and filled elsewhere:

spec:
  runtime: Sandbox
  sandbox:
    volumes:
      - name: resolve
        claimName: davinci-resolve
        mountPath: /opt/resolve
      - name: footage
        claimName: raw-footage
        mountPath: /mnt/footage
        readOnly: true
  • readOnly: true mounts the claim read-only, for content the agents copy into /workspace before they change it. The original stays as it is.
  • The Sandbox never owns a mounted claim. A suspension, spec.sandbox.retention, spec.sandbox.shutdownTime and deleting the Orchestrator all leave it and its data in place.
  • A mountPath is absolute and cannot be, or lie under, a path the operator mounts: /workspace, /home/infer/.infer/projects and /var/lib/cloud/seed/nocloud. Two volumes cannot share one.
  • Changing the list cycles a running pod through Suspended, like any other change to the daemon's pod.
  • A claim that does not exist leaves the pod unscheduled. The Orchestrator reports it as Ready=False with reason SandboxNotReady and the scheduler's message.
  • A ReadWriteOnce claim is shared by pods on one node only. Orchestrators on several nodes need a ReadWriteMany or ReadOnlyMany claim.
  • A pod that mounts a claim sets fsGroupChangePolicy: OnRootMismatch, so a large volume is not walked on every boot.

For an app alone, an image with the app installed needs none of this - a node pulls it once and caches it.

The pod follows agent-sandbox's threat model for untrusted, LLM-driven code: it has no Kubernetes API access (automountServiceAccountToken: false), runs as a non-root user without capabilities or privilege escalation under the RuntimeDefault seccomp profile, and spec.sandbox.runtimeClassName runs it under gVisor or Kata. Set spec.resources limits too. The operator creates no NetworkPolicy, because the daemon reaches the inference gateway, MCP servers and A2A agents inside the cluster. Restrict its traffic with one of your own, for example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: sandboxed-daemon
  namespace: orchestrators
spec:
  podSelector:
    matchLabels:
      app: sandboxed-daemon
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - ports:
        - port: 52789
  egress:
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: inference-gateway
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
              - 169.254.0.0/16

Boot a computer with user-data

spec.userData hands the pod a cloud-init user-data script that runs as root when it boots, before the daemon starts, the way a cloud VM runs it - for example to install a browser on a computer-use image. The Secret key is mounted read-only for root at cloud-init's NoCloud seed path, /var/lib/cloud/seed/nocloud/user-data. The image runs it and then drops to the daemon's user, as ghcr.io/inference-gateway/computer does.

spec:
  image: ghcr.io/inference-gateway/computer:latest
  runtime: Sandbox
  userData:
    secretRef:
      name: computer-user-data
      key: userdata

The pod then boots as root in a user namespace (hostUsers: false), so its root maps to an unprivileged user on the node and keeps only the capabilities a package install needs (CHOWN, DAC_OVERRIDE, FOWNER, FSETID, SETUID, SETGID), still without privilege escalation. Nodes need Kubernetes 1.36 or newer with containerd 2.0+ or CRI-O 1.25+ and Linux 6.3+. The secretRef has the shape of KubeVirt's cloudInitNoCloud volume, which reads the key userdata, so the same Secret boots a VM unchanged.

πŸ”„ Upgrade

To upgrade the operator to a newer version:

# Upgrade to latest version
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

The operator Deployment uses the Recreate strategy, so an upgrade terminates the running operator pod before starting the new one. Existing Gateway, Agent, MCP, Orchestrator and GPU workloads keep serving during that gap - only reconciliation pauses until the new pod is ready.

πŸ—‘οΈ Uninstallation

To completely remove the operator:

# Delete all Gateway instances first
kubectl delete gateway --all --all-namespaces

# Uninstall the operator
kubectl delete -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

πŸ—οΈ Supported Architectures

The operator supports multi-architecture deployments:

  • linux/amd64 - Intel/AMD 64-bit processors
  • linux/arm64 - ARM 64-bit processors (Apple Silicon, AWS Graviton, etc.)

Container images are automatically selected based on your cluster's node architecture.

βš™οΈ Example Configurations

Every Gateway, Agent, MCP and GPU example below only reconciles if its namespace is labeled inference-gateway.com/managed=true - see Namespace Scoping.

Minimal Gateway

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: simple-gateway
  namespace: default
spec:
  replicas: 1
  environment: development
  telemetry:
    enabled: true
    metrics:
      enabled: true
      port: 9464
  providers:
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: api_key

Production Gateway with Authentication

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: production-gateway
  namespace: inference-gateway
spec:
  replicas: 3
  image: "ghcr.io/inference-gateway/inference-gateway:0.23.6"
  environment: production

  auth:
    enabled: true
    provider: oidc
    oidc:
      issuerUrl: "https://auth.company.com/realms/ai"
      clientId: "inference-gateway"
      audiences:
        - "api://inference-gateway"
      clientSecretRef:
        name: auth-secrets
        key: client-secret

  providers:
    - name: OpenAI
      enabled: true
      env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: ai-secrets
              key: openai-key

  resources:
    requests:
      cpu: "500m"
      memory: "512Mi"
    limits:
      cpu: "2000m"
      memory: "2Gi"

  gatewayAPI:
    enabled: true
    gateway:
      gatewayClassName: envoy
      tls:
        enabled: true
        issuer: letsencrypt-prod
        secretName: ai-gateway-tls
    httpRoute:
      hostnames:
        - "ai-gateway.company.com"

A2A Service Discovery Configuration

Both the Gateway and the Orchestrator discover Agent CRs by label selector. The Gateway unions them with its static spec.a2a.agents[] and renders A2A_AGENTS for the gateway pod, the Orchestrator writes them into ~/.infer/agents.yaml inside its pod.

With spec.a2a.enabled, the gateway becomes an A2A server: it publishes one merged card at GET /.well-known/agent-card.json and relays every POST /a2a call to the agent the request names.

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: gateway-with-a2a
  namespace: default
spec:
  a2a:
    # Turns the gateway into an A2A server (A2A_ENABLED).
    enabled: true
    # Static agents are {name, url} objects, kept alongside discovered ones.
    # (The Orchestrator takes plain URL strings instead.)
    agents:
      - name: static-agent
        url: "http://static-agent.agents.svc.cluster.local:8080"
    # Automatic discovery of Agent CRs by label selector.
    serviceDiscovery:
      enabled: true
      namespace: "agents" # Namespace to search (defaults to the Gateway's own namespace)
      selector:
        matchLabels:
          agent-group: group1
    # Canonical public /a2a URL published on the agent card and as the RFC 9728
    # `resource` (A2A_RESOURCE_URL). Defaults to
    # "<scheme>://<first httpRoute hostname>/a2a".
    resourceUrl: "https://ai-gateway.company.com/a2a"
    timeouts:
      client: "30s" # A2A_CLIENT_TIMEOUT - one non-streaming call, including the card fetch
      streamIdle: "5m" # A2A_STREAM_IDLE_TIMEOUT - idle cutoff for a relayed stream, "0" disables
    # How often agent cards are re-fetched in the background
    # (A2A_CARD_REFRESH_INTERVAL) - "0" disables the refresh.
    cardRefreshInterval: "5m"

Agent alias rule: each entry is rendered into A2A_AGENTS as <name>=<url>, and that name prefixes the agent's skill ids on the gateway card and every task id the gateway hands out. To be used as an alias the name must match ^[a-z0-9_-]+$ and be unique across static and discovered agents. Otherwise the entry is rendered as a bare URL and the gateway derives the alias from the host. Discovered agents are named after metadata.name and use status.card.url, falling back to the in-cluster Service URL while status is still empty. Entries are deduped on URL and sorted for determinism.

The Orchestrator configures the same discovery with plain URL strings:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
  name: orchestrator-with-service-discovery
  namespace: default
spec:
  image: ghcr.io/inference-gateway/cli:latest
  channels:
    telegram:
      enabled: true
      tokenSecretRef:
        name: telegram-bot-credentials
        key: token
  gateway:
    url: http://inference-gateway.inference-gateway.svc.cluster.local:8080
  agent:
    model: deepseek/deepseek-v4-pro
  a2a:
    enabled: true
    # Static agent URLs are kept alongside discovered ones.
    agents:
      - "http://static-agent.agents.svc.cluster.local:8080"
    # Automatic discovery of Agent CRs by label selector.
    serviceDiscovery:
      enabled: true
      namespace: "agents" # Namespace to search (defaults to the Orchestrator's own namespace)
      selector:
        matchLabels:
          agent-group: group1

Service Discovery Features:

  • Automatic Agent Discovery: Discovers Agent CRs based on their Kubernetes labels
  • Dynamic Configuration: The gateway's A2A_AGENTS is re-rendered and the orchestrator's mounted agents.yaml rewritten when the set changes, rolling the pod
  • Label-Based Selection: Uses a configurable label selector to identify Agent CRs
  • Namespace Scoping: Can search for Agent CRs in a specific namespace (defaults to the Gateway's or Orchestrator's own namespace)

Agent Requirements:

For an Agent CR to be discovered, it must carry labels matching the selector:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Agent
metadata:
  name: my-agent
  namespace: agents
  labels:
    agent-group: group1 # matches spec.a2a.serviceDiscovery.selector
spec:
  # Agent configuration

MCP OAuth Protected Resource Metadata (RFC 9728)

When spec.mcp.expose is on, the gateway serves OAuth 2.0 Protected Resource Metadata at GET /.well-known/oauth-protected-resource/mcp, so an MCP client with no token discovers the IdP from the 401 on /mcp (required by MCP 2026-07-28). The operator-managed HTTPRoute matches the / path prefix, so that document is routed to the gateway Service wherever /mcp is - no extra route needed.

spec.mcp.resourceUrl pins the canonical public /mcp URL the gateway puts in the document's resource and in the challenge's resource_metadata. It is emitted as MCP_RESOURCE_URL:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: gateway-with-mcp-auth
  namespace: inference-gateway
spec:
  mcp:
    enabled: true
    expose: true
    resourceUrl: "https://api.example.com/mcp"
  gatewayAPI:
    enabled: true
    gateway:
      tls:
        enabled: true
    httpRoute:
      hostnames:
        - api.example.com

When resourceUrl is omitted the operator defaults it to <scheme>://<first httpRoute hostname>/mcp (https when gatewayAPI.gateway.tls.enabled), so the example above works without the explicit value. Nothing is emitted when routing is disabled or the hostname is a wildcard, and the gateway then derives the URL from the inbound request scheme (honouring X-Forwarded-Proto) and Host - set resourceUrl explicitly whenever the ingress rewrites either, or clients are handed a URL they cannot reach.

MCP Service Discovery Configuration

MCP Custom Resources can be discovered automatically by both Gateway and Orchestrator via the same label-selector pattern used for A2A agents. Opt-in on the MCP side is via plain metadata.labels - no extra field on MCPSpec.

Gateway-side discovery - discovered MCPs are unioned with the static spec.mcp.servers[] list, deduped on URL, sorted, and exposed via the gateway pod's MCP_SERVERS env var as name=url entries. The name (spec.mcp.servers[].name for static servers, metadata.name for discovered MCP CRs) becomes the tool namespace, so tools reach the model as mcp_<name>_<tool> - for example mcp_time_get_current_time rather than a long host-derived alias. To be used as an alias a name must match ^[a-z0-9_-]+$, be unique across static and discovered servers, and must not be the reserved alias tools. A name that fails these rules is rendered as a bare URL and the gateway falls back to deriving the alias from the host.

apiVersion: core.inference-gateway.com/v1alpha1
kind: Gateway
metadata:
  name: gateway-with-mcp-discovery
  namespace: inference-gateway
spec:
  mcp:
    enabled: true
    # Static entries still work alongside discovery.
    servers:
      - name: external-mcp
        url: "https://mcp.example.com/mcp"
    serviceDiscovery:
      enabled: true
      namespace: mcp # defaults to the Gateway's own namespace
      selector:
        matchLabels:
          mcp-group: group1

Orchestrator-side discovery - discovered MCPs are rendered into an mcp.yaml ConfigMap mounted at ~/.infer/mcp.yaml inside the orchestrator pod. A content-hash pod annotation rolls the singleton when the set changes:

apiVersion: core.inference-gateway.com/v1alpha1
kind: Orchestrator
metadata:
  name: orchestrator
  namespace: orchestrator
spec:
  image: ghcr.io/inference-gateway/cli:latest
  channels:
    telegram:
      enabled: true
      tokenSecretRef:
        name: telegram-bot-credentials
        key: token
  gateway:
    url: http://inference-gateway.inference-gateway.svc.cluster.local:8080
  agent:
    model: deepseek/deepseek-v4-pro
  mcp:
    enabled: true
    servers: [] # {name, url, headers, includeTools, excludeTools, timeoutSeconds} entries, optional
    serviceDiscovery:
      enabled: true
      namespace: mcp
      selector:
        matchLabels:
          mcp-group: group1

MCP CR labeling - tag each MCP you want discovered with a matching label:

apiVersion: core.inference-gateway.com/v1alpha1
kind: MCP
metadata:
  name: my-mcp
  namespace: mcp
  labels:
    mcp-group: group1 # picked up by both selectors above
spec:
  image: ghcr.io/your-org/your-mcp-server:latest
  server:
    port: 8080
    # Endpoint path the operator appends when building the in-cluster URL.
    # Defaults to "/mcp". Override if your server listens elsewhere.
    path: "/mcp"

MCP servers by URL - spec.mcp.servers lists servers the pod reaches by URL, each with a name, a url and optional headers, as a Gateway's do. They are written into the pod's mcp.yaml with or without discovery. Header values are written as given, so a ${VAR} stays a reference the infer CLI in the pod expands from its environment. Fill it from a Secret through spec.env, and never put a secret value in a header:

spec:
  env:
    - name: MCP_EXAMPLE_API_KEY
      valueFrom:
        secretKeyRef:
          name: mcp-example-credentials
          key: MCP_EXAMPLE_API_KEY
  mcp:
    enabled: true
    servers:
      - name: example
        url: https://mcp.example.com/mcp
        headers:
          x-api-key: ${MCP_EXAMPLE_API_KEY}

A server can also narrow and bound what the pod gets from it. includeTools limits the pod to the listed tools, named as the server lists them, and excludeTools hides the listed ones. When both are set, includeTools wins. timeoutSeconds bounds one request to the server, and unset the pod's default applies:

    servers:
      - name: example
        url: https://mcp.example.com/mcp
        includeTools:
          - search
        timeoutSeconds: 10

The pod's infer image must be v0.248.0 or newer, which reads url and headers.

Visibility:

kubectl get gateway       # MCPS column shows static + discovered MCP servers,
                          # A2A column (.status.a2aAgentCount) static + discovered A2A agents,
                          # Audio column (.spec.audio.enabled) whether the Audio API is on
kubectl get orchestrator  # MCPS column shows the discovered count

For a runnable MCP server to discover, see examples/mcp-server/.

Focused Examples

Each example shows one feature. kubectl explain gateway.core.inference-gateway.com.spec documents every field.

βœ… Configuration Validation

The operator includes comprehensive validation:

  • Replica limits: 1-100 replicas
  • Port ranges: Valid port numbers (1024-65535 for server ports)
  • Environment values: Restricted to development, staging, production
  • Resource limits: Proper CPU/memory specifications

πŸ“Š Monitoring and Status

Check Gateway status and health:

# Check gateway resources
kubectl get gateways -A

# Get detailed gateway status
kubectl describe gateway my-first-gateway

# Check generated resources (the Deployment and Service are named after the
# Gateway - the routing ConfigMap, HTTPRoute, upstream Gateway and HPA are the
# ones labeled app=<gateway-name>)
kubectl get deployment,service my-first-gateway
kubectl get configmaps,httproutes,hpa -l app=my-first-gateway

# View Gateway logs
kubectl logs -l app=my-first-gateway -f

status reports:

  • url: the address the gateway is reachable at (HTTPRoute hostname when Gateway API routing is enabled, otherwise the in-cluster service FQDN)
  • providerSummary: comma-separated list of configured providers
  • serviceAccountName: the ServiceAccount the gateway pods run as
  • mcpServers: sorted <name>=<url> entries the pod is configured with
  • mcpServerCount: number of static plus discovered MCP servers
  • a2aAgents: sorted <name>=<url> entries mirroring A2A_AGENTS
  • a2aAgentCount: number of static plus discovered A2A agents

For replica counts and rollout health, check the Deployment directly (kubectl get deployment my-first-gateway) - the Gateway status does not mirror them.

❓ Frequently Asked Questions

Do I need to install CRDs separately?

It depends on your installation method:

  • One-command installation (Recommended): No! CRDs are included automatically:

    kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml
  • GitOps/ArgoCD installations: No! Use the manifests/ directory which includes CRDs:

    source:
      repoURL: https://github.com/inference-gateway/operator
      path: manifests # Includes both CRDs and operator
  • Custom namespace installations: No! When you substitute the namespace in install.yaml, the CRDs are still included:

    curl -L https://github.com/inference-gateway/operator/releases/latest/download/install.yaml | \
      sed 's/inference-gateway-system/my-namespace/g' | kubectl apply --server-side -f -
  • Advanced scenarios: You can install CRDs separately for more control:

    kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/crds.yaml

What architectures are supported?

The operator supports both arm64 and amd64 architectures:

  • Container images are built for both platforms
  • Kubernetes automatically selects the correct image for your nodes
  • Works on Apple Silicon (M1/M2), AWS Graviton, Intel/AMD processors

How do I check if the installation was successful?

Run these commands to verify your installation:

# 1. Check operator pods
kubectl get pods -n inference-gateway-system

# 2. Verify CRDs are installed
kubectl get crd | grep inference-gateway

# 3. Test creating a Gateway resource
kubectl get gateways --all-namespaces

Can I install in a different namespace?

Yes! The operator defaults to inference-gateway-system but can be deployed to any namespace:

Quick Method:

curl -L https://github.com/inference-gateway/operator/releases/latest/download/install.yaml | \
  sed 's/inference-gateway-system/my-namespace/g' | \
  kubectl apply --server-side -f -

Development Method:

# Generate manifests for custom namespace
task manifests-for-namespace NAMESPACE=my-namespace
kubectl apply --server-side -f manifests/my-namespace/install.yaml

GitOps Method: Kustomize namespace patches do not work - the upstream install.yaml hardcodes namespace: inference-gateway-system on every namespaced object and in the operator-manager-rolebinding subjects. Generate the manifests with task manifests-for-namespace NAMESPACE=my-namespace, commit manifests/my-namespace/ to your GitOps repository, and point ArgoCD or Flux at that directory. See Method 5 Option C.

How do I upgrade the operator?

Simply reapply the installation with a newer version:

# Upgrade to latest
kubectl apply --server-side -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

The operator Deployment uses the Recreate strategy, so the old operator pod is terminated before the new one starts. Running Gateway instances are unaffected - reconciliation pauses briefly.

What happens to my Gateways if I delete the operator?

Your Gateway resources will remain in the cluster but will no longer be managed. To completely clean up:

# 1. Delete all Gateway instances first
kubectl delete gateway --all --all-namespaces

# 2. Then uninstall the operator
kubectl delete -f https://github.com/inference-gateway/operator/releases/latest/download/install.yaml

πŸ—οΈ Development

Prerequisites for Development

The recommended way to get a complete toolchain is via the project's Flox environment - flox activate provides Go, task, kubectl, kubebuilder, kustomize, golangci-lint, k3d, ctlptl, gh, and prettier at the pinned versions used by CI. See CONTRIBUTING.md for alternatives (flox environment or manual install).

If installing manually:

  • Task runner (task)
  • Go 1.27+
  • Docker
  • k3d (via ctlptl) or similar local Kubernetes cluster

Development Workflow

# Run tests
task test

# Run linting
task lint

# Generate code and manifests (including install.yaml)
task generate manifests

# Build locally
task build

# Run against local cluster
task run

Note: The task manifests command automatically generates:

  • CRDs in config/crd/bases/
  • Installation manifests in manifests/install.yaml
  • CRD-only manifests in manifests/crds.yaml

These files are version-controlled and included in releases.

Testing

The operator includes comprehensive unit and integration tests:

# Run all tests
task test

# Run e2e tests (requires running cluster)
task test:e2e

πŸ“Š Monitoring & Management

πŸ” OpenTelemetry Observability

The Inference Gateway provides enterprise-grade observability through OpenTelemetry:

Metrics Collection:

# Access Prometheus metrics. The metrics port is exposed on the Service named
# after the Gateway, and only when both telemetry.enabled and
# telemetry.metrics.enabled are true.
curl http://my-gateway:9464/metrics

# Key metrics include:
# - llm_requests_total: Request counts by provider/model
# - llm_tokens_*: Token usage tracking
# - llm_request_duration_seconds: Request latency histograms
# - llm_latency_*: Detailed timing breakdowns

Distributed Tracing:

# Configure OTLP trace export - OTLP is the only supported exporter
telemetry:
  enabled: true
  traces:
    exporter:
      otlp:
        endpoint: "http://otel-collector:4318"
        protocol: "http/protobuf"

The Gateway ignores protocol and always exports OTLP over HTTP, so endpoint must point at an OTLP/HTTP receiver such as http://otel-collector:4318. A gRPC endpoint like :4317 receives OTLP/HTTP and no spans arrive. (protocol is forwarded only for Agent workloads, as A2A_OTEL_EXPORTER_OTLP_PROTOCOL.)

TELEMETRY_TRACING_ENABLED and TELEMETRY_TRACING_OTLP_ENDPOINT are only set on the gateway container when telemetry.enabled is true and telemetry.traces.exporter.otlp is present. Non-OTLP collector endpoints (for example Jaeger's :14268/api/traces) are not supported - point the exporter at an OTLP receiver instead.

Supported Backends:

  • Metrics: Prometheus, Grafana, any OpenTelemetry-compatible backend
  • Tracing: Jaeger, Zipkin, Lightstep, Honeycomb, Datadog (via OTLP)
  • Logs: Structured JSON with trace correlation for any log aggregation system

πŸ“ˆ Monitoring Gateway Health

# Check gateway status
kubectl get gateway my-gateway -o yaml

# Check deployment health
kubectl get deployment my-gateway

# Check service endpoints
kubectl get service my-gateway

# View the inline model-routing config (only created when spec.routing.config is set)
kubectl get configmap my-gateway-routing -o yaml

πŸ”§ Troubleshooting

Common issues and solutions:

  1. Nothing happens after applying a Gateway/Agent/MCP/GPU: the namespace is missing the inference-gateway.com/managed=true label - see Namespace Scoping
  2. Operator pod in CrashLoopBackOff: the Kubernetes Gateway API standard CRDs are missing - see Prerequisites
  3. Gateway not starting: Check image pull policy and secrets
  4. Authentication failures: Verify OIDC configuration and secrets
  5. Provider connection issues: Check network policies and secret references
  6. Resource constraints: Review resource requests/limits

Upgrade Process

The operator supports rolling upgrades:

  1. Update the Gateway spec with new image version
  2. Operator automatically performs rolling update
  3. Monitor status for completion

Cleanup

Delete gateway instances:

kubectl delete gateway --all

Uninstall operator:

task undeploy
task uninstall

πŸ“– API Reference

For complete API reference, see the generated CRD documentation or use:

kubectl explain gateway.spec
kubectl explain gateway.spec.providers
kubectl explain gateway.spec.auth
# etc.

🀝 Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make changes with tests
  4. Run task lint test
  5. Submit a pull request

πŸ“„ License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

About

This project provides a Kubernetes Operator for managing the lifecycle of the inference-gateway and its related components. It simplifies deployment, configuration, and scaling of the gateway within Kubernetes clusters, enabling seamless integration of inference workflows.

Topics

Resources

Contributing

Security policy

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages