Skip to content

aks.md

Latest commit

 

History

History
1093 lines (862 loc) · 47.7 KB

aks.md

File metadata and controls

1093 lines (862 loc) · 47.7 KB
Raw

Deploying to Azure Kubernetes Service (AKS)

This guide walks through deploying PilotSwarm workers to AKS for production multi-node operation.

Two deployment paths. The repo ships two side-by-side ways to deploy PilotSwarm to AKS:

  1. Legacy scripts/deploy-aks.sh (the rest of this document). Imperative bash + raw kubectl apply against deploy/k8s/*.yaml. Stable, well-trodden, and not going away.
  2. GitOps IaC pipeline under deploy/ (described below in GitOps IaC Path). Bicep-managed Azure infra + Flux-driven cluster manifests, pulled from versioned blob containers. Modeled on a known-good internal reference implementation, simplified for PilotSwarm's single-service Node.js shape. This path adds Edge Mode (AFD vs Private AppGw) and TLS Source (AKV vs Let's Encrypt) topology choices.

Choose one. They share the same Kubernetes cluster shape but stamp it out via different mechanisms; running both against the same cluster will fight over the same resources.

GitOps IaC Path

The IaC pipeline under deploy/ provisions Azure infra via Bicep and keeps cluster state in sync via Flux Kustomizations sourced from blob containers. Public entry points:

  • deploy/scripts/new-env.mjs — generate a per-env .env + bicep parameter files. Picks defaults for the topology axes below.
  • deploy/scripts/deploy.mjs — orchestrates the per-service bicep + manifest stage in infraOrder then services order from deploy/services/deploy-manifest.json.
  • deploy/scripts/test/*.test.mjs — npm run test:deploy-scripts.

Topology Matrix

The IaC path supports an (EDGE_MODE × TLS_SOURCE) matrix plus the optional VPN_GATEWAY_ENABLED axis. Five combinations are supported; the rest are blocked at preflight (see Unsupported Combinations and the named diagnostic codes in Optional: VPN Gateway P2S):

EDGE_MODE TLS_SOURCE Edge ingress Cert source Notes
afd letsencrypt AFD → AppGw (Private Link) + AGIC cert-manager + Let's Encrypt prod (HTTP-01) OSS default. Zero CA setup.
afd akv AFD → AppGw (Private Link) + AGIC OneCertV2-PublicCA via AKV (registered automatically) enterprise default. BYO public CA.
afd + VPN akv AFD → AppGw (Private Link) and Azure VPN Gateway P2S → same AppGw private FE OneCertV2-PublicCA via AKV (shared with AFD path) Hybrid trusted-bypass; opt-in via VPN_GATEWAY_ENABLED=true. AKV-only. See Optional: VPN Gateway P2S.
private akv AKS web-app-routing addon (NGINX) + ILB OneCertV2-PrivateCA via AKV (registered automatically) Enterprise / AME. No AFD, no AppGw, no AGIC.
private akv-selfsigned AKS web-app-routing addon (NGINX) + ILB AKV Self issuer (auto-generated, in-place) No CA; private-VNet smoke tests.

Default for OSS = afd + letsencrypt. Default for the enterprise path = afd + akv.

  • EDGE_MODE=afd — Azure Front Door fronts a regional Application Gateway over Private Link. Public TLS terminates at AFD; AppGw is reachable only via the AFD private endpoint. Use for any Internet- facing deployment.
  • EDGE_MODE=private — No AFD, no AppGw, no AGIC, no GlobalInfra resource group. AKS uses the web-app-routing addon (managed NGINX) with an internal-only Azure Load Balancer. Reachable from peered VNets / Bastion / VPN / ExpressRoute. The globalinfra and afd services are skipped by deploy.mjs. Bicep also provisions a Private DNS Zone (PRIVATE_DNS_ZONE) and links it to the AKS VNet; deploy.mjs writes an A record ${HOST}.${PRIVATE_DNS_ZONE} → internal LB IP after the Portal rolls out.
  • TLS_SOURCE=akv — Portal cert is issued by an AKV cert issuer. The bicep auto-registers OneCertV2-PublicCA (afd mode) or OneCertV2-PrivateCA (private mode) on the Key Vault using the shared akv-certificate-issuer.bicep module — no manual issuer setup. Override with PORTAL_TLS_ISSUER_NAME if you have a different registered CA. Cert is projected into the cluster via Secret Store CSI; afd mode binds it to AppGw via the appgw-ssl-certificate AGIC annotation, private mode mounts it directly into the NGINX-fronted Portal pod's TLS secret.
  • TLS_SOURCE=akv-selfsigned (private only) — uses the AKV built-in Self issuer to mint a self-signed cert. Browsers will warn; only suitable for private-VNet smoke tests where you control the trust store.
  • TLS_SOURCE=letsencrypt (afd only) — cert-manager is installed in-cluster via Flux (HelmRelease pinned to v1.20.2 exact). The letsencrypt-prod ClusterIssuer (HTTP-01 solver) issues a real CA cert and writes it into the K8s Secret named in the Ingress tls.secretName. AGIC imports it from there. No AKV cert, no manual upload step. Requires ACME_EMAIL in the env.

Variant Overlays

deploy/gitops/portal/overlays/ ships three flavors, one per supported combo (akv and akv-selfsigned share an overlay because the only difference is the AKV issuer name, set by Portal bicep, not by kustomize):

  • afd-letsencrypt/ — AFD + AppGw + AGIC + cert-manager-managed Secret.
  • afd-akv/ — AFD + AppGw + AGIC + Secret Store CSI (AKV cert).
  • private-akv/ — web-app-routing NGINX + ILB + Secret Store CSI (AKV cert, OneCertV2-PrivateCA or Self).

Portal bicep selects the overlay automatically:

kustomizationPath: 'overlays/${edgeMode}-${
  tlsSource == 'akv-selfsigned' ? 'akv' : tlsSource
}'

Skip Logic

deploy.mjs skips entire services based on env flags:

Service Skip when
globalinfra, afd EDGE_MODE != afd
cert-manager TLS_SOURCE != letsencrypt
cert-manager-issuers TLS_SOURCE != letsencrypt

Both single-service runs (deploy.mjs <svc>) and deploy.mjs all honor these gates.

cert-manager Pinning

deploy/gitops/cert-manager/base/helm-release.yaml pins version: 1.20.2 exact (no semver range). To upgrade, edit that field in a PR — Flux will not auto-roll. The OCI HelmRepository points at oci://quay.io/jetstack/charts (official Jetstack registry) for OSS; the enterprise path stays on the AKV path so this chart source is OSS-only.

ClusterIssuers live in a separate Kustomization (cert-manager-issuers) so the issuer install retries cleanly while cert-manager CRDs are landing — Flux retry handles the ordering, no explicit dependsOn between the two fluxConfigurations resources.

Private Mode: NGINX + Internal LB + Private DNS

In EDGE_MODE=private the cluster uses the AKS web-app-routing addon (addonProfiles.webAppRouting) instead of AGIC. The addon installs a managed NGINX ingress controller in the app-routing-system namespace; the default ingress class is webapprouting.kubernetes.azure.com.

Two pieces are wired by deploy.mjs after Flux reconciles the Portal manifests, because they depend on runtime state Bicep can't observe:

  1. Internal LB. deploy.mjs patches the cluster-scoped nginxingresscontroller/default CR with spec.loadBalancerAnnotations.service.beta.kubernetes.io/azure-load-balancer-internal=true. The addon controller propagates the annotation onto the underlying app-routing-system/nginx Service, and Azure recreates the LB as internal-only.
  2. Private DNS A record. deploy.mjs polls the Service for its internal IP, then idempotently upserts an A record ${HOST} → internal-LB IP on the Bicep-provisioned Private DNS Zone (PRIVATE_DNS_ZONE). Re-running the Portal deploy refreshes the record if the LB IP changes.

Callers reach the Portal at https://${HOST}.${PRIVATE_DNS_ZONE} from inside the AKS VNet (or any VNet linked to the same Private DNS Zone: peered VNets, Bastion-attached jump boxes, VPN, ExpressRoute). The zone is not publicly resolvable.

Unsupported Combinations

  • EDGE_MODE=private with TLS_SOURCE=letsencrypt — Let's Encrypt HTTP-01 needs a public IP for ACME validation; private mode has none. DNS-01 against an Azure Public DNS zone is not in scope (we don't provision public zones). Use TLS_SOURCE=akv (OneCertV2-PrivateCA / AME) or TLS_SOURCE=akv-selfsigned for private deployments.
  • EDGE_MODE=afd with TLS_SOURCE=akv-selfsigned — Azure Front Door rejects self-signed origin certs at the TLS validation step. Use TLS_SOURCE=letsencrypt (free, public CA) or TLS_SOURCE=akv with a public CA (e.g. OneCertV2-PublicCA).

new-env.mjs and deploy.mjs both refuse these combos at preflight.

Optional: VPN Gateway P2S (hybrid AFD + VPN)

The IaC path supports an optional, additive Azure VPN Gateway (Point-to-Site, OpenVPN protocol, Microsoft Entra ID authentication) that coexists with EDGE_MODE=afd. The VPN tunnel terminates inside the stamp VNet and reaches the same AppGw private listener as the AFD path, with the same AKV cert — so an allow-listed user (through AFD) and an off-allow-list authenticated user (through VPN) both reach https://<PORTAL_RESOURCE_NAME>.<SSL_CERT_DOMAIN_SUFFIX> and observe an identical cert chain. The Private DNS A record is keyed on PORTAL_RESOURCE_NAME so it matches the AppGw listener hostname and AKV cert subject; RESOURCE_PREFIX alone is only a backwards-compat fallback when PORTAL_RESOURCE_NAME is empty. This is the "trusted-bypass" pattern for tenant users with a valid Entra ID token who would otherwise be blocked at the public edge by operator-defined AFD WAF allow-lists (typically service-tag, IP-range, or header-based rules that gate the public ingress to a known managed-network population). VPN is not a replacement for any existing edge mode.

Architecture

                            (allow-listed public user)
                                  ────────────►  AFD Premium
                                                    │  (Private Link)
                                                    ▼
                                            AppGw v2 (private FE, WAF_v2)
                                                    │  (single AKV cert)
                                                    ▼
                                                AKS portal pod
                                                    ▲
                                                    │
                       (off-allow-list authenticated user)
                                  ────────────►  Azure VPN Gateway P2S
                                  Entra ID + MFA   (GatewaySubnet, OpenVPN)

Both paths share one AppGw, one WAF policy, one cert. The AppGw WAF custom-rules pipeline disambiguates them at L7 (see "WAF guard rules" below).

Required preconditions

  • EDGE_MODE=afd + TLS_SOURCE=akv — validateVpnGatewayCombo() in deploy/scripts/lib/overlay-contracts.mjs requires AFD (code: vpn-requires-afd) and an AKV-family TLS_SOURCE — akv or akv-selfsigned (code: vpn-requires-akv); letsencrypt is rejected because ACME HTTP-01 cannot reach a VPN-only client. The akv-selfsigned variant is also rejected end-to-end on AFD stamps by UNSUPPORTED_COMBOS in deploy/scripts/deploy.mjs (AFD rejects self-signed origin certs at TLS validation), so the only effective combo for the AFD+VPN trusted-bypass is EDGE_MODE=afd + TLS_SOURCE=akv. private mode is rejected because the auto-seeded WAF guards assume AFD as the public ingress.
  • SSL_CERT_DOMAIN_SUFFIX must be set — the managed Private DNS zone uses it (code: vpn-requires-domain-suffix).
  • VPN_CLIENT_ADDRESS_POOL must not overlap the VNet VNET_CIDR (default 10.20.0.0/16); validated against both endpoints (code: vpn-pool-overlap). Default pool is 172.16.200.0/24 (~250 concurrent clients).
  • AZURE_TENANT_ID must be set — VPN runs in the same Entra ID tenant as the rest of the stamp (code: vpn-requires-tenant-id).
  • Tenant admin access to author the Conditional Access policy below.

Env vars

Var Default Notes
VPN_GATEWAY_ENABLED false Master switch. true provisions the gateway + GatewaySubnet + managed Private DNS zone + Azure Private DNS Resolver inbound endpoint (with its own /28 subnet), and seeds the WAF guard rules. The Resolver IP is advertised to P2S clients via the parent VNet's dhcpOptions.dnsServers (the supported P2S DNS-push path) so the portal hostname resolves through the tunnel without hosts-file edits.
VPN_GATEWAY_SKU VpnGw2AZ VpnGw2AZ, VpnGw3AZ, VpnGw4AZ, or VpnGw5AZ. Generation2 AZ SKUs only. VpnGw1AZ (Generation1) is excluded — silently drops OpenVPN+AAD HardResetClientV2 packets ~5s after TCP accept with no diagnostic event. Non-AZ SKUs (VpnGw1/2/3) and Basic are also rejected.
VPN_CLIENT_ADDRESS_POOL 172.16.200.0/24 CIDR assigned to connected clients. MUST NOT overlap the VNet.
VPN_AAD_AUDIENCE c632b3df-fb67-4d84-bdcf-b95ad541b5c8 Microsoft-registered Azure VPN Client app. Set to 41b23e61-6c1e-4545-b367-cd054e0ed4b4 only on tenants that must interop with older Azure VPN client builds.
APPGW_WAF_CUSTOM_RULES_FILE unset See AppGw WAF custom rules below — a general-purpose facility, not VPN-specific.

Conditional Access policy (tenant admin, required before first connect)

Create one CA policy targeting the Azure VPN Client app:

  • Target app: c632b3df-fb67-4d84-bdcf-b95ad541b5c8 (default), or 41b23e61-6c1e-4545-b367-cd054e0ed4b4 if you set VPN_AAD_AUDIENCE to the legacy override.
  • Assignment: a NAMED users group. Do not target "all users" — every Entra principal in your tenant would otherwise inherit VPN reachability to the AppGw private FE.
  • Grant: require MFA.
  • Grant: do NOT require device compliance — the OpenVPN client cannot satisfy that grant and connect attempts will fail with an opaque AAD error.

The post-scaffold reminder block in new-env.mjs re-prints these requirements when VPN_GATEWAY_ENABLED=true. The full skill-side reference lives in pilotswarm-new-env-deploy (Step 4 → Optional: VPN Gateway P2S).

Distributing the VPN client profile

After the first deploy completes, hand operators the OpenVPN client profile via one of these paths (in order of preference):

  • Helper script (recommended): pwsh -File deploy/scripts/auth/Get-VpnClientProfile.ps1 -EnvName <stamp>. Wraps the az network vnet-gateway vpn-client generate call, downloads the signed zip, and extracts it under the gitignored deploy/envs/local/<stamp>/vpn-client/ folder. See the pilotswarm-vpn-client-profile skill for full usage.
  • Azure portal: Resource group → <gateway-name> → Point-to-site configuration → Download VPN client.
  • CLI: az network vnet-gateway vpn-client generate --resource-group <rg> --name <gateway-name> --authentication-method EAPTLS.

All three emit the same .zip that imports directly into the Azure VPN Client app (Windows / macOS / iOS / Android). The profile embeds the AAD audience and the gateway public IP — re-issue it (-Force on the helper script) if you rotate VPN_AAD_AUDIENCE. The XML carries no per-user credentials; end users authenticate with their own Entra ID at connect time.

WAF guard rules (auto-seeded at priorities 90 / 91 / 92)

When VPN_GATEWAY_ENABLED=true, base-infra bicep (deploy/services/base-infra/bicep/application-gateway.bicep) prepends three custom rules to the AppGw WAF policy's customRules.rules array:

Priority Name Action Match
90 AllowAfd Allow RequestHeaders[X-Azure-FDID] == <frontDoorId> (threaded from global-infra)
91 AllowVpn Allow RemoteAddr ∈ VPN_CLIENT_ADDRESS_POOL (IPMatch)
92 BlockOther Block RemoteAddr ∈ 0.0.0.0/0 (catch-all)

Together: AFD-origin-authenticity check (90) + VPN-pool allow-list (91) + catch-all block (92). The catch-all is belt-and-braces against future NSG/route changes accidentally exposing the AppGw private FE outside both paths.

The 90–92 priority band is reserved. Operator-supplied rules from APPGW_WAF_CUSTOM_RULES_FILE (next section) MUST start at priority ≥ 100; the bicep concats them after the auto-seeded set.

When VPN_GATEWAY_ENABLED=false, no auto-seeded rules are emitted — a VPN-off stamp's AppGw WAF policy diff vs a no-VPN baseline is empty.

AppGw WAF custom rules (applies to any AFD stamp)

APPGW_WAF_CUSTOM_RULES_FILE is a general-purpose facility, parallel to the AFD-side WAF_CUSTOM_RULES_FILE, and works on any EDGE_MODE=afd stamp regardless of whether VPN is enabled. Point it at a JSON array file containing AppGw WAF custom rule objects; the orchestrator resolves the path (relative-to-repo-root or absolute), parses the JSON, and threads it into the bicep deploy as appgwWafCustomRules.

# Recommended location (gitignored under deploy/envs/local/<stamp>/):
APPGW_WAF_CUSTOM_RULES_FILE=deploy/envs/local/<stamp>/appgw-waf-custom-rules.json
// Example contents — operator priorities start at 100 to leave
// 90–92 free for the auto-seeded VPN guards.
[
  {
    "name": "BlockExploitUA",
    "priority": 100,
    "ruleType": "MatchRule",
    "action": "Block",
    "matchConditions": [
      {
        "matchVariables": [
          { "variableName": "RequestHeaders", "selector": "User-Agent" }
        ],
        "operator": "Contains",
        "matchValues": ["sqlmap", "nikto"]
      }
    ]
  }
]

The merge logic mirrors the bicep exactly in deploy/scripts/lib/appgw-waf-rules.mjs so test cases can assert merged rule shape without shelling out to az. Missing-file or non-JSON-array inputs fail loudly at preflight with a single named diagnostic.

Cost and time

  • Cost: ~$450/month total for a VPN-enabled stamp:
    • ~$280/month for VpnGw2AZ (Public IP + gateway hours). Higher AZ SKUs (VpnGw3AZ / VpnGw4AZ / VpnGw5AZ) scale linearly — see Azure VPN Gateway pricing for current rates. VpnGw1AZ (Generation1) and non-AZ SKUs (VpnGw1/2/3) are excluded — see the SKU notes above.
    • ~$170/month for the Azure Private DNS Resolver inbound endpoint, which is co-provisioned with the VPN gateway so P2S clients can resolve Private DNS Zone records (e.g. the portal hostname) through the tunnel without hosts-file edits. P2S clients cannot reach the Azure-magic 168.63.129.16 resolver — that IP only works from inside Azure VMs — so the Resolver inbound endpoint sits on a regular VNet IP. P2S clients inherit this IP via the parent VNet's dhcpOptions.dnsServers block, which the gateway pushes at connect time (the classic VPN gateway resource has no dedicated DNS-push property — VNet DHCP options are the supported path).
  • First-deploy time: 45+ minutes. Gateway provisioning is the long pole; the rest of the stamp finishes well before the gateway reports Succeeded. Subsequent param-change deploys are minutes, not 45+.

See also

  • pilotswarm-new-env-deploy skill — full step-by-step including the EDGE_MODE × TLS_SOURCE × VPN_GATEWAY_ENABLED matrix, scaffolder prompts, and the post-scaffold reminder.
  • pilotswarm-aks-deploy skill — legacy bash deploy path; surfaces VPN cost / time as operator context but does not orchestrate VPN itself.

Model Providers (LLM catalog)

The IaC path mounts the worker's model catalog as a kustomize-generated ConfigMap (copilot-worker-model-providers) sourced from deploy/gitops/worker/base/model_providers.json, exposed to the runtime via PS_MODEL_PROVIDERS_PATH=/app/config/model_providers.json. This is separate from the legacy scripts/deploy-aks.sh flow, which bakes deploy/config/model_providers.ghcp.json into the image.

Built-in providers in the base catalog:

Provider Auth secret (KV → SPC) Endpoint When it loads
github-copilot GITHUB_TOKEN https://api.githubcopilot.com Always
anthropic (direct) ANTHROPIC_API_KEY https://api.anthropic.com When the key is set (sentinel-tolerant)
azure-foundry, azure-kimi AZURE_OAI_KEY __FOUNDRY_ENDPOINT__/openai/v1 Only when FOUNDRY_ENABLED=true

__FOUNDRY_ENDPOINT__ is rewritten to the live Foundry account URL during --steps manifests (from the FOUNDRY_ENDPOINT Bicep output). When Foundry is disabled, the placeholder stays in the file and the worker's catalog loader silently drops the Foundry providers (apiKey: env:AZURE_OAI_KEY resolves to the stripped __PS_UNSET__ sentinel, i.e. undefined).

Enabling Foundry on a stamp

npm run deploy:new-env -- <name> --foundry-enabled y
# scaffolds deploy/envs/local/<name>/foundry-deployments.json
# (a JSON array; the stdout banner lists common entries to copy in)

# edit the JSON to your desired model deployments, then:
npm run deploy -- --env <name>

foundry.bicep provisions one Microsoft.CognitiveServices/accounts (kind=AIServices) per stamp, each entry as a child accounts/deployments, and writes azure-oai-key directly to KV via co-located listKeys(). When FOUNDRY_ENABLED=false, auto-secrets-sentinel.bicep writes the __PS_UNSET__ sentinel into the same KV secret so the SPC mount still succeeds.

Phase 1 only. Foundry uses key auth; Claude is direct-Anthropic. Phase 2 (SDK Entra-mode for Foundry) and Phase 3 (Foundry-hosted Claude) are tracked in docs/proposals/foundry-entra-mode-auth.md and docs/proposals/foundry-hosted-claude.md.

Per-stamp catalog overrides

Drop a kustomize overlay patch on the copilot-worker-model-providers ConfigMap in deploy/gitops/worker/overlays/<overlay>/ to diverge from the base catalog for one stamp. Keep __FOUNDRY_ENDPOINT__ in the patched JSON to keep endpoint substitution; hard-code the URL to opt out.

Local Development

Local dev outside Azure (e.g. kind, k3d, plain Docker) is not part of the IaC path. Use the legacy local scripts (./run.sh, scripts/deploy-aks.sh) for those scenarios. The IaC path assumes an Azure target.

Querying Logs (KQL)

The base-infra Bicep provisions a per-stamp Log Analytics workspace (<RESOURCE_PREFIX>-log) and an AKS Container Insights Data Collection Rule that ships pod stdout/stderr to the workspace using the modern ContainerLogV2 schema. This gives you historical, queryable logs that survive pod restarts — a step up from kubectl logs, which only shows the current and previous container instance.

Find the workspace:

# From base-infra deployment outputs (set after `--steps bicep`):
az deployment group show \
  -g "$RESOURCE_GROUP" \
  -n base-infra \
  --query "properties.outputs.logAnalyticsWorkspaceName.value" -o tsv
# → <RESOURCE_PREFIX>-log

Open the workspace in the Azure portal → Logs, and run KQL like:

// Last 200 portal log lines
ContainerLogV2
| where PodNamespace == "pilotswarm"
| where PodName startswith "pilotswarm-portal"
| order by TimeGenerated desc
| take 200
| project TimeGenerated, PodName, ContainerName, LogMessage

// Worker errors in the last 1h
ContainerLogV2
| where PodNamespace == "pilotswarm"
| where PodName startswith "copilot-runtime-worker"
| where TimeGenerated > ago(1h)
| where LogLevel in ("error", "warn") or LogMessage contains_cs "ERROR"
| order by TimeGenerated desc

// Pod restarts / OOMKills
KubeEvents
| where Namespace == "pilotswarm"
| where Reason in ("BackOff", "Failed", "OOMKilling", "Killing")
| order by TimeGenerated desc
| project TimeGenerated, Name, Reason, Message

Notes:

  • Ingestion lag is ~3–10 minutes. Tail follow-ups should still use kubectl logs -f for live debugging; KQL is the historical view.
  • Retention defaults to 30 days (free tier). Tune via LOG_ANALYTICS_RETENTION_DAYS in deploy/envs/local/<env>/env. Up to 730 days is supported; 30+ is billed at the workspace's PerGB2018 rate.
  • ContainerLogV2 is the only schema enabled. The legacy ContainerLog table is deprecated (retiring 2026-09-30) and is not populated.
  • Cost. A small dev stamp typically lands at 1–3 GB/day; ingestion is ~$2.30/GB after the free 5 GB/month per workspace. Use the Usage | summarize sum(Quantity) by DataType query to monitor.

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│  Your App (Client)                                                   │
│  PilotSwarmClient({ apiUrl: "https://<portal-url>" })                 │
│  → createSession, sendAndWait, on()                                  │
└────────────────────┬─────────────────────────────────────────────────┘
                     │ HTTP /api/v1 + WebSocket /api/v1/ws
                     ▼
┌──────────────────────────────────────────────────────────────────────┐
│  Portal Pod (hosts the versioned Web API, port 3001)                  │
└────────────────────┬─────────────────────────────────────────────────┘
                     │ PostgreSQL
                     ▼
┌──────────────────────────────────────────────────────────────────────┐
│  PostgreSQL (Azure Database for PostgreSQL)                           │
│  ┌─────────────────┐  ┌──────────────────┐                           │
│  │ duroxide schema  │  │ copilot_sessions │                           │
│  │ (orchestrations) │  │ (session catalog)│                           │
│  └─────────────────┘  └──────────────────┘                           │
└────────────────────┬─────────────────────────────────────────────────┘
                     │ PostgreSQL
                     ▼
┌──────────────────────────────────────────────────────────────────────┐
│  AKS Worker Pods (N replicas)                                        │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐                │
│  │ worker-1 │ │ worker-2 │ │ worker-3 │ │ worker-N │                │
│  │ polls PG │ │ polls PG │ │ polls PG │ │ polls PG │                │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘                │
│                                                                      │
│  Each pod: node packages/sdk/examples/worker.js                                   │
│  → Picks up orchestrations from the queue                            │
│  → Runs LLM turns via Copilot SDK                                   │
│  → Dehydrates/hydrates sessions via Azure Blob Storage               │
└──────────────────────────────────────────────────────────────────────┘

Client apps talk to the versioned Web API hosted by the portal — see docs/api/reference.md. Constructing a client with { store: DATABASE_URL } still exists but is internal (workers, portal-host embedding, and internal testing only). Workers always use { store } — they are trusted backend components with direct database access.

Prerequisites

  • Azure CLI (az) installed and logged in
  • kubectl configured for your AKS cluster
  • An Azure Container Registry (ACR) for Docker images
  • An Azure Database for PostgreSQL (Flexible Server)
  • An Azure Storage Account (for session blob storage)

WARNING: Runaway Deployments

Before deploying, always check for old worker pods in other namespaces that may still be connected to the same database:

kubectl get pods --all-namespaces -l app.kubernetes.io/component=worker --no-headers

Old workers from a previous namespace (e.g. copilot-sdk vs copilot-runtime) will process orchestrations with stale orchestration code, causing nondeterminism errors. Delete any old deployments before deploying:

kubectl delete deployment copilot-runtime-worker -n <old-namespace>

Step 1: Create Kubernetes Resources

Prefer The Repo Script

For this repository, prefer the canonical deploy/reset path:

./scripts/deploy-aks.sh

That script refreshes the Kubernetes secret, optionally wipes remote state, builds the SDK, pushes the worker image, and waits for rollout completion.

Namespace

kubectl apply -f deploy/k8s/namespace.yaml

This creates the copilot-runtime namespace.

Secrets

PilotSwarm's checked-in model-catalog template is .model_providers.example.json. Workers actually load the local .model_providers.json, which is gitignored so teams can keep personal endpoint URLs and similar local details out of source control. The Kubernetes secret only needs the env vars referenced by that real runtime catalog and the worker runtime.

Store your credentials as a Kubernetes secret:

kubectl create secret generic copilot-runtime-secrets \
    -n copilot-runtime \
    --from-literal=DATABASE_URL="postgresql://user:pass@myserver.postgres.database.azure.com:5432/postgres?options=-csearch_path%3Dcopilot_runtime&sslmode=require" \
    --from-literal=GITHUB_TOKEN="ghp_xxxxxxxxxxxx" \
    --from-literal=DUROXIDE_PG_POOL_MAX="10" \
    --from-literal=PILOTSWARM_CMS_PG_POOL_MAX="3" \
    --from-literal=PILOTSWARM_FACTS_PG_POOL_MAX="3" \
    --from-literal=PILOTSWARM_ORCHESTRATION_CONCURRENCY="2" \
    --from-literal=PILOTSWARM_WORKER_CONCURRENCY="2" \
    --from-literal=PILOTSWARM_TURN_TIMEOUT_MS="1200000" \
    --from-literal=AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=..." \
    --from-literal=AZURE_STORAGE_CONTAINER="copilot-sessions"

Worker sizing is env-driven:

  • DUROXIDE_PG_POOL_MAX — duroxide-pg provider pool size. Default: 10.
  • PILOTSWARM_CMS_PG_POOL_MAX — CMS pg.Pool max size. Default: 3.
  • PILOTSWARM_FACTS_PG_POOL_MAX — facts pg.Pool max size. Default: 3.
  • PILOTSWARM_ORCHESTRATION_CONCURRENCY — Duroxide orchestration concurrency. Default: 2.
  • PILOTSWARM_WORKER_CONCURRENCY — Duroxide activity/worker concurrency. Default: 2.
  • PILOTSWARM_TURN_TIMEOUT_MS — wall-clock cap for one Copilot turn. Default: 1200000 (20 minutes); 0 disables it.

Provider availability in selectors is env-driven at worker startup. If you add or remove a provider key, refresh the secret and restart the workers; changing the checked-in template alone is not enough, and changing the real .model_providers.json only takes effect after the updated file is present in the runtime environment.

Portal Pods

If you also deploy the shipped browser portal, treat it as a separate runtime surface from the worker pods:

  • package the same app plugin into the portal image
  • set PLUGIN_DIRS in the portal deployment so the web process can read plugin.json.portal, plugin.json.tui, creatable agent metadata, and session policy
  • keep portal branding in plugin.json.portal, using plugin.json.tui as a fallback or shared source only when that is intentional

If the portal pod cannot see the app plugin, the browser UI falls back to generic PilotSwarm branding and generic-session creation even when the worker supports named agents.

The portal process also hosts the versioned Web API — HTTP /api/v1 and WebSocket /api/v1/ws — used by the SDK's web mode, the pilotswarm-sdk/api package, and pilotswarm remote --api-url (see docs/api/reference.md). This requires no new ingress rules: the existing portal ingress (path / prefix, 3600s WebSocket timeouts) already covers /api/v1 and /api/v1/ws, and the readiness probe stays /api/health. TUI and SDK users of the deployment need exactly one value: the portal URL.

Portal auth is provider-based. For the shipped Entra add-on, add these env vars to copilot-runtime-secrets (or the portal deployment env) before restarting the portal:

PORTAL_AUTH_PROVIDER=entra
PORTAL_AUTH_ENTRA_TENANT_ID=<tenant-id>
PORTAL_AUTH_ENTRA_CLIENT_ID=<client-id>
PORTAL_AUTHZ_ADMIN_GROUPS=admin1@contoso.com,admin2@contoso.com
PORTAL_AUTHZ_USER_GROUPS=user1@contoso.com,user2@contoso.com

Register the portal ingress URL as the SPA redirect URI in Entra. The portal core does not require Entra specifically, so alternate providers can use the same deployment slot without changing the portal shell contract.

Automating the app registration. Rather than clicking through the Azure Portal, use deploy/scripts/auth/Setup-PortalAuth.ps1 to create the Entra application, register the SPA redirect URI, and (optionally) define the admin/user app roles consumed by the portal authz engine. The script requires -ServiceTreeId (operator-supplied) and exposes -CreateAppRoles (recommended for production stamps) and -AssignmentRequired (advanced opt-in — see caveat in the README) switches for role-driven and Entra-level lockdown postures. The recommended production lockdown is -CreateAppRoles plus role assignments in Entra (the role assignment is the allowlist); the portal engine is deny-by-default for any signed-in principal without a role claim. Full operator docs: deploy/scripts/auth/README.md. Agent-driven invocation: pilotswarm-portal-app-reg skill.

Use the canonical PORTAL_AUTH_* / PORTAL_AUTHZ_* keys only. The portal no longer reads legacy ENTRA_* aliases.

For tenants driving admission from Entra app roles instead of an email allowlist, see docs/portal-entra-app-roles.md for the recommended end-state setup (define roles → assign principals → rely on engine deny-by-default → optionally enable appRoleAssignmentRequired=true if your tenant allows it without tripping admin-consent → align CA). In the Roles posture, the role assignment in Entra is the allowlist — do not also populate PORTAL_AUTHZ_ADMIN_GROUPS, since the engine bypasses it whenever the JWT carries a roles[] claim.

Portal authz supports both email allowlists and Entra app-role claims:

  • authenticated users whose email appears in the configured admin/user allowlists are allowed in
  • authenticated users whose token carries an Entra app-role claim (admin / user) are decided from that claim; see portal-entra-app-roles.md
  • admin and user have the same portal permissions today
  • per-user session visibility is a later phase

Refresh GitHub Token

The GitHub token expires periodically. To update:

kubectl create secret generic copilot-runtime-secrets \
    -n copilot-runtime \
    --from-literal=DATABASE_URL="..." \
    --from-literal=GITHUB_TOKEN="$(gh auth token)" \
    --from-literal=DUROXIDE_PG_POOL_MAX="10" \
    --from-literal=PILOTSWARM_CMS_PG_POOL_MAX="3" \
    --from-literal=PILOTSWARM_FACTS_PG_POOL_MAX="3" \
    --from-literal=PILOTSWARM_ORCHESTRATION_CONCURRENCY="2" \
    --from-literal=PILOTSWARM_WORKER_CONCURRENCY="2" \
    --from-literal=PILOTSWARM_TURN_TIMEOUT_MS="1200000" \
    --from-literal=AZURE_STORAGE_CONNECTION_STRING="..." \
    --from-literal=AZURE_STORAGE_CONTAINER="copilot-sessions" \
    --dry-run=client -o yaml | kubectl apply -f -

The same pattern applies to Azure/OpenAI or Anthropic BYOK keys. If a provider should disappear from selectors, make sure its env var is absent when the secret is reapplied, then restart the deployment and verify the live model surface.

Step 2: Build and Push Docker Image

Login to ACR

az acr login --name <your-acr-name>

Build and Push

# Build TypeScript first
npm run build

# Build and push Docker image
docker buildx build \
    --platform linux/amd64 \
    -f deploy/Dockerfile.worker \
    -t <your-acr-name>.azurecr.io/copilot-runtime-worker:latest \
    --push .

The Dockerfile (deploy/Dockerfile.worker) builds a minimal image:

  • node:24-trixie-slim base
  • Production dependencies only (npm ci --omit=dev --force)
  • Copies packages/sdk/dist/ and packages/sdk/examples/worker.js
  • Runs as non-root node user

Step 3: Deploy Workers

Edit the Deployment

Update deploy/k8s/worker-deployment.yaml with your ACR URL:

containers:
  - name: worker
    image: <your-acr-name>.azurecr.io/copilot-runtime-worker:latest

Apply

kubectl apply -f deploy/k8s/worker-deployment.yaml

Verify

kubectl get pods -n copilot-runtime -l app.kubernetes.io/component=worker

Expected output:

NAME                                  READY   STATUS    RESTARTS   AGE
copilot-runtime-worker-xxxxx-aaaaa        1/1     Running   0          30s
copilot-runtime-worker-xxxxx-bbbbb        1/1     Running   0          30s
copilot-runtime-worker-xxxxx-ccccc        1/1     Running   0          30s
copilot-runtime-worker-xxxxx-ddddd        1/1     Running   0          30s

Check Logs

kubectl logs -n copilot-runtime -l app.kubernetes.io/component=worker --prefix --tail=20

You should see:

[pod/copilot-runtime-worker-xxxxx/worker] [worker] Pod: copilot-runtime-worker-xxxxx
[pod/copilot-runtime-worker-xxxxx/worker] [worker] Started ✓ Polling for orchestrations...

After a cold start or destructive reset, the workers will automatically recreate the built-in system sessions (PilotSwarm Agent, Sweeper Agent, Resource Manager Agent, Facts Manager). A truly empty session list is therefore temporary.

Reset Remote State For Reproduction Or Replay Cleanup

Deploys never reset data. A reset is a separate, deliberate operation — run it only when a wipe was explicitly requested:

./scripts/reset-db-aks.sh --i-understand-this-deletes-all-data

This scales workers to 0, wipes the database, and restores the previous replica count. (Equivalent manual sequence: scale to 0 → NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts/db-reset.js --yes → scale back up.)

This drops:

  • duroxide
  • copilot_sessions
  • pilotswarm_facts
  • all blobs in copilot-sessions when blob storage is configured

After the workers come back, expect the built-in system sessions to be recreated immediately. For replay-sensitive testing, verify that the recreated root PilotSwarm Agent is healthy before starting new user sessions.

Step 4: Connect Your Client

From your application (anywhere that can reach the portal URL over HTTPS):

import { PilotSwarmClient } from "pilotswarm-sdk";

const client = new PilotSwarmClient({
    apiUrl: "https://<portal-url>",
    // getAccessToken: async () => "...", // when the portal runs Entra auth
});
await client.start();

// Sessions are processed by AKS worker pods
const session = await client.createSession();
await session.send("Monitor this service every 5 minutes for the next 24 hours");

console.log(`Session ${session.sessionId} is running on AKS`);

The client talks to the portal's Web API (/api/v1 + /api/v1/ws) — see docs/api/reference.md. Direct { store: DATABASE_URL } construction is internal (workers, portal-host embedding, and internal testing only); don't use it for client apps.

Or use the TUI in remote mode — the only value you need is the portal URL:

npx pilotswarm remote --api-url https://<portal-url>

Auth is discovered from the deployment: a no-auth portal starts immediately; an Entra-protected portal opens your browser for an interactive sign-in in the terminal, with tokens cached at ~/.config/pilotswarm/auth/. You can also sign in ahead of time (or check/clear the cache) with:

npx pilotswarm auth login --api-url https://<portal-url>
npx pilotswarm auth status --api-url https://<portal-url>
npx pilotswarm auth logout --api-url https://<portal-url>

No DATABASE_URL and no kubectl needed — logs stream over the API. (PILOTSWARM_API_URL, e.g. in .env.remote, works instead of the flag.)

Operator note. pilotswarm remote --store "$DATABASE_URL" still works for direct database access, but it is for operators/internal use. --api-url and --store together are an error.

Scaling

Horizontal Scaling

Adjust the replica count:

kubectl scale deployment copilot-runtime-worker -n copilot-runtime --replicas=8

Workers are stateless — each polls the PostgreSQL queue for available work. duroxide ensures exactly-once execution.

Resource Tuning

The default resource requests/limits in the deployment:

resources:
    requests:
        cpu: "250m"
        memory: "1Gi"
    limits:
        cpu: "1000m"
        memory: "4Gi"

Each worker runs one LLM turn at a time. Increase CPU limits if tool execution is compute-heavy.

Spot Instances

The deployment includes a toleration for Azure spot instances:

tolerations:
    - key: "kubernetes.azure.com/scalesetpriority"
      operator: "Equal"
      value: "spot"
      effect: "NoSchedule"

Spot instances are safe because sessions are durable — if a spot node is evicted, the orchestration retries automatically on another node.

Updating Workers

Rolling Update

# Rebuild and push
npm run build
docker buildx build --platform linux/amd64 -f deploy/Dockerfile.worker \
    -t <your-acr-name>.azurecr.io/copilot-runtime-worker:latest --push .

# Restart pods (pulls latest image)
kubectl rollout restart deployment/copilot-runtime-worker -n copilot-runtime

# Wait for rollout to complete
kubectl rollout status deployment/copilot-runtime-worker -n copilot-runtime

In-flight orchestrations are safe during rollouts. If a worker is killed mid-turn, duroxide will retry the activity on another worker after the lock timeout.

Database Reset

To wipe all orchestration and session state (only when a wipe was explicitly requested — deploys never do this):

./scripts/reset-db-aks.sh --i-understand-this-deletes-all-data

This drops the ps_duroxide, copilot_sessions, and pilotswarm_facts schemas (plus the legacy duroxide schema if present). Use with caution — all in-flight sessions will be lost.

Troubleshooting

Workers Not Picking Up Work

# Check pods are running
kubectl get pods -n copilot-runtime -l app.kubernetes.io/component=worker

# Check logs for errors
kubectl logs -n copilot-runtime -l app.kubernetes.io/component=worker --tail=50

# Verify database connectivity
kubectl exec -n copilot-runtime deploy/copilot-runtime-worker -- \
    node -e "console.log('DB OK')" --env-file=/dev/null

Session Stuck in "running"

A session may be stuck if the activity timed out. Check the orchestration status:

# From your machine, via the deployment's Web API
# (.env.remote carries PILOTSWARM_API_URL)
node --env-file=.env.remote -e "
    import { PilotSwarmClient } from './dist/index.js';
    const c = new PilotSwarmClient({ apiUrl: process.env.PILOTSWARM_API_URL });
    await c.start();
    const s = await c.resumeSession('SESSION_ID');
    console.log(await s.getInfo());
    await c.stop();
"

(Operators with direct database access can construct the client with { store: process.env.DATABASE_URL } instead — internal use only.)

GitHub Token Expired

If workers log authentication errors, refresh the secret:

kubectl create secret generic copilot-runtime-secrets -n copilot-runtime \
    --from-literal=GITHUB_TOKEN="$(gh auth token)" \
    --dry-run=client -o yaml | kubectl apply -f -

# Restart workers to pick up new secret
kubectl rollout restart deployment/copilot-runtime-worker -n copilot-runtime

Sharing An Existing AKS Cluster

Multiple teams or projects can share one AKS cluster. Each deployment gets its own Kubernetes namespace, secrets, and optionally its own database schemas.

Option A: Separate Databases (Simplest)

Each deployment uses a different PostgreSQL database on the same server. No code changes needed — just different DATABASE_URLs.

Team Alpha: postgresql://user:pass@pg-server:5432/alpha_pilotswarm
Team Beta:  postgresql://user:pass@pg-server:5432/beta_pilotswarm

Option B: Separate Schemas (Same Database)

Use custom schema names to isolate deployments within a single database. Set duroxideSchema and cmsSchema on both worker and client (see Getting Started → Custom Schema Names).

Setup Per Team

Each team creates their own namespace and secrets:

TEAM_NS=copilot-alpha

kubectl create namespace $TEAM_NS

kubectl create secret generic copilot-runtime-secrets \
    -n $TEAM_NS \
    --from-literal=DATABASE_URL="postgresql://..." \
    --from-literal=GITHUB_TOKEN="$(gh auth token)" \
    --from-literal=AZURE_STORAGE_CONNECTION_STRING="..." \
    --from-literal=AZURE_STORAGE_CONTAINER="alpha-sessions"

Copy and customize the deployment manifests:

cp deploy/k8s/worker-deployment.yaml deploy/k8s/worker-deployment-alpha.yaml

Edit the copy to update:

  • metadata.namespace → your team namespace
  • spec.template.spec.containers[0].image → your ACR image

Then deploy:

kubectl apply -f deploy/k8s/worker-deployment-alpha.yaml

Connect The TUI To A Specific Namespace

Point the TUI at that deployment's portal URL — logs stream over the API, so no kubectl access or namespace/label flags are needed:

npx pilotswarm remote --api-url https://<alpha-portal-url>

If the deployment runs Entra auth, the TUI walks you through a browser sign-in (or run npx pilotswarm auth login --api-url https://<alpha-portal-url> first).

Operator note. Direct-store mode still works and is where the K8s log-tail flags (-c/--namespace/--label) apply:

node packages/app/tui/bin/tui.js remote \
    --env .env.alpha \
    --namespace copilot-alpha \
    --label app.kubernetes.io/component=worker

Resource Isolation

For tighter isolation, use Kubernetes resource quotas:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: copilot-quota
  namespace: copilot-alpha
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 4Gi
    limits.cpu: "8"
    limits.memory: 8Gi
    pods: "10"