[P1] Implement canary deployments

[wonderwomancode]

Problem

Currently direct deploys to production proxy. No way to test changes with subset of traffic first.

Acceptance Criteria

• Can deploy canary version alongside stable
• Can route X% of traffic to canary (configurable)
• Can route by header (e.g., X-Canary: true)
• Metrics differentiate canary vs stable
• Can rollback to 0% canary instantly
• Documentation for canary deployment process

Traffic Splitting Options

Option A: Weighted Upstreams (Recommended)

[upstreams.api-stable]
addrs = ["api-stable:4000"]
weight = 90

[upstreams.api-canary]
addrs = ["api-canary:4000"]
weight = 10

Option B: Header-Based Routing

[locations.api-canary]
upstream = "api-canary"
headers = ["X-Canary: true"]

[locations.api-stable]
upstream = "api-stable"

Implementation Steps

1. Deploy canary service

Deploy second instance of service-cloud-api with different DSEQ:

# deploy-canary.yaml with different image tag
akash tx deployment create deploy-canary.yaml

2. Update proxy config for weighted routing

[upstreams.api]
addrs = ["api-stable:4000", "api-canary:4000"]
weights = [90, 10]  # 90% stable, 10% canary

# Or use etcd for dynamic updates

3. Add canary label to metrics

Ensure metrics include version label to differentiate:

pingap_requests_total{upstream="api",version="stable"} 900
pingap_requests_total{upstream="api",version="canary"} 100

4. Create deployment workflow

File: .github/workflows/deploy-canary.yml

name: Deploy Canary

on:
  workflow_dispatch:
    inputs:
      traffic_percentage:
        description: 'Percentage of traffic to canary (0-100)'
        required: true
        default: '10'

Rollback Procedure

# Set canary weight to 0
etcdctl put /pingap/config/upstreams/api-canary/weight 0

# Or update pingap.toml and redeploy

Testing

# Send 100 requests, verify ~10% go to canary
for i in {1..100}; do curl -s https://api.alternatefutures.ai/health; done

# Check metrics
curl http://proxy:3018/metrics | grep canary

Definition of Done

• Canary deployment documented
• Weight-based routing working
• Header-based routing working (optional)
• Metrics distinguish canary vs stable
• Rollback procedure documented and tested
• Team trained on canary process