Add gateway capacity cost planning proof

ARCHITECTURE.md

@@ -11,6 +11,7 @@ flowchart LR
     API --> Model["Mock Inference Backend"]
     API --> Audit["JSONL Audit Sink"]
     API --> TraceExport["Sanitized Trace Export"]
+    Policy --> Capacity["Synthetic Capacity Plan"]
 ```
 
 ## Components
@@ -23,6 +24,7 @@ flowchart LR
 - `gateway/token_budget.py`: deterministic estimated input-token accounting.
 - `gateway/audit.py`: structured JSONL audit events with identity and trace evidence.
 - `gateway/trace_exporter.py`: opt-in sanitized trace spans for local OpenTelemetry-shaped evidence.
+- `gateway/capacity_plan.py`: synthetic capacity and cost-to-serve planning from model policy and benchmark assumptions.
 - `gateway/mock_inference.py`: synthetic model response with latency metadata.
 - `gateway/registry.py`: demo principals and model policies.
 - `deploy/prometheus` and `deploy/grafana`: local metrics scrape and dashboard provisioning.
@@ -35,4 +37,5 @@ flowchart LR
 - Replace in-memory request and token-budget limiting with Redis or Envoy global rate limits.
 - Upgrade sanitized trace JSONL to OpenTelemetry SDK export through an OTLP collector while keeping Prometheus metrics for low-cardinality service health.
 - Capture GPU telemetry from DCGM and attach it to inference metrics.
+- Replace synthetic capacity assumptions with measured backend profiles after real model-serving integration exists.
 - Add per-model data handling rules and prompt/output redaction.

README.md

@@ -1,6 +1,6 @@
 # Secure GPU Inference Gateway
 
-Security-focused AI infrastructure demo for OIDC/JWT-authenticated model access, role-based authorization, per-model request and token-budget limits, trace-aware audit logs, and policy-driven inference routing.
+Security-focused AI infrastructure demo for OIDC/JWT-authenticated model access, role-based authorization, per-model request and token-budget limits, trace-aware audit logs, synthetic capacity/cost planning, and policy-driven inference routing.
 
 This repository uses a mock inference backend so the security and infrastructure logic can be reviewed without GPU hardware, model weights, proprietary data, or cloud credentials.
 
@@ -21,6 +21,7 @@ This is the flagship portfolio project for the platform-security-to-AI-infrastru
 - W3C `traceparent` propagation for OpenTelemetry-compatible request correlation.
 - Prometheus-compatible `/metrics` endpoint for authentication, policy, limiter, token-throughput, and latency telemetry.
 - Opt-in sanitized trace JSONL export for local OpenTelemetry-shaped span evidence.
+- Synthetic capacity and cost-to-serve planning artifact tied to configured model policies.
 - Prometheus and Grafana provisioning files for local observability review.
 - Mock GPU inference backend with latency metadata.
 - Focused unit tests for policy and limiter behavior.
@@ -49,6 +50,7 @@ Relevant areas:
 - Review `gateway/audit.py` for structured evidence.
 - Review `gateway/trace_exporter.py` for sanitized trace span export without prompt, output, access-reason, or principal identifiers.
 - Review `gateway/trace_context.py` for W3C trace context parsing and response propagation.
+- Review `gateway/capacity_plan.py` and `artifacts/capacity-plan-evidence.json` for aggregate synthetic capacity and cost-to-serve modeling.
 - Review `deploy/grafana/dashboards/security-gateway.json` for dashboard queries over the gateway metrics.
 - Review `docs/OPERATIONS.md` and `deploy/kubernetes/gateway.yaml` for SLO/runbook and deployment thinking.
 - Check `tests/` for behavior-focused coverage.
@@ -85,6 +87,14 @@ uvicorn gateway.app:app --reload
 
 Requests still write the normal audit event, but trace export is intentionally narrower. It records service, route, model, outcome, auth method, estimated input-token count, configured token budget, latency, and trace identifiers; it does not record prompt text, model output, access reason, subject, or principal ID. A checked example is in `artifacts/sanitized-trace-evidence.jsonl`.
 
+Capacity plan evidence:
+
+```bash
+python -m gateway.capacity_plan --output artifacts/capacity-plan-evidence.json
+```
+
+The checked capacity artifact is synthetic aggregate data. It compares configured request and input-token policy limits against modeled per-model request capacity, input-token capacity, decode-token capacity, p95 latency, utilization assumptions, and cost-to-serve estimates. It is meant for review of the planning logic, not as a claim about a production fleet.
+
 Local dashboard stack:
 
 ```bash
@@ -129,6 +139,7 @@ This project covers:
 - W3C trace context propagation for request correlation across a model-serving control plane.
 - Prometheus-compatible metrics for authentication outcomes, policy denials, request/token limiting, input-token throughput, and inference latency.
 - Sanitized trace export that proves request correlation without leaking prompts, outputs, reasons, or principal identifiers.
+- Synthetic capacity and cost-to-serve projection that connects policy budgets to modeled request, token, latency, utilization, and cost assumptions.
 - Local Prometheus/Grafana review files for model-access, auth, denial, and latency telemetry.
 - Kubernetes-ready health probes, scrape annotations, and non-root runtime posture.
 - Backend service design with clear separation between API, policy, rate limiting, and inference.
@@ -139,6 +150,7 @@ This project covers:
 - Replace local HS256 review tokens with JWKS-backed OIDC key rotation.
 - Replace in-memory request and token-budget limiters with Redis-backed or gateway-level distributed controls.
 - Upgrade the local trace JSONL proof into full OpenTelemetry SDK export through an OTLP collector, then capture Grafana screenshots from synthetic traffic.
+- Replace synthetic capacity inputs with measured backend profiles once a real model-serving adapter exists.
 - Add policy-as-code examples, redaction controls, and negative authorization tests.
 - Add CI supply-chain evidence such as SBOM generation, dependency scanning, and container scanning.
 - Add SOC2/FedRAMP-inspired control mapping notes without claiming certification or production authorization.

ROADMAP.md

@@ -58,6 +58,7 @@ Status: partially implemented.
 - Keep the Prometheus `/metrics` endpoint and provisioned Grafana dashboard reviewable under local `docker compose`.
 - Keep sanitized trace JSONL export opt-in and free of prompt text, model outputs, access reason, subjects, and principal IDs.
 - Keep estimated input-token budget metrics and audit fields free of prompt text and principal-identifying trace attributes.
+- Keep the synthetic capacity plan artifact aggregate-only and tied to configured policy budgets.
 - Upgrade the trace JSONL proof to full OpenTelemetry SDK export through an OTLP collector.
 - Add Grafana dashboard screenshots using synthetic traffic.
 - Add SLO burn-rate alert examples for auth failures, policy denials, rate limiting, and inference latency.
@@ -69,6 +70,7 @@ Status: planned.
 
 - Add adapters for Triton-compatible, vLLM-compatible, or SGLang-compatible backends.
 - Keep mock backend tests so the repo remains reviewable without GPU hardware.
+- Replace synthetic capacity profiles with measured backend profiles once a real adapter exists.
 - Add GPU/DCGM telemetry correlation when real backend integration exists.
 - Extend the current estimated input-token throughput proof with queue depth, backend error rate, and real per-model latency when backend integration exists.
 

artifacts/capacity-plan-evidence.json

@@ -0,0 +1,62 @@
+{
+  "assumptions": {
+    "cost_basis": "single synthetic accelerator-hour",
+    "safety_margin": 0.2,
+    "target_gpu_utilization": 0.72
+  },
+  "data_scope": "synthetic aggregate benchmark profile; excludes request bodies, decoded text, identities, secrets, and production logs",
+  "models": [
+    {
+      "accelerator_class": "synthetic-review-gpu",
+      "cost_per_1000_requests_usd": 0.073457,
+      "cost_per_1m_input_tokens_usd": 0.278245,
+      "model_id": "mission-summarizer",
+      "modeled_decode_tokens_per_minute": 41675.294,
+      "modeled_input_tokens_per_minute": 167717.647,
+      "modeled_requests_per_minute": 635.294,
+      "observed_gpu_utilization": 0.68,
+      "observed_p95_latency_ms": 410,
+      "policy_input_token_utilization": 0.047699,
+      "policy_input_tokens_per_minute": 8000,
+      "policy_request_utilization": 0.047222,
+      "policy_requests_per_minute": 30,
+      "profile_name": "synthetic-standard-summary",
+      "status": "within_synthetic_capacity"
+    },
+    {
+      "accelerator_class": "synthetic-review-gpu",
+      "cost_per_1000_requests_usd": 0.106301,
+      "cost_per_1m_input_tokens_usd": 0.356867,
+      "model_id": "threat-triage",
+      "modeled_decode_tokens_per_minute": 32224.865,
+      "modeled_input_tokens_per_minute": 130767.568,
+      "modeled_requests_per_minute": 439.005,
+      "observed_gpu_utilization": 0.74,
+      "observed_p95_latency_ms": 530,
+      "policy_input_token_utilization": 0.045883,
+      "policy_input_tokens_per_minute": 6000,
+      "policy_request_utilization": 0.045558,
+      "policy_requests_per_minute": 20,
+      "profile_name": "synthetic-restricted-triage",
+      "status": "within_synthetic_capacity"
+    },
+    {
+      "accelerator_class": "synthetic-review-gpu",
+      "cost_per_1000_requests_usd": 0.003788,
+      "cost_per_1m_input_tokens_usd": 0.022894,
+      "model_id": "benchmark-echo",
+      "modeled_decode_tokens_per_minute": 124800.0,
+      "modeled_input_tokens_per_minute": 873600.0,
+      "modeled_requests_per_minute": 5280.0,
+      "observed_gpu_utilization": 0.36,
+      "observed_p95_latency_ms": 95,
+      "policy_input_token_utilization": 0.022894,
+      "policy_input_tokens_per_minute": 20000,
+      "policy_request_utilization": 0.022727,
+      "policy_requests_per_minute": 120,
+      "profile_name": "synthetic-health-check",
+      "status": "within_synthetic_capacity"
+    }
+  ],
+  "schema_version": 1
+}

docs/OPERATIONS.md

@@ -11,6 +11,7 @@ This service is intentionally small, but it is shaped like an AI infrastructure
 - Traceability: every inference response and audit event carries a trace ID for request correlation.
 - Trace privacy: optional trace exports omit prompt text, generated output, access reason, subject, and principal ID.
 - Budget visibility: request-count and estimated input-token budget decisions are visible in audit events and metrics.
+- Capacity review: model policy changes compare request and input-token limits against a synthetic capacity/cost plan before rollout.
 
 ## Metrics
 
@@ -29,6 +30,14 @@ Set `TRACE_EXPORT_PATH` or `OTEL_TRACE_EXPORT_PATH` to write one sanitized JSONL
 
 The checked example in `artifacts/sanitized-trace-evidence.jsonl` shows the expected shape. `tests/test_trace_exporter.py` covers the privacy boundary.
 
+## Capacity Plan Artifact
+
+Run `python -m gateway.capacity_plan --output artifacts/capacity-plan-evidence.json` to regenerate the checked synthetic capacity plan.
+
+The report compares each configured model policy with a synthetic benchmark profile and records modeled request capacity, input-token capacity, decode-token capacity, p95 latency, target utilization, safety margin, and cost-to-serve estimates. The report is aggregate-only and intentionally excludes request bodies, decoded text, identities, secrets, and production logs.
+
+Use the artifact before raising `requests_per_minute` or `input_tokens_per_minute`; if a policy would exceed modeled capacity, treat that as a rollout blocker until the benchmark profile or backend fleet shape is updated.
+
 ## Local Dashboard
 
 `docker compose up --build` starts the gateway, Prometheus, and Grafana with the dashboard provisioned from `deploy/grafana/dashboards/security-gateway.json`.
@@ -92,7 +101,15 @@ Dashboard panels cover:
 2. Compare `security_gateway_input_tokens_total` against the model's configured `input_tokens_per_minute`.
 3. Review audit events for repeated principal/model pairs, estimated token counts, and shared trace IDs.
 4. If traffic is abusive, block or throttle at the edge before raising the model budget.
-5. If traffic is legitimate, raise the budget only with a capacity note and rollback time.
+5. If traffic is legitimate, regenerate `artifacts/capacity-plan-evidence.json` and raise the budget only with a capacity note and rollback time.
+
+### Capacity Policy Change
+
+1. Regenerate `artifacts/capacity-plan-evidence.json`.
+2. Confirm the target model status is `within_synthetic_capacity`.
+3. Compare `policy_request_utilization` and `policy_input_token_utilization` against the intended rollout margin.
+4. Keep the prior policy available for rollback if the change increases either utilization materially.
+5. Record the policy diff, modeled capacity result, and rollback owner in the release note.
 
 ### Latency Regression
 
@@ -112,4 +129,4 @@ Dashboard panels cover:
 - Kubernetes readiness and liveness probes should target `/health`.
 - Prometheus should scrape `/metrics` on port 8000.
 - Optional trace export path must point at a writable location and must not be treated as a prompt or output sink.
-- Model policy changes should review both `requests_per_minute` and `input_tokens_per_minute`.
+- Model policy changes should review both `requests_per_minute` and `input_tokens_per_minute`, then regenerate the capacity plan artifact.

docs/PORTFOLIO_REVIEW.md

@@ -13,7 +13,9 @@ This project is intentionally designed as a public-safe AI security infrastructu
 - `gateway/metrics.py`: Prometheus-compatible authentication, policy, limiter, token-throughput, and latency samples.
 - `gateway/audit.py`: structured JSONL evidence for allowed and denied requests with auth and trace fields.
 - `gateway/trace_exporter.py`: sanitized trace span export for local observability evidence.
+- `gateway/capacity_plan.py`: synthetic capacity and cost-to-serve projection tied to model policies.
 - `gateway/mock_inference.py`: reviewable model-serving boundary without private models or GPU hardware.
+- `artifacts/capacity-plan-evidence.json`: checked aggregate capacity artifact for local review.
 - `deploy/grafana/dashboards/security-gateway.json`: dashboard queries for request outcomes, latency, auth, denial, and model policy review.
 - `docs/OPERATIONS.md`: SLOs, alert candidates, and incident runbooks.
 - `ROADMAP.md`: secure AI / cloud governance roadmap for policy-as-code, redaction, supply-chain evidence, telemetry, and control mapping.
@@ -30,6 +32,7 @@ This project is intentionally designed as a public-safe AI security infrastructu
 - Proving token-budget abuse control without writing prompt text into metrics or trace spans.
 - Handling request correlation through OpenTelemetry-compatible W3C trace context.
 - Exporting sanitized trace evidence that keeps prompt, output, access-reason, subject, and principal identifiers out of observability artifacts.
+- Connecting model policy limits to synthetic capacity, utilization, latency, and cost-to-serve estimates without using private workload data.
 - Providing Prometheus/Grafana review files that turn gateway metrics into operational panels.
 - Designing a mockable inference boundary that can later route to real model-serving backends.
 - Showing Kubernetes deployment thinking without requiring cloud credentials.
@@ -49,6 +52,7 @@ This project is intentionally designed as a public-safe AI security infrastructu
 - Add mTLS notes for gateway-to-backend communication.
 - Replace in-memory request/token budget controls with distributed limiters.
 - Upgrade the local trace JSONL proof into full OpenTelemetry SDK export through an OTLP collector and capture dashboard screenshots from synthetic traffic.
+- Replace synthetic capacity profiles with measured backend profiles after model-serving integration exists.
 - Add policy-as-code and redaction examples with positive and negative test cases.
 - Add CI supply-chain evidence such as SBOM generation, dependency scanning, and container scanning.
 - Add SOC2/FedRAMP-inspired control mapping notes without implying certification or production authorization.