Skip to content

Add distributed limiter readiness proof #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WaffleBits merged 1 commit into from
Jul 6, 2026
Merged

Add distributed limiter readiness proof #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ flowchart LR
API --> Policy["Policy Engine"]
API --> Limiter["Request Limiter"]
API --> TokenBudget["Token Budget Limiter"]
API --> DistLimit["Distributed Limiter Readiness"]
API --> Model["Mock Inference Backend"]
API --> Audit["JSONL Audit Sink"]
API --> TraceExport["Sanitized Trace Export"]
Expand All @@ -26,6 +27,7 @@ flowchart LR
- `gateway/policy.py`: role and reason-for-access decisions.
- `gateway/rate_limit.py`: in-memory fixed-window request and token-budget limiters.
- `gateway/token_budget.py`: deterministic estimated input-token accounting.
- `gateway/distributed_limiter.py`: Redis/Envoy migration readiness evidence for external request and token-budget controls.
- `gateway/audit.py`: structured JSONL audit events with identity and trace evidence.
- `gateway/trace_exporter.py`: opt-in sanitized trace spans and OTLP/HTTP collector payload generation.
- `gateway/otlp_export.py`: CLI for converting sanitized trace JSONL into checked collector-ready payloads.
Expand All @@ -40,7 +42,7 @@ flowchart LR
- Replace local HS256 review tokens with JWKS-backed OIDC key rotation.
- Add mTLS between gateway and model backends.
- Move policy definitions to OPA, Cedar, or a signed config bundle.
- Replace in-memory request and token-budget limiting with Redis or Envoy global rate limits.
- Wire the checked distributed-limiter readiness plan into Redis or Envoy global rate limits.
- Keep sanitized trace JSONL and OTLP/HTTP collector export aligned while using Prometheus metrics for low-cardinality service health.
- Capture GPU telemetry from DCGM and attach it to inference metrics.
- Replace synthetic capacity assumptions with measured backend profiles after real model-serving integration exists.
Expand Down
15 changes: 13 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Secure GPU Inference Gateway

Security-focused AI infrastructure demo for OIDC/JWT-authenticated model access, role-based authorization, per-model request and token-budget limits, trace-aware audit logs, OTLP collector-ready trace export, synthetic workload-readiness replay, synthetic capacity/cost planning, and policy-driven inference routing.
Security-focused AI infrastructure demo for OIDC/JWT-authenticated model access, role-based authorization, per-model request and token-budget limits, distributed-limiter readiness evidence, trace-aware audit logs, OTLP collector-ready trace export, synthetic workload-readiness replay, synthetic capacity/cost planning, and policy-driven inference routing.

This repository uses a mock inference backend so the security and infrastructure logic can be reviewed without GPU hardware, model weights, proprietary data, or cloud credentials.

Expand All @@ -17,6 +17,7 @@ This is the flagship portfolio project for the platform-security-to-AI-infrastru
- Demo principals and role-based model policies for local review.
- Reason-for-access enforcement for sensitive models.
- Fixed-window request and estimated input-token limiting by principal and model.
- Distributed-limiter readiness artifact that maps per-model request and token budgets to Redis/Envoy-style global controls.
- Structured JSONL audit logging with authentication and trace context evidence.
- W3C `traceparent` propagation for OpenTelemetry-compatible request correlation.
- Prometheus-compatible `/metrics` endpoint for authentication, policy, limiter, token-throughput, and latency telemetry.
Expand Down Expand Up @@ -48,6 +49,7 @@ Relevant areas:
- Review `gateway/identity.py` for bearer JWT verification and demo-principal fallback controls.
- Review `gateway/policy.py` for role and reason-for-access decisions.
- Review `gateway/rate_limit.py` and `gateway/token_budget.py` for request-count and token-budget limiter behavior.
- Review `gateway/distributed_limiter.py` and `artifacts/distributed-limiter-evidence.json` for Redis/Envoy migration readiness evidence.
- Review `gateway/metrics.py` and `/metrics` for Prometheus-compatible operational telemetry.
- Review `gateway/audit.py` for structured evidence.
- Review `gateway/trace_exporter.py` for sanitized trace span export without prompt, output, access-reason, or principal identifiers.
Expand Down Expand Up @@ -115,6 +117,14 @@ python -m gateway.workload_replay --output artifacts/workload-readiness-evidence

The checked workload artifact replays synthetic aggregate traffic through the same policy, request-limit, and token-budget logic used by the gateway. It records outcome coverage, latency gates, and per-model pressure summaries while omitting request bodies, decoded text, identities, secrets, access reasons, and production logs.

Distributed limiter readiness evidence:

```bash
python -m gateway.distributed_limiter --output artifacts/distributed-limiter-evidence.json
```

The checked distributed-limiter artifact maps every configured model policy to request-count and estimated-input-token rules for Redis fixed-window and Envoy global-rate-limit style backends. It records atomic script shape, descriptor shape, rule coverage, sample allow/deny decisions, and release gates without request bodies, decoded text, subject identifiers, secrets, access reasons, or production logs.

Local dashboard stack:

```bash
Expand Down Expand Up @@ -160,6 +170,7 @@ This project covers:
- Prometheus-compatible metrics for authentication outcomes, policy denials, request/token limiting, input-token throughput, and inference latency.
- Sanitized trace export that proves request correlation without leaking prompts, outputs, reasons, or principal identifiers.
- OTLP/HTTP collector export proof that converts sanitized span records into collector-ready trace payloads and can post them to a local collector.
- Distributed-limiter readiness evidence that connects configured request/token budgets to Redis atomic-window and Envoy descriptor shapes before a live external limiter is introduced.
- Synthetic workload-readiness replay that proves guardrail coverage and latency gates for allowed, policy-denied, rate-limited, and token-budget-limited paths without persisting sensitive request data.
- Synthetic capacity and cost-to-serve projection that connects policy budgets to modeled request, token, latency, utilization, and cost assumptions.
- Local Prometheus/Grafana review files for model-access, auth, denial, and latency telemetry.
Expand All @@ -170,7 +181,7 @@ This project covers:
## Gaps Worth Closing Next

- Replace local HS256 review tokens with JWKS-backed OIDC key rotation.
- Replace in-memory request and token-budget limiters with Redis-backed or gateway-level distributed controls.
- Wire the distributed-limiter readiness plan into a live Redis or Envoy global-rate-limit integration.
- Capture Grafana and collector screenshots from synthetic traffic after local docker review.
- Replace synthetic capacity inputs with measured backend profiles once a real model-serving adapter exists.
- Extend workload-readiness replay with backend error-rate, queue-depth, and resilience probes once a real serving adapter exists.
Expand Down
2 changes: 2 additions & 0 deletions ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,11 @@ Status: partially implemented.
- Keep sanitized trace JSONL export opt-in and free of prompt text, model outputs, access reason, subjects, and principal IDs.
- Keep OTLP/HTTP collector payload generation tied to the same sanitized span attributes.
- Keep estimated input-token budget metrics and audit fields free of prompt text and principal-identifying trace attributes.
- Keep distributed-limiter readiness evidence tied to configured request and token budgets before introducing Redis or Envoy global rate-limit state.
- Keep the synthetic capacity plan artifact aggregate-only and tied to configured policy budgets.
- Keep the workload-readiness replay aggregate-only and tied to allowed, policy-denied, rate-limited, and token-budget-limited paths.
- Add Grafana and collector screenshots using synthetic traffic.
- Wire the distributed-limiter readiness artifact into a live Redis or Envoy integration after the rule and key-shape review stays green.
- Add SLO burn-rate alert examples for auth failures, policy denials, rate limiting, and inference latency.
- Add incident exercises for token abuse, model access denial spikes, audit sink failure, and backend saturation.

Expand Down
301 changes: 301 additions & 0 deletions artifacts/distributed-limiter-evidence.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,301 @@
{
"data_scope": "synthetic limiter migration evidence; excludes request bodies, decoded text, subject identifiers, secrets, access reasons, and production logs",
"generated_by": "gateway.distributed_limiter",
"observability": {
"existing_metrics": [
"security_gateway_requests_total",
"security_gateway_denials_total",
"security_gateway_input_tokens_total"
],
"recommended_low_cardinality_labels": [
"model_id",
"budget_type",
"outcome"
]
},
"readiness_status": "pass",
"redis_atomic_script": {
"properties": [
"single-key atomic increment",
"ttl applied on first window increment",
"over-limit result returned without payload capture"
],
"sha256": "17c63bc82db51970aef6c8871e36c95103841cefb55863d175b3f39a440ef9f0"
},
"release_gates": [
{
"expected_rules": 6,
"name": "policy_budget_rule_coverage",
"observed_rules": 6,
"status": "pass"
},
{
"name": "request_and_token_budget_coverage",
"observed_budget_types": [
"estimated_input_tokens",
"request_count"
],
"status": "pass"
},
{
"name": "redis_atomic_window_shape",
"script_sha256": "17c63bc82db51970aef6c8871e36c95103841cefb55863d175b3f39a440ef9f0",
"status": "pass"
},
{
"name": "public_safe_artifact",
"omitted_categories": [
"request payloads",
"model responses",
"subject identifiers",
"access justification text",
"auth material",
"production logs"
],
"status": "pass"
}
],
"rollout_recommendation": "ready_for_redis_or_envoy_integration_spike",
"rule_count": 6,
"rules": [
{
"budget_type": "request_count",
"denial_outcome": "rate_limited",
"envoy_descriptor": {
"entries": [
{
"key": "service",
"value": "secure-gpu-inference-gateway"
},
{
"key": "model_id",
"value": "benchmark-echo"
},
{
"key": "budget_type",
"value": "request_count"
}
],
"limit_per_unit": 120,
"unit": "minute"
},
"existing_metric": "security_gateway_requests_total",
"limit": 120,
"model_id": "benchmark-echo",
"redis_key_template": "sgig:{environment}:{model_id}:{budget_type}:{caller_hash}:{window_epoch_minute}",
"window_seconds": 60
},
{
"budget_type": "estimated_input_tokens",
"denial_outcome": "token_budget_limited",
"envoy_descriptor": {
"entries": [
{
"key": "service",
"value": "secure-gpu-inference-gateway"
},
{
"key": "model_id",
"value": "benchmark-echo"
},
{
"key": "budget_type",
"value": "estimated_input_tokens"
}
],
"limit_per_unit": 20000,
"unit": "minute"
},
"existing_metric": "security_gateway_input_tokens_total",
"limit": 20000,
"model_id": "benchmark-echo",
"redis_key_template": "sgig:{environment}:{model_id}:{budget_type}:{caller_hash}:{window_epoch_minute}",
"window_seconds": 60
},
{
"budget_type": "request_count",
"denial_outcome": "rate_limited",
"envoy_descriptor": {
"entries": [
{
"key": "service",
"value": "secure-gpu-inference-gateway"
},
{
"key": "model_id",
"value": "mission-summarizer"
},
{
"key": "budget_type",
"value": "request_count"
}
],
"limit_per_unit": 30,
"unit": "minute"
},
"existing_metric": "security_gateway_requests_total",
"limit": 30,
"model_id": "mission-summarizer",
"redis_key_template": "sgig:{environment}:{model_id}:{budget_type}:{caller_hash}:{window_epoch_minute}",
"window_seconds": 60
},
{
"budget_type": "estimated_input_tokens",
"denial_outcome": "token_budget_limited",
"envoy_descriptor": {
"entries": [
{
"key": "service",
"value": "secure-gpu-inference-gateway"
},
{
"key": "model_id",
"value": "mission-summarizer"
},
{
"key": "budget_type",
"value": "estimated_input_tokens"
}
],
"limit_per_unit": 8000,
"unit": "minute"
},
"existing_metric": "security_gateway_input_tokens_total",
"limit": 8000,
"model_id": "mission-summarizer",
"redis_key_template": "sgig:{environment}:{model_id}:{budget_type}:{caller_hash}:{window_epoch_minute}",
"window_seconds": 60
},
{
"budget_type": "request_count",
"denial_outcome": "rate_limited",
"envoy_descriptor": {
"entries": [
{
"key": "service",
"value": "secure-gpu-inference-gateway"
},
{
"key": "model_id",
"value": "threat-triage"
},
{
"key": "budget_type",
"value": "request_count"
}
],
"limit_per_unit": 20,
"unit": "minute"
},
"existing_metric": "security_gateway_requests_total",
"limit": 20,
"model_id": "threat-triage",
"redis_key_template": "sgig:{environment}:{model_id}:{budget_type}:{caller_hash}:{window_epoch_minute}",
"window_seconds": 60
},
{
"budget_type": "estimated_input_tokens",
"denial_outcome": "token_budget_limited",
"envoy_descriptor": {
"entries": [
{
"key": "service",
"value": "secure-gpu-inference-gateway"
},
{
"key": "model_id",
"value": "threat-triage"
},
{
"key": "budget_type",
"value": "estimated_input_tokens"
}
],
"limit_per_unit": 6000,
"unit": "minute"
},
"existing_metric": "security_gateway_input_tokens_total",
"limit": 6000,
"model_id": "threat-triage",
"redis_key_template": "sgig:{environment}:{model_id}:{budget_type}:{caller_hash}:{window_epoch_minute}",
"window_seconds": 60
}
],
"sample_decisions": [
{
"allowed": true,
"budget_type": "request_count",
"caller_hash_example": "1b7c3b0124a15e1b",
"limit": 120,
"model_id": "benchmark-echo",
"window_cost": 1
},
{
"allowed": false,
"budget_type": "request_count",
"caller_hash_example": "f96f9822b6789bc8",
"denial_outcome": "rate_limited",
"limit": 120,
"model_id": "benchmark-echo",
"window_cost": 121
},
{
"allowed": true,
"budget_type": "estimated_input_tokens",
"caller_hash_example": "b07357577bb92f77",
"limit": 20000,
"model_id": "benchmark-echo",
"window_cost": 64
},
{
"allowed": false,
"budget_type": "estimated_input_tokens",
"caller_hash_example": "18d87739380ab485",
"denial_outcome": "token_budget_limited",
"limit": 20000,
"model_id": "benchmark-echo",
"window_cost": 20001
},
{
"allowed": true,
"budget_type": "request_count",
"caller_hash_example": "aadd189d466799a1",
"limit": 30,
"model_id": "mission-summarizer",
"window_cost": 1
},
{
"allowed": false,
"budget_type": "request_count",
"caller_hash_example": "bc039d19142826be",
"denial_outcome": "rate_limited",
"limit": 30,
"model_id": "mission-summarizer",
"window_cost": 31
},
{
"allowed": true,
"budget_type": "estimated_input_tokens",
"caller_hash_example": "43bc083f9e77eb74",
"limit": 8000,
"model_id": "mission-summarizer",
"window_cost": 64
},
{
"allowed": false,
"budget_type": "estimated_input_tokens",
"caller_hash_example": "059bedc453cc0bc8",
"denial_outcome": "token_budget_limited",
"limit": 8000,
"model_id": "mission-summarizer",
"window_cost": 8001
}
],
"scenario_name": "distributed_request_token_limiter_readiness",
"schema_version": 1,
"target_backends": [
"redis_fixed_window_lua",
"envoy_global_rate_limit_descriptors"
],
"window_seconds": 60
}
Loading
Loading