Skip to content

Commit a43c918

Browse files
JSONboredJSONbored
authored
feat(selfhost): GPU + AI-provider observability (exporter, Prometheus, dashboard) (#4536)
- gpu-exporter service (utkuozdemir/nvidia_gpu_exporter:1.3.2), own --profile gpu, matching the ollama/postgres-exporter style - gpu Prometheus scrape job, absent/down gracefully on non-GPU installs - New gittensory_ai_provider_request_duration_seconds histogram and gittensory_ai_provider_request_errors_total counter, labeled by provider and request_kind, instrumented around the existing withReviewSpan call site in runProviderWithOtel — the error counter mirrors gittensory_ai_provider_failures_total's exemption for expected embedding-routing fallbacks - grafana/dashboards/gpu-metrics.json: GPU Summary (utilization/VRAM/ temp/power stats), AI Request Performance (rate/p50-p95-p99 latency/ error rate), and GPU Trends rows - Resource-footprint table entry in the self-hosting operations docs This is the baseline the rest of the GPU-optimization work (embedding cache, advisory routing, vision-verify) needs to measure real impact against, rather than one-off manual benchmarks.
1 parent 3e9fc32 commit a43c918

7 files changed

Lines changed: 295 additions & 2 deletions

File tree

‎apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,21 @@ DISCORD_REPO_WEBHOOKS={"owner/repoA":"https://discord.com/api/webhooks/...","own
307307
the file.
308308
</td>
309309
</tr>
310+
<tr>
311+
<td className="py-2 pr-4 align-top">
312+
+ <code>--profile gpu</code>
313+
</td>
314+
<td className="py-2 pr-4 align-top text-muted-foreground">Near-zero</td>
315+
<td className="py-2 pr-4 align-top text-muted-foreground">
316+
Tens of MiB — a single Go binary shelling out to <code>nvidia-smi</code>
317+
</td>
318+
<td className="py-2 align-top text-muted-foreground">
319+
Estimated. Adds <code>gpu-exporter</code> (nvidia_gpu_exporter) feeding the{" "}
320+
<code>gpu</code> Prometheus job and the GPU metrics Grafana dashboard — requires the
321+
NVIDIA Container Toolkit on the host; a device reservation only takes effect once
322+
this profile is activated, so a non-GPU host is unaffected either way.
323+
</td>
324+
</tr>
310325
<tr>
311326
<td className="py-2 pr-4 align-top">
312327
+ <code>--profile backup</code>

‎docker-compose.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
# --profile pgbouncer PgBouncer connection pooler in front of Postgres
1616
# --profile qdrant Qdrant vector database for RAG
1717
# --profile ollama Local Ollama AI backend
18+
# --profile gpu GPU metrics exporter (nvidia-smi -> Prometheus -> Grafana), requires the NVIDIA
19+
# Container Toolkit; combine with --profile ollama on a GPU host
1820
# --profile visual-review Headless Chromium (browserless) for before/after PR screenshot capture
1921
# --profile rees Review-enrichment service (REES) for heavier PR analysis, in-network only
2022
# --profile litestream Continuous SQLite backup to S3/B2/R2 via Litestream
@@ -369,6 +371,30 @@ services:
369371
limits:
370372
memory: "${OLLAMA_MEM_LIMIT:-20g}"
371373

374+
# ── GPU metrics exporter (--profile gpu, #4367) ─────────────────────────────
375+
# Shells to `nvidia-smi` via the NVIDIA Container Toolkit -- DCGM is built for datacenter fleets and is
376+
# the wrong fit for a single consumer/workstation-class card like the one this deployment targets. The
377+
# device reservation below only takes effect once this profile is actually activated, so a host with no
378+
# GPU (and no --profile gpu) never evaluates it.
379+
gpu-exporter:
380+
image: utkuozdemir/nvidia_gpu_exporter:1.3.2
381+
restart: unless-stopped
382+
<<: *default-logging
383+
profiles: ["gpu"]
384+
environment:
385+
# The exporter only shells out to nvidia-smi -- it never runs CUDA workloads itself, so the
386+
# lightest capability tier is sufficient (matches the upstream project's own recommended config).
387+
NVIDIA_DRIVER_CAPABILITIES: utility
388+
deploy:
389+
resources:
390+
reservations:
391+
devices:
392+
- driver: nvidia
393+
count: all
394+
capabilities: [gpu]
395+
expose:
396+
- "9835"
397+
372398
# ── Browserless / visual review (--profile visual-review) ──────────────────
373399
# Headless Chromium for automated before/after PR screenshot capture (src/review/visual/**).
374400
# Set BOTH of these in .env (browserless always requires a token; there is no way around setting

‎grafana/dashboards/gpu-metrics.json‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
{
2+
"__inputs": [],
3+
"__requires": [
4+
{ "type": "grafana", "id": "grafana", "name": "Grafana", "version": "10.0.0" },
5+
{ "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" }
6+
],
7+
"annotations": { "list": [] },
8+
"editable": false,
9+
"fiscalYearStartMonth": 0,
10+
"graphTooltip": 1,
11+
"id": null,
12+
"links": [],
13+
"panels": [
14+
{ "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, "id": 1, "title": "GPU Summary", "type": "row" },
15+
{
16+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
17+
"fieldConfig": {
18+
"defaults": {
19+
"color": { "mode": "thresholds" },
20+
"mappings": [],
21+
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] },
22+
"unit": "percentunit"
23+
}
24+
},
25+
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 1 },
26+
"id": 2,
27+
"options": { "colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
28+
"title": "GPU Utilization",
29+
"type": "stat",
30+
"targets": [{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_utilization_gpu_ratio", "legendFormat": "util" }]
31+
},
32+
{
33+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
34+
"fieldConfig": {
35+
"defaults": {
36+
"color": { "mode": "thresholds" },
37+
"mappings": [],
38+
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
39+
"unit": "bytes"
40+
}
41+
},
42+
"gridPos": { "h": 4, "w": 6, "x": 6, "y": 1 },
43+
"id": 3,
44+
"options": { "colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
45+
"title": "VRAM Used",
46+
"description": "See the VRAM Used vs Total trend panel below for headroom against the card's total capacity.",
47+
"type": "stat",
48+
"targets": [{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_memory_used_bytes", "legendFormat": "used" }]
49+
},
50+
{
51+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
52+
"fieldConfig": {
53+
"defaults": {
54+
"color": { "mode": "thresholds" },
55+
"mappings": [],
56+
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 75 }, { "color": "red", "value": 85 }] },
57+
"unit": "celsius"
58+
}
59+
},
60+
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 1 },
61+
"id": 4,
62+
"options": { "colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
63+
"title": "GPU Temperature",
64+
"type": "stat",
65+
"targets": [{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_temperature_gpu", "legendFormat": "temp" }]
66+
},
67+
{
68+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
69+
"fieldConfig": {
70+
"defaults": {
71+
"color": { "mode": "thresholds" },
72+
"mappings": [],
73+
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] },
74+
"unit": "watt"
75+
}
76+
},
77+
"gridPos": { "h": 4, "w": 6, "x": 18, "y": 1 },
78+
"id": 5,
79+
"options": { "colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
80+
"title": "Power Draw",
81+
"type": "stat",
82+
"targets": [{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_power_draw_watts", "legendFormat": "power" }]
83+
},
84+
85+
{ "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, "id": 10, "title": "AI Request Performance", "type": "row" },
86+
{
87+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
88+
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 }, "unit": "reqps" } },
89+
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 6 },
90+
"id": 11,
91+
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
92+
"title": "AI Request Rate",
93+
"description": "Request rate by provider and request kind (embedding vs review), derived from the new request-duration histogram's observation count.",
94+
"type": "timeseries",
95+
"targets": [
96+
{
97+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
98+
"expr": "sum by (provider, request_kind) (rate(gittensory_ai_provider_request_duration_seconds_count[5m]))",
99+
"legendFormat": "{{provider}} / {{request_kind}}",
100+
"refId": "A"
101+
}
102+
]
103+
},
104+
{
105+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
106+
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 }, "unit": "s" } },
107+
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 6 },
108+
"id": 12,
109+
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
110+
"title": "AI Request Latency (p50 / p95 / p99)",
111+
"type": "timeseries",
112+
"targets": [
113+
{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "histogram_quantile(0.50, sum by (le) (rate(gittensory_ai_provider_request_duration_seconds_bucket[5m])))", "legendFormat": "p50", "refId": "A" },
114+
{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "histogram_quantile(0.95, sum by (le) (rate(gittensory_ai_provider_request_duration_seconds_bucket[5m])))", "legendFormat": "p95", "refId": "B" },
115+
{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "histogram_quantile(0.99, sum by (le) (rate(gittensory_ai_provider_request_duration_seconds_bucket[5m])))", "legendFormat": "p99", "refId": "C" }
116+
]
117+
},
118+
{
119+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
120+
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 }, "unit": "reqps" } },
121+
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 6 },
122+
"id": 13,
123+
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
124+
"title": "AI Request Error Rate",
125+
"description": "Excludes expected embedding-routing fallbacks (those never increment this counter -- see gittensory_ai_provider_request_errors_total's HELP text).",
126+
"type": "timeseries",
127+
"targets": [
128+
{
129+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
130+
"expr": "sum by (provider, request_kind) (rate(gittensory_ai_provider_request_errors_total[5m]))",
131+
"legendFormat": "{{provider}} / {{request_kind}}",
132+
"refId": "A"
133+
}
134+
]
135+
},
136+
137+
{ "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 }, "id": 20, "title": "GPU Trends", "type": "row" },
138+
{
139+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
140+
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 }, "unit": "percentunit", "max": 1, "min": 0 } },
141+
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 15 },
142+
"id": 21,
143+
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
144+
"title": "GPU Utilization Over Time",
145+
"type": "timeseries",
146+
"targets": [{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_utilization_gpu_ratio", "legendFormat": "util", "refId": "A" }]
147+
},
148+
{
149+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
150+
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 }, "unit": "bytes" } },
151+
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 15 },
152+
"id": 22,
153+
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
154+
"title": "VRAM Used vs Total",
155+
"type": "timeseries",
156+
"targets": [
157+
{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_memory_used_bytes", "legendFormat": "used", "refId": "A" },
158+
{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_memory_total_bytes", "legendFormat": "total", "refId": "B" }
159+
]
160+
},
161+
{
162+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
163+
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 }, "unit": "celsius" } },
164+
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 23 },
165+
"id": 23,
166+
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
167+
"title": "GPU Temperature Over Time",
168+
"type": "timeseries",
169+
"targets": [{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_temperature_gpu", "legendFormat": "temp", "refId": "A" }]
170+
},
171+
{
172+
"datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
173+
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 2, "fillOpacity": 10 }, "unit": "watt" } },
174+
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 23 },
175+
"id": 24,
176+
"options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } },
177+
"title": "Power Draw Over Time",
178+
"type": "timeseries",
179+
"targets": [{ "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, "expr": "nvidia_smi_power_draw_watts", "legendFormat": "power", "refId": "A" }]
180+
}
181+
],
182+
"refresh": "30s",
183+
"schemaVersion": 38,
184+
"tags": ["gittensory", "gpu"],
185+
"templating": {
186+
"list": [
187+
{ "current": {}, "hide": 0, "includeAll": false, "name": "DS_PROMETHEUS", "options": [], "query": "prometheus", "refresh": 1, "type": "datasource" }
188+
]
189+
},
190+
"time": { "from": "now-6h", "to": "now" },
191+
"timepicker": {},
192+
"timezone": "browser",
193+
"title": "Gittensory — GPU Metrics",
194+
"uid": "gittensory-gpu",
195+
"version": 1
196+
}

‎prometheus/prometheus.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,13 @@ scrape_configs:
3030
- targets: ["postgres-exporter:9187"]
3131
scrape_interval: 30s
3232

33+
# GPU utilization/VRAM/temp/power from nvidia_gpu_exporter (#4367). The exporter starts only with
34+
# --profile gpu; on a non-GPU install this target just shows as absent/down, no alert fires on it.
35+
- job_name: gpu
36+
static_configs:
37+
- targets: ["gpu-exporter:9835"]
38+
scrape_interval: 15s
39+
3340
# Backup freshness from the read-only backup-exporter sidecar. The metrics exist only when the backup
3441
# profile is active, which keeps backup alerts opt-in with the backup feature.
3542
- job_name: gittensory-backup

‎src/selfhost/ai.ts‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
import type { AiContentBlock, CombineStrategy, OnMerge } from "../services/ai-review";
1010
import { isConfiguredSelfHostProvider, resolveConfiguredProviderNames } from "./ai-config";
1111
export { assertNoLegacySharedAiEnv } from "./ai-config";
12-
import { incr } from "./metrics";
12+
import { incr, observe } from "./metrics";
1313
import { withReviewSpan } from "./tracing";
1414
import { delimiter } from "node:path";
1515

@@ -1096,12 +1096,18 @@ async function runProviderWithOtel(
10961096
`circuit_open: provider "${provider.name}" is in cooldown after ${AI_PROVIDER_FAILURE_THRESHOLD} consecutive failures — skipping this attempt`,
10971097
);
10981098
}
1099+
const requestKindLabel = requestKind(options);
1100+
const startedAtMs = Date.now();
10991101
try {
11001102
const result = await withReviewSpan(
11011103
"selfhost.ai.provider",
1102-
{ "ai.provider": provider.name, "ai.model": model || "default", "ai.request_kind": requestKind(options) },
1104+
{ "ai.provider": provider.name, "ai.model": model || "default", "ai.request_kind": requestKindLabel },
11031105
() => provider.ai.run(model, options),
11041106
);
1107+
observe("gittensory_ai_provider_request_duration_seconds", (Date.now() - startedAtMs) / 1000, {
1108+
provider: provider.name,
1109+
request_kind: requestKindLabel,
1110+
});
11051111
aiProviderCircuits.delete(provider.name);
11061112
if (result.usage) {
11071113
return {
@@ -1115,8 +1121,13 @@ async function runProviderWithOtel(
11151121
}
11161122
return result;
11171123
} catch (error) {
1124+
observe("gittensory_ai_provider_request_duration_seconds", (Date.now() - startedAtMs) / 1000, {
1125+
provider: provider.name,
1126+
request_kind: requestKindLabel,
1127+
});
11181128
if (isExpectedEmbeddingRoutingError(options, error)) throw error;
11191129
incr("gittensory_ai_provider_failures_total", { provider: provider.name });
1130+
incr("gittensory_ai_provider_request_errors_total", { provider: provider.name, request_kind: requestKindLabel });
11201131
// Re-read the map here rather than reusing the `circuit` captured above: that read happened BEFORE the
11211132
// `await` on the real provider call, so under concurrent same-provider calls it can be stale by the time
11221133
// this catch runs, and computing `failures` from it would clobber a sibling call's write (lost-update race)

‎src/selfhost/metrics.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,8 @@ const DEFAULT_METRIC_META: readonly (readonly [string, MetricMeta])[] = [
111111
["gittensory_ai_total_tokens_total", { help: "AI provider total tokens observed.", type: "counter" }],
112112
["gittensory_ai_provider_circuit_open_total", { help: "AI provider circuit-open events.", type: "counter" }],
113113
["gittensory_ai_provider_failures_total", { help: "AI provider failures by provider.", type: "counter" }],
114+
["gittensory_ai_provider_request_duration_seconds", { help: "AI provider request duration in seconds, by provider and request kind.", type: "histogram" }],
115+
["gittensory_ai_provider_request_errors_total", { help: "AI provider request errors, by provider and request kind (excludes expected embedding-routing fallbacks).", type: "counter" }],
114116
["gittensory_ai_review_cache_hit_total", { help: "AI review cache hits.", type: "counter" }],
115117
["gittensory_ai_review_cache_miss_total", { help: "AI review cache misses.", type: "counter" }],
116118
["gittensory_ai_review_cache_write_error_total", { help: "AI review cache write errors.", type: "counter" }],

0 commit comments

Comments
 (0)