Skip to content

Commit 577a69e

Browse files
authored
refactor(observability): hard-cutover rename gittensory_ metric prefix to loopover_ (#5522)
* refactor(observability): hard-cutover rename gittensory_ metric prefix to loopover_ Renames all 139 emitted Prometheus/OTEL metric names across src/selfhost/metrics.ts and ~98 call sites, plus every downstream consumer: prometheus/rules/alerts.yml, all 10 Grafana dashboard JSON files (panel queries, titles, uids), the sqlite/AMS datasource provisioning and dashboard-provider config, scripts/backup-metrics.sh, scripts/smoke-selfhost.sh, two smoke-observability .mjs scripts, docker-compose.yml's backup-exporter healthcheck, the selfhost.yml CI smoke step, and three self-hosting docs pages that document these exact metric/alert names. Also fixes a pre-existing, unrelated bug surfaced while cross-checking the rename: metrics.ts registered HELP/TYPE metadata for the PR-files-fetch counter under an abbreviated name that never matched what backfill.ts actually emitted. Deliberately unchanged (real, separate, unrenamed identifier surfaces): GITTENSORY_* env vars, MCP tool names, the gittensory-api Worker/domain and Cloudflare Queue bindings, the Qdrant default collection name (renaming would orphan existing self-hosters' indexed embeddings without a migration step), gittensory_session/ oauth_state cookie names, the gittensory-native DB discriminant, GraphQL operation names, and packages/gittensory-engine's own miner-prediction metric names (deferred to the still-unplanned npm package rename). Closes #5330. * style: fix prettier formatting after metric-name rename
1 parent c81f314 commit 577a69e

101 files changed

Lines changed: 1207 additions & 1208 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/selfhost.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,6 @@ jobs:
140140
if [ "$ok" != "1" ]; then echo "::error::container did not become healthy"; docker logs gt; exit 1; fi
141141
curl -sf http://127.0.0.1:8787/health | grep -q '"status":"ok"'
142142
curl -sf http://127.0.0.1:8787/ready | grep -q '"ok":true'
143-
curl -sf http://127.0.0.1:8787/metrics | grep -q 'gittensory_uptime_seconds'
143+
curl -sf http://127.0.0.1:8787/metrics | grep -q 'loopover_uptime_seconds'
144144
docker logs gt 2>&1 | grep -q 'selfhost_migrations_applied'
145145
echo "self-host smoke test passed"

apps/gittensory-ui/src/routes/docs.self-hosting-configuration.tsx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -289,7 +289,7 @@ GITHUB_METADATA_CACHE_TTL_SECONDS=600`}
289289
above so operators can keep repeated policy reads hot without broadening stale cache risk.
290290
Live CI status, check-run, check-suite, pull/issue subresources, pull mergeability, token
291291
minting, rate-limit, and collaborator-permission endpoints are never served from this cache.
292-
Prometheus exports <code>gittensory_github_response_cache_total</code>, and the bundled
292+
Prometheus exports <code>loopover_github_response_cache_total</code>, and the bundled
293293
self-host Grafana dashboard includes the hit/miss/coalesced/error breakdown.
294294
</Callout>
295295

apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -110,12 +110,12 @@ docker compose --profile postgres --profile observability --profile backup up -d
110110
Check sync health with <code>chronyc sources</code> (or <code>ntpq -p</code> on an{" "}
111111
<code>ntpd</code> host) — every configured source should show a nonzero <code>Reach</code>{" "}
112112
value; <code>Reach: 0</code> means that source has never successfully synced. The{" "}
113-
<code>gittensory_clock_skew_seconds</code> gauge on the <strong>Clock Sync (NTP)</strong>{" "}
114-
row of the main Grafana dashboard tracks the live drift between this process and GitHub's
115-
server time, sampled from the <code>Date</code> header of the GitHub App's own
116-
installation-token mint calls — no extra network probe required. The bundled Prometheus
117-
rules alert at 60s (warning) and 120s (critical) drift, both well under the margin that
118-
actually breaks JWT auth.
113+
<code>loopover_clock_skew_seconds</code> gauge on the <strong>Clock Sync (NTP)</strong> row
114+
of the main Grafana dashboard tracks the live drift between this process and GitHub's server
115+
time, sampled from the <code>Date</code> header of the GitHub App's own installation-token
116+
mint calls — no extra network probe required. The bundled Prometheus rules alert at 60s
117+
(warning) and 120s (critical) drift, both well under the margin that actually breaks JWT
118+
auth.
119119
</p>
120120

121121
<h2>Alerting — required for a 24/7 deployment</h2>
@@ -650,7 +650,7 @@ SENTRY_RELEASE=gittensory-selfhost@2026.07.05
650650
</td>
651651
<td className="py-2 align-top text-muted-foreground">
652652
Check Grafana dead-job panel, <code>/metrics</code>{" "}
653-
<code>gittensory_jobs_dead_total</code>, queue logs; replay from DLQ dashboard only
653+
<code>loopover_jobs_dead_total</code>, queue logs; replay from DLQ dashboard only
654654
after fixing root cause
655655
</td>
656656
</tr>

apps/gittensory-ui/src/routes/docs.self-hosting-troubleshooting.tsx

Lines changed: 41 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ function SelfHostingTroubleshooting() {
3636
<CodeBlock
3737
lang="bash"
3838
code={`docker compose ps
39-
docker compose logs --tail=200 gittensory
39+
docker compose logs --tail=200 loopover
4040
curl http://localhost:8787/ready
4141
curl http://localhost:8787/metrics`}
4242
/>
@@ -122,18 +122,17 @@ rees_analyzer_config_invalid`}
122122
</p>
123123
<CodeBlock
124124
lang="bash"
125-
code={`curl http://localhost:8787/metrics | grep gittensory_queue
126-
docker compose logs gittensory | grep selfhost_job_dead`}
125+
code={`curl http://localhost:8787/metrics | grep loopover_queue
126+
docker compose logs loopover | grep selfhost_job_dead`}
127127
/>
128128

129129
<h2>GitHub rate-limit responses or admission deferrals</h2>
130130
<p>
131131
Two independent signals cover this:{" "}
132-
<code>gittensory_github_rest_rate_limit_responses_total</code> counts actual 403/429
133-
responses from GitHub, and the{" "}
134-
<code>gittensory_jobs_rate_limit_admission_deferred_total</code> /{" "}
135-
<code>gittensory_jobs_rate_limit_budget_deferred_total</code> /{" "}
136-
<code>gittensory_jobs_rate_limited_by_type_total</code> counters track jobs the queue itself
132+
<code>loopover_github_rest_rate_limit_responses_total</code> counts actual 403/429 responses
133+
from GitHub, and the <code>loopover_jobs_rate_limit_admission_deferred_total</code> /{" "}
134+
<code>loopover_jobs_rate_limit_budget_deferred_total</code> /{" "}
135+
<code>loopover_jobs_rate_limited_by_type_total</code> counters track jobs the queue itself
137136
held back <em>before</em> making a request, to avoid tripping a limit. All three job-side
138137
counters carry the same three labels — <code>kind</code> (<code>webhook</code> or{" "}
139138
<code>background</code>), <code>key_scope</code> (<code>installation</code>,{" "}
@@ -145,20 +144,20 @@ docker compose logs gittensory | grep selfhost_job_dead`}
145144
A short burst of deferrals is expected and self-resolving: the queue is deliberately trading
146145
a few seconds of delay to avoid a real 429. Treat it as a real problem only once it&apos;s
147146
<strong> sustained</strong> — which is exactly what{" "}
148-
<code>GittensoryGitHubRateLimitResponses</code> (real 403/429s observed) and{" "}
149-
<code>GittensoryQueueRateLimitDeferralsHigh</code> (a sustained deferral rate, not a blip)
150-
are tuned to alert on, rather than firing on every brief admission hold.
147+
<code>LoopoverGitHubRateLimitResponses</code> (real 403/429s observed) and{" "}
148+
<code>LoopoverQueueRateLimitDeferralsHigh</code> (a sustained deferral rate, not a blip) are
149+
tuned to alert on, rather than firing on every brief admission hold.
151150
</p>
152151
<CodeBlock
153152
lang="promql"
154153
code={`# Deferrals broken down by token pool and job type over the last 10m
155-
sum by (key_scope, job_type) (rate(gittensory_jobs_rate_limit_admission_deferred_total[10m]))
154+
sum by (key_scope, job_type) (rate(loopover_jobs_rate_limit_admission_deferred_total[10m]))
156155
157156
# Is one key_scope (e.g. a single installation token) the bottleneck?
158-
topk(5, sum by (key_scope) (rate(gittensory_jobs_rate_limit_budget_deferred_total[10m])))
157+
topk(5, sum by (key_scope) (rate(loopover_jobs_rate_limit_budget_deferred_total[10m])))
159158
160159
# Real rate-limit responses from GitHub itself (not just internal deferrals)
161-
sum(rate(gittensory_github_rest_rate_limit_responses_total[10m]))`}
160+
sum(rate(loopover_github_rest_rate_limit_responses_total[10m]))`}
162161
/>
163162
<p>
164163
If a single <code>key_scope=installation</code> pool is consistently the bottleneck, the fix
@@ -168,33 +167,33 @@ sum(rate(gittensory_github_rest_rate_limit_responses_total[10m]))`}
168167

169168
<h2>Low GitHub response-cache hit rate</h2>
170169
<p>
171-
<code>gittensory_github_response_cache_total</code> (REST) and{" "}
172-
<code>gittensory_github_graphql_cache_total</code> (GraphQL) both carry a{" "}
173-
<code>result</code> label — <code>hit</code>, <code>miss</code>, <code>set</code>,{" "}
174-
<code>coalesced</code>, <code>bypassed</code>, or <code>error</code> — and a{" "}
175-
<code>class</code> label identifying the endpoint family. A healthy cache should show most
176-
traffic as <code>hit</code> for endpoints that are read repeatedly in one review/maintenance
177-
pass (PR reads, check-run lookups); a low hit rate on those specific classes, not the
178-
overall average, is the useful signal.
170+
<code>loopover_github_response_cache_total</code> (REST) and{" "}
171+
<code>loopover_github_graphql_cache_total</code> (GraphQL) both carry a <code>result</code>{" "}
172+
label — <code>hit</code>, <code>miss</code>, <code>set</code>, <code>coalesced</code>,{" "}
173+
<code>bypassed</code>, or <code>error</code> — and a <code>class</code> label identifying
174+
the endpoint family. A healthy cache should show most traffic as <code>hit</code> for
175+
endpoints that are read repeatedly in one review/maintenance pass (PR reads, check-run
176+
lookups); a low hit rate on those specific classes, not the overall average, is the useful
177+
signal.
179178
</p>
180179
<CodeBlock
181180
lang="promql"
182181
code={`# REST hit rate by endpoint class over the last 15m
183-
sum by (class) (rate(gittensory_github_response_cache_total{result="hit"}[15m]))
182+
sum by (class) (rate(loopover_github_response_cache_total{result="hit"}[15m]))
184183
/
185-
sum by (class) (rate(gittensory_github_response_cache_total[15m]))
184+
sum by (class) (rate(loopover_github_response_cache_total[15m]))
186185
187186
# GraphQL hit rate — same shape, separate metric
188-
sum by (class) (rate(gittensory_github_graphql_cache_total{result="hit"}[15m]))
187+
sum by (class) (rate(loopover_github_graphql_cache_total{result="hit"}[15m]))
189188
/
190-
sum by (class) (rate(gittensory_github_graphql_cache_total[15m]))`}
189+
sum by (class) (rate(loopover_github_graphql_cache_total[15m]))`}
191190
/>
192191

193192
<h2>Qdrant / vector-store errors</h2>
194193
<p>
195-
<code>gittensory_qdrant_errors_total</code> carries an <code>op</code> label (
194+
<code>loopover_qdrant_errors_total</code> carries an <code>op</code> label (
196195
<code>upsert</code>, <code>query</code>, or <code>delete</code>) so you can tell whether
197-
indexing or retrieval is failing. <code>GittensoryQdrantErrorRateHigh</code> fires on a
196+
indexing or retrieval is failing. <code>LoopoverQdrantErrorRateHigh</code> fires on a
198197
sustained error ratio, not an isolated blip.
199198
</p>
200199
<ul>
@@ -226,14 +225,14 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}
226225

227226
<h2>Orb export or relay problems</h2>
228227
<p>
229-
For brokered self-host deployments, <code>gittensory_orb_events_exported_total</code> and{" "}
230-
<code>gittensory_orb_export_errors_total</code> track the hourly outcome-export loop;{" "}
231-
<code>GittensoryOrbExportErrorRateHigh</code> fires on a sustained error ratio there. The
228+
For brokered self-host deployments, <code>loopover_orb_events_exported_total</code> and{" "}
229+
<code>loopover_orb_export_errors_total</code> track the hourly outcome-export loop;{" "}
230+
<code>LoopoverOrbExportErrorRateHigh</code> fires on a sustained error ratio there. The
232231
pull-mode relay loop (for installations receiving events outbound from Orb) reports through{" "}
233-
<code>gittensory_orb_relay_drains_total</code> (<code>result=events</code> when it drained
234-
something, <code>result=empty</code> otherwise) and{" "}
235-
<code>gittensory_orb_webhook_total</code> (<code>event</code> + <code>result</code> labels)
236-
for what happened to each relayed event once enqueued locally.
232+
<code>loopover_orb_relay_drains_total</code> (<code>result=events</code> when it drained
233+
something, <code>result=empty</code> otherwise) and <code>loopover_orb_webhook_total</code>{" "}
234+
(<code>event</code> + <code>result</code> labels) for what happened to each relayed event
235+
once enqueued locally.
237236
</p>
238237
<p>
239238
If exports are failing but the relay itself looks healthy, the export loop&apos;s Sentry
@@ -257,7 +256,7 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}
257256
</li>
258257
<li>
259258
<strong>Pull mode</strong> — logged at <code>warn</code> and non-fatal: the drain loop (
260-
<code>gittensory_orb_relay_drains_total</code>) keeps retrying on its own schedule
259+
<code>loopover_orb_relay_drains_total</code>) keeps retrying on its own schedule
261260
regardless, so a transient failure here recovers on its own once the broker is reachable
262261
again. A registration failure that never clears across many retries still points at{" "}
263262
<code>ORB_ENROLLMENT_SECRET</code> being wrong, revoked, or not yet provisioned
@@ -274,9 +273,9 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}
274273
<p>
275274
Each AI provider (self-host <code>AI_PROVIDER</code> entries) has its own circuit breaker:
276275
after 3 consecutive failures it stops attempting real calls to that provider for 60 seconds,
277-
recorded as <code>gittensory_ai_provider_circuit_open_total{'{provider="..."}'}</code>{" "}
276+
recorded as <code>loopover_ai_provider_circuit_open_total{'{provider="..."}'}</code>{" "}
278277
(skipped calls) alongside{" "}
279-
<code>gittensory_ai_provider_failures_total{'{provider="..."}'}</code> (real failures). It
278+
<code>loopover_ai_provider_failures_total{'{provider="..."}'}</code> (real failures). It
280279
self-heals automatically — there is no manual reset — but it will reopen immediately if the
281280
underlying problem is still there.
282281
</p>
@@ -293,9 +292,9 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}
293292
unreachable from the container.
294293
</li>
295294
<li>
296-
<code>GittensoryAiProviderCircuitOpen</code> fires on any circuit-open event in a
297-
15-minute window — a single trip during a real but brief outage is expected; a rule that
298-
keeps firing across multiple windows points at the persistent case above.
295+
<code>LoopoverAiProviderCircuitOpen</code> fires on any circuit-open event in a 15-minute
296+
window — a single trip during a real but brief outage is expected; a rule that keeps
297+
firing across multiple windows points at the persistent case above.
299298
</li>
300299
</ul>
301300

@@ -326,7 +325,7 @@ npm run test:smoke:observability:metrics`}
326325
</li>
327326
<li>
328327
If the app's own <code>/metrics</code> check fails, that is unrelated to the OTEL
329-
collector — check the app container directly (<code>docker compose logs gittensory</code>
328+
collector — check the app container directly (<code>docker compose logs loopover</code>
330329
).
331330
</li>
332331
<li>

docker-compose.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@
3838
# (chrony/ntpd), not just one. A single dead NTP source silently drifts the clock with no local
3939
# symptom until GitHub App JWT auth starts failing ("Bad credentials") once the drift exceeds the
4040
# JWT's 60s skew tolerance. See docs/self-hosting/operations → "Host clock sync (NTP)" and the
41-
# gittensory_clock_skew_seconds Grafana panel/alert (--profile observability) for live drift.
41+
# loopover_clock_skew_seconds Grafana panel/alert (--profile observability) for live drift.
4242

4343
# Bounded container logging (#audit-rate-headroom): every service below defaults to Docker's
4444
# json-file driver, which has NO size cap on its own -- a long-running 24/7 stack can fill the
@@ -1160,7 +1160,7 @@ services:
11601160
- -c
11611161
- "apk add --no-cache busybox-extras && sh /scripts/backup-metrics.sh"
11621162
healthcheck:
1163-
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9101/metrics | grep -q '^gittensory_backup_latest_timestamp_seconds'"]
1163+
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9101/metrics | grep -q '^loopover_backup_latest_timestamp_seconds'"]
11641164
interval: 30s
11651165
retries: 5
11661166

0 commit comments

Comments
 (0)