Skip to content

feat(observability): expand self-host Sentry context and monitors #3752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JSONbored merged 1 commit into from
Jul 6, 2026
Merged

feat(observability): expand self-host Sentry context and monitors #3752

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 177 additions & 3 deletions apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,177 @@ volumes:
runner-work-2:`}
/>

<h2>Enabling Sentry (your own DSN)</h2>
<p>
Sentry is <strong>opt-in and off by default</strong>. Leave <code>SENTRY_DSN</code> unset
for a complete no-op with negligible overhead — no events leave your box. When you want
error tracking, point the runtime at a project you control in your own Sentry organization.
</p>
<CodeBlock
filename=".env"
code={`# Your Sentry project DSN — never commit this to git
SENTRY_DSN=https://examplePublicKey@o0.ingest.sentry.io/0
SENTRY_ENVIRONMENT=production
SENTRY_SERVER_NAME=gittensory-us-east
SENTRY_RELEASE=gittensory-selfhost@2026.07.05
# Optional: sample review tracing spans (0.05 = 5%)
# SENTRY_TRACES_SAMPLE_RATE=0.05`}
/>
<p>
Official release images bake <code>GITTENSORY_VERSION</code> as the default release id;
override with <code>SENTRY_RELEASE</code> when you tag custom builds. Mount secrets with{" "}
<code>SENTRY_DSN_FILE</code> instead of inline env when you prefer a file-backed DSN. After
changing Sentry env, restart the <code>gittensory</code> service — there is no hot reload.
</p>
<Callout variant="note">
Community self-hosters should send events only to their own DSN. The shipped stack never
phones home to a maintainer-owned project unless you configure one.
</Callout>

<h2>Sentry context taxonomy</h2>
<p>
Self-host Sentry events carry a small, scrubbed taxonomy so operators can filter by
subsystem without opening raw payloads. Structured error logs forwarded from{" "}
<code>console.error</code> use the JSON <code>event</code> slug as the issue type; direct
captures use a <code>kind</code> or review <code>operation</code> tag instead.
</p>
<FeatureRow
items={[
{
title: "Tags (indexed, low cardinality)",
description:
"repo (also indexed when logs emit repository), pr, head_sha, kind, subsystem, jobType, operation, agent, decision_outcome, provider, model, monitor, installation_id_hash (never raw installation id), trace_id, span_id.",
},
{
title: "Contexts (full scrubbed detail)",
description:
"gittensory (engine captures), review (failed reviews), log (structured console lines), sentry_monitor (cron failures), otel (active trace ids). Secrets, tokens, bodies, diffs, prompts, and review text are redacted before send.",
},
{
title: "Subsystems",
description:
"webhook, queue, github, ai, gate, publish, scheduled, backup, relay — map to the engine paths named in issue #1824.",
},
]}
/>
<p>
Cron monitor slugs follow{" "}
<code>gittensory-selfhost-&#123;environment&#125;-&#123;loop&#125;</code> (for example{" "}
<code>gittensory-selfhost-production-scheduled-loop</code>). Pair monitor alerts with queue
depth, dead-job counts, and the matching structured log event.
</p>

<h2>Sentry alert classes and runbook</h2>
<p>
Tune Sentry alert rules for <strong>persistent failure classes</strong>, not one-off
fail-open noise. The table below lists actionable signals, what they usually mean, and the
first checks an operator should run. Prometheus/Grafana alerts in the observability profile
cover the same failure modes from a metrics angle — use both when Sentry is enabled.
</p>
<div className="overflow-x-auto">
<table className="w-full border-collapse text-token-sm">
<thead>
<tr className="border-hairline text-left text-token-xs text-muted-foreground">
<th className="py-2 pr-4 font-medium">Alert class</th>
<th className="py-2 pr-4 font-medium">Sentry signal</th>
<th className="py-2 pr-4 font-medium">Threshold guidance</th>
<th className="py-2 font-medium">First response</th>
</tr>
</thead>
<tbody className="divide-hairline">
<tr>
<td className="py-2 pr-4 align-top">Dead-letter growth</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
<code>selfhost_job_dead</code>, <code>queue_pump_crashed</code>, missed{" "}
<code>queue-dead-letter-revive</code> monitor
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Page when dead jobs stay &gt; 0 for &gt;30m or the revive monitor misses twice
</td>
<td className="py-2 align-top text-muted-foreground">
Check Grafana dead-job panel, <code>/metrics</code>{" "}
<code>gittensory_jobs_dead_total</code>, queue logs; replay from DLQ dashboard only
after fixing root cause
</td>
</tr>
<tr>
<td className="py-2 pr-4 align-top">Check-run permission gaps</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
<code>check_run_post_denied</code> grouped by repo
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Alert when the same repo hits ≥3 denials in 1h (transient GitHub blips are normal)
</td>
<td className="py-2 align-top text-muted-foreground">
Re-run activation, confirm Checks:write on the GitHub App, verify installation still
has access to the repo
</td>
</tr>
<tr>
<td className="py-2 pr-4 align-top">AI provider exhaustion</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
<code>close_breaker_engaged</code>, <code>selfhost_ai_provider</code> errors, review
failures tagged <code>operation=ai_review</code>
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Warn on first breaker; page when breaker stays engaged &gt;15m or error rate &gt;25%
of reviews in 1h
</td>
<td className="py-2 align-top text-muted-foreground">
Check provider quotas, <code>AI_*</code> env, CLI availability (
<code>INSTALL_AI_CLIS</code>), and Grafana AI usage panels
</td>
</tr>
<tr>
<td className="py-2 pr-4 align-top">Relay / broker failures</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
<code>orb_relay_drain</code>, <code>orb_relay_register</code>,{" "}
<code>orb_broker_unavailable</code>, missed relay monitors
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Page when relay register/drain monitors miss twice or broker errors persist &gt;10m
</td>
<td className="py-2 align-top text-muted-foreground">
Verify Orb enrollment secrets, outbound connectivity, and broker health; check relay
logs without exposing enrollment material
</td>
</tr>
<tr>
<td className="py-2 pr-4 align-top">Backup failures</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
<code>selfhost_backup_advisory</code>, backup container non-zero exits, stale backup
freshness metric
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Page when backup freshness &gt;2× <code>BACKUP_INTERVAL_SECONDS</code> or verify
script fails twice
</td>
<td className="py-2 align-top text-muted-foreground">
Inspect <code>docker compose logs backup</code>, disk space, and{" "}
<Link to="/docs/self-hosting-backup-scaling">backup docs</Link>; do not delete the
last good backup after a failed run
</td>
</tr>
<tr>
<td className="py-2 pr-4 align-top">Scheduled monitor misses</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Sentry monitor alert on <code>scheduled-loop</code>, <code>orb-export</code>, or
other wrapped loops
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Use Sentry&apos;s built-in monitor failure thresholds (2 consecutive misses on most
loops)
</td>
<td className="py-2 align-top text-muted-foreground">
Process may still be alive but cron work stopped — check{" "}
<code>selfhost_cron_error</code>, queue pump logs, and restart the app container if
the loop crashed without taking down the process
</td>
</tr>
</tbody>
</table>
</div>

<h2>Sentry server name</h2>
<p>
<code>SENTRY_SERVER_NAME</code> sets a clean, human name for this instance in Sentry (for
Expand Down Expand Up @@ -551,9 +722,12 @@ volumes:
]}
/>
<p>
A missed monitor means the process may still be alive but the recurring work is not checking
in on schedule. Pair the monitor with queue depth, dead-job counts, and the structured error
log for the same subsystem.
Monitor loop slugs (the <code>&#123;loop&#125;</code> segment in the slug) are{" "}
<code>scheduled-loop</code>, <code>orb-export</code>, <code>orb-relay-drain</code>,{" "}
<code>orb-relay-register</code>, and <code>queue-dead-letter-revive</code>. A missed monitor
means the process may still be alive but the recurring work is not checking in on schedule.
Pair the monitor with queue depth, dead-job counts, and the structured error log for the
same subsystem.
</p>

<h2>Re-gate sweeps (agent-regate-sweep)</h2>
Expand Down
34 changes: 12 additions & 22 deletions src/selfhost/sentry.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ type SentryNs = typeof import("@sentry/node");
type SentryClient = NonNullable<ReturnType<SentryNs["init"]>>;
type SentryMonitorConfig = NonNullable<Parameters<SentryNs["captureCheckIn"]>[1]>;
export type SentryMonitorName = "scheduled-loop" | "orb-export" | "orb-relay-drain" | "orb-relay-register" | "queue-dead-letter-revive";
export const SENTRY_MONITOR_NAMES: readonly SentryMonitorName[] = ["scheduled-loop", "orb-export", "orb-relay-drain", "orb-relay-register", "queue-dead-letter-revive"];
export const SENTRY_OPERATIONAL_SUBSYSTEMS = { webhook: "GitHub webhook ingest and enqueue", queue: "Job claim, process, dead-letter revival, and pump loops", github: "GitHub App token minting and broker calls", ai: "AI provider attempts, rate limits, and close-breaker engagement", gate: "Gate verdict and check-run publish", publish: "PR comment and public-surface publish", scheduled: "Maintenance tick, regate sweeps, and cron fan-out", backup: "Backup profile runs and freshness advisories", relay: "Orb relay register/drain and broker export loops" } as const;
export const SENTRY_OPERATIONAL_TAG_KEYS = ["repo", "repository", "owner", "installation_id_hash", "pull", "pullNumber", "pr", "head_sha", "project", "kind", "subsystem", "job_type", "jobType", "reason", "result", "deliveryId", "provider", "model", "effort", "timeoutMs", "trace_id", "span_id", "operation", "agent", "decision_outcome", "event", "monitor"] as const;
type SentryScope = {
setContext(name: string, context: Record<string, unknown>): void;
setTag(key: string, value: string): void;
Expand All @@ -34,7 +37,7 @@ let digestHexSync: DigestHex | undefined;
const SECRET_KEY =
/(token|secret|key|password|passwd|authorization|auth|dsn|cookie|bearer|credential|private)/i;
const PAYLOAD_KEY =
/(^|[_-])(body|payload|patch|diff|prompt|rubric|guardrail|headers?|cookies?|title|config|review[-_]?text|review[-_]?content)([_-]|$)|^(body|payload|patch|diff|prompt|rubric|guardrail|headers?|cookies?|title|config|review[-_]?text|review[-_]?content)$/i;
/(^|[_-])(body|payload|patch|diff|prompt|rubric|guardrail|headers?|cookies?|title|config|review[-_]?text|review[-_]?content|comment[-_]?text|comment[-_]?body)([_-]|$)|^(body|payload|patch|diff|prompt|rubric|guardrail|headers?|cookies?|title|config|review[-_]?text|review[-_]?content|comment[-_]?text|comment[-_]?body)$/i;
const SECRET_VALUE = new RegExp(
[
`${"github" + "_pat_"}[A-Za-z0-9_]+`,
Expand Down Expand Up @@ -271,6 +274,7 @@ function tagHashedInstallation(scope: SentryScope, context: Record<string, unkno
if (hash) scope.setTag("installation_id_hash", hash);
}

function applyOperationalTags(scope: SentryScope, context: Record<string, unknown>): void { const normalized: Record<string, unknown> = typeof context.repository === "string" && context.repo === undefined ? { ...context, repo: context.repository } : { ...context }; tagHashedInstallation(scope, normalized); for (const key of SENTRY_OPERATIONAL_TAG_KEYS) { const tagValue = normalized[key]; if (typeof tagValue === "string" || typeof tagValue === "number") scope.setTag(key, String(tagValue)); } }
function scrubString(value: string): string {
return value
.replace(QUERY_SECRET_VALUE, `$1${REDACTED}`)
Expand Down Expand Up @@ -435,8 +439,7 @@ export function captureError(
if (!active || !Sentry) return;
Sentry.withScope((scope) => {
setOtelTraceScope(scope);
if (context) scope.setContext("gittensory", hashedInstallationContext(context));
if (context) tagHashedInstallation(scope, context);
if (context) { const safeContext = hashedInstallationContext(context); scope.setContext("gittensory", safeContext); applyOperationalTags(scope, safeContext); }
Sentry!.captureException(
error instanceof Error ? error : new Error(String(error)),
);
Expand All @@ -456,23 +459,14 @@ export function captureReviewFailure(
if (context) {
const safeContext = hashedInstallationContext(context);
scope.setContext("review", safeContext);
tagHashedInstallation(scope, context);
for (const tag of ["owner", "repo", "pr", "head_sha", "operation", "agent", "decision_outcome"]) {
const value = safeContext[tag];
if (value !== undefined && value !== null)
scope.setTag(tag, String(value));
}
applyOperationalTags(scope, safeContext);
}
Sentry!.captureException(
error instanceof Error ? error : new Error(String(error)),
);
});
}

// The structured-log fields worth indexing as Sentry tags — the dimensions operators filter + group by. Only
// string|number values are tagged; everything else stays in the full "log" context.
const SENTRY_LOG_TAG_KEYS = ["repo", "repository", "installation_id_hash", "pull", "pullNumber", "pr", "project", "kind", "deliveryId", "provider", "model", "effort", "timeoutMs", "trace_id", "span_id", "operation", "agent", "decision_outcome"] as const;

/** A SHORT location suffix — " (repo#pr)" — for a no-message error title, so the issue list shows WHERE without
* dumping every scalar field (which made titles unreadably long, e.g. trailing a full deliveryId). The complete
* field set is still indexed as Sentry tags + kept in the "log" context. Empty when the log carries no repo. */
Expand Down Expand Up @@ -591,13 +585,8 @@ export function forwardStructuredLogToSentry(line: unknown, fromErrorSink = fals
scope.setLevel(severity);
setOtelTraceScope(scope);
scope.setContext("log", safeObj);
if (event) scope.setTag("event", event);
// Index the dimensions operators filter + group by, so issues are findable without digging into the context.
for (const key of SENTRY_LOG_TAG_KEYS) {
const tagValue = safeObj[key];
if (typeof tagValue === "string" || typeof tagValue === "number")
scope.setTag(key, String(tagValue));
}
if (event) safeObj.event = event;
applyOperationalTags(scope, safeObj);
// Group recurrences of ONE failure into a single issue (by event, not the variable detail in the value).
if (event) scope.setFingerprint(["gittensory-log", event]);
// Sentry uses event.transaction as the issue culprit fallback when the stack has no frames; point it at the
Expand Down Expand Up @@ -642,8 +631,9 @@ export async function withSentryMonitor<T>(
Sentry.withScope((scope) => {
scope.setLevel("error");
setOtelTraceScope(scope);
scope.setContext("sentry_monitor", safeMonitorContext(name, monitorSlug, context));
scope.setTag("monitor", monitorSlug);
const monitorContext = safeMonitorContext(name, monitorSlug, context);
scope.setContext("sentry_monitor", monitorContext);
applyOperationalTags(scope, { ...monitorContext, monitor: monitorSlug, kind: `sentry_monitor_${name}`, subsystem: "scheduled" });
scope.setFingerprint(["gittensory-sentry-monitor", name]);
Sentry!.captureException(error instanceof Error ? error : new Error(String(error)));
});
Expand Down
48 changes: 48 additions & 0 deletions test/unit/docs-selfhost-sentry-observability.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import { readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";

import {
SENTRY_MONITOR_NAMES,
SENTRY_OPERATIONAL_SUBSYSTEMS,
SENTRY_OPERATIONAL_TAG_KEYS,
} from "../../src/selfhost/sentry";

// Drift guard (#1824): self-host Sentry docs must stay aligned with exported taxonomy and runbook signals.

const OPERATIONS = "apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx";
const operations = readFileSync(OPERATIONS, "utf8");

describe("self-host Sentry observability docs (#1824)", () => {
it("documents enabling Sentry with an operator-owned DSN", () => {
expect(operations).toContain("Enabling Sentry (your own DSN)");
expect(operations).toContain("SENTRY_DSN");
expect(operations).toContain("opt-in and off by default");
expect(operations).toContain("SENTRY_DSN_FILE");
});

it("documents context taxonomy and cron monitor slugs", () => {
expect(operations).toContain("Sentry context taxonomy");
expect(operations).toContain("installation_id_hash");
for (const monitor of SENTRY_MONITOR_NAMES) {
expect(operations).toContain(monitor);
}
for (const subsystem of Object.keys(SENTRY_OPERATIONAL_SUBSYSTEMS)) {
expect(operations).toContain(subsystem);
}
for (const tag of ["kind", "subsystem", "jobType", "operation"]) {
expect(SENTRY_OPERATIONAL_TAG_KEYS).toContain(tag);
expect(operations).toContain(tag);
}
expect(operations).toContain("repository");
});

it("documents alert classes and runbook first-response checks", () => {
expect(operations).toContain("Sentry alert classes and runbook");
expect(operations).toContain("selfhost_job_dead");
expect(operations).toContain("check_run_post_denied");
expect(operations).toContain("close_breaker_engaged");
expect(operations).toContain("orb_broker_unavailable");
expect(operations).toContain("selfhost_backup_advisory");
expect(operations).toContain("scheduled-loop");
});
});
Loading
Loading