chore(selfhost): validate Grafana dashboard JSON and Prometheus alert-rule syntax

[JSONbored]

Closes #1943

Summary

PR #2638 (this session) audited every deliverable in #1943 against the actual code and found dashboards (6), alert rules (13 groups, ~21 rules), and metrics already comprehensive — the genuine gap was 5 missing troubleshooting runbooks, which shipped there. The one deliverable left unverified was "add validation for dashboard JSON / alert rule syntax if available."

scripts/validate-observability-configs.mjs closes that gap: Grafana and Prometheus both fail OPEN on a malformed dashboard/rule file (skip it, log a warning), so nothing else catches a broken file until an operator notices a panel or alert is simply missing.

• Checks every grafana/dashboards/*.json file parses as JSON and has a non-empty title + a panels array.
• Checks prometheus/rules/alerts.yml parses as YAML and every rule has alert, expr, labels.severity, and annotations.summary.
• Wired into test:ci as selfhost:validate-observability.

Test plan

• npm run selfhost:validate-observability — passes against the real dashboards/rules
• npm run typecheck, npm run test:coverage — clean; 12 new unit tests cover both validators' happy paths and failure modes (invalid JSON/YAML, missing fields, empty directory, unreadable path)
• npm run test:ci (full local gate, unsharded) — green
• npm audit --audit-level=moderate — 0 vulnerabilities

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-03 01:53:02 UTC}

4 files · 1 AI reviewer · no blockers · readiness 82/100 · CI pending · blocked

⏸️ Suggested Action - Manual Review

• Touches a guarded path — held for manual review

Review summary
The change adds a small observability-config validator, wires it into `test:ci`, and covers the exported validators with direct unit tests plus real repository fixtures. The dashboard and rule-shape checks are correctly guarded against non-object YAML/JSON values, and the PromQL sanity check now catches the explicit dangling-operator and bracket-mismatch cases shown in the tests. The notable limitation is that this is not full PromQL validation, but the script documents that boundary and still closes the basic fail-open coverage gap.

Nits — 6 non-blocking

• nit: `scripts/validate-observability-configs.mjs:58` builds child paths with string concatenation, so using `join(dir, file)` would be more robust if this helper is ever called with a trailing slash or platform-specific path.
• nit: `test/unit/validate-observability-configs-script.test.ts:7` creates temporary files but never removes them, so repeated local runs leave `gt-dash-*` and `gt-alerts-*` directories behind.
• Use `node:path`'s `join` in `validateDashboards` for `const path = join(dir, file)` to match the test helper style and avoid path-format assumptions.
• Consider adding one explicit test that a syntactically invalid but bracket-balanced PromQL expression remains out of scope, or rename the PromQL helper/test wording to make the boundary impossible to misread later.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
• Touches a guarded path — held for manual review — A maintainer must review and merge this change.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	✅ Linked	#1943
Related work	⚠️ 2 scoped overlaps	Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope	❌ 8/20	High review scope from cached public metadata (size label size:L; 1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 61 registered-repo PR(s), 52 merged, 506 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 61 PR(s), 506 issue(s).
Gate result	⚠️ Not blocking	Advisory; not blocking this PR.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 61 PR(s), 506 issue(s).
• Related work: Items reference the same linked issue docs(ops): finish self-host rate-limit dashboards and runbooks #1943. (issue #1943)
• Related work: Open PR work references issue docs(ops): finish self-host rate-limit dashboards and runbooks #1943. (issue #1943)

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Review top overlaps.
• Add a concise scope and risk note.
• No action.
• Check active issues and PRs before submitting.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.10%. Comparing base (52f770f) to head (0da7a3e).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2659   +/-   ##
=======================================
  Coverage   96.10%   96.10%           
=======================================
  Files         237      237           
  Lines       26505    26505           
  Branches     9614     9614           
=======================================
  Hits        25472    25472           
  Misses        424      424           
  Partials      609      609           

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.