Skip to content

fix(observability): close two zero-trace webhook/relay loss points#3838

Merged
JSONbored merged 1 commit into
mainfrom
fix/zero-trace-webhook-relay-loss
Jul 6, 2026
Merged

fix(observability): close two zero-trace webhook/relay loss points#3838
JSONbored merged 1 commit into
mainfrom
fix/zero-trace-webhook-relay-loss

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Part of #3812 — the two "adjacent zero-trace loss points on the same ingestion path" fixes it calls out. The new fast open-PR reconciliation job (the third deliverable) is a larger, separate change tracked in a follow-up PR so this one stays reviewable.

  • src/github/webhook.ts: enqueueWebhookByEnv attempted JSON.parse(rawBody) before writing anything to webhook_events. An unparseable delivery returned "invalid_json" with zero durable trace anywhere — indistinguishable from GitHub never having sent it at all. Hoists the raw-body hash (independent of parse validity) above the parse attempt and writes a webhook_events row with status: "error" on a parse failure, so every delivery — parseable or not — leaves a trace an operator can find.
  • src/orb/broker-client.ts: drainOrbRelay's pull-mode batch parser silently dropped any relayed event missing/mistyping one of its three required fields (deliveryId/eventName/rawBody), with no log and no counter. Adds a structured orb_relay_malformed_event_dropped error log (Sentry-visible) and a gittensory_orb_relay_malformed_events_total counter on the drop, naming which of the three fields was missing.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session/CORS surface touched; this only adds a trace row and a log/counter on already-existing failure paths.)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — no API/OpenAPI/MCP surface changed.)
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — backend-only change, no UI touched.)
  • Visible UI changes include a UI Evidence section below. (N/A — no visible UI change.)
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

enqueueWebhookByEnv attempted JSON.parse before writing any
webhook_events row, so an unparseable delivery left zero durable trace
anywhere -- indistinguishable from GitHub never having sent it. Hash the
raw body before the parse attempt and record an "error" row on a parse
failure so every delivery is traceable.

drainOrbRelay's pull-mode batch parser silently filtered out any relayed
event missing/mistyping one of its three required fields, with no log
and no counter. Log a structured, Sentry-visible event and increment a
counter on the drop, naming which field was missing.

Part of #3812; the new fast open-PR reconciliation job is tracked
separately.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.41%. Comparing base (ca6a6d7) to head (42e8f4b).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3838   +/-   ##
=======================================
  Coverage   93.41%   93.41%           
=======================================
  Files         326      326           
  Lines       32855    32859    +4     
  Branches    12035    12035           
=======================================
+ Hits        30691    30695    +4     
  Misses       1530     1530           
  Partials      634      634           
Files with missing lines Coverage Δ
src/github/webhook.ts 100.00% <100.00%> (ø)
src/orb/broker-client.ts 99.14% <100.00%> (+0.02%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 6, 2026
@loopover-orb

loopover-orb Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-06 19:22:47 UTC

4 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

Review summary
This PR closes two real zero-trace loss points on the ingestion path: webhook.ts now hashes the raw body before the JSON.parse attempt and writes a durable webhook_events row with status "error" on parse failure, and broker-client.ts's drainOrbRelay now logs a structured error and increments a counter for any malformed relay batch entry instead of silently dropping it. Both changes are narrowly scoped, match the stated intent, and are covered by tests that exercise the actual code paths (an unparseable body via enqueueWebhookByEnv, and a relay batch entry missing rawBody via drainOrbRelay) rather than fabricated scenarios.

Nits — 5 non-blocking
  • src/github/webhook.ts: the new error-path recordWebhookEvent call omits installationId/action/repositoryFullName (pre-parse they're unknown), which is reasonable, but worth confirming the repository layer treats those as nullable/optional rather than requiring them.
  • src/orb/broker-client.ts:266's console.error is intentional structured logging (matches the existing selfhost_webhook_enqueue_failed pattern), not a debug leftover — the external brief's flag on this line is a false positive worth disregarding.
  • Repeated redelivery of the same malformed/unparseable payload will just overwrite the same webhook_events row by deliveryId each time; fine for tracing but worth a one-line note if operators expect a per-attempt count.
  • Consider whether the new invalid_json webhook_events row should also carry a truncated hint of the parse error (e.g. exception message) the way registerOrbRelayTarget's error hint does, to speed up operator triage beyond just "invalid_json".
  • In drainOrbRelay's dropped-event log, consider also including the batch index or a truncated raw fragment (size-bounded) to help correlate which upstream delivery was lost when deliveryId itself is the missing field.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
Signal Result Evidence
Code review ❌ 2 blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 51 registered-repo PR(s), 43 merged, 382 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 51 PR(s), 382 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
  • Official Gittensor activity: 51 PR(s), 382 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 6, 2026
@JSONbored
JSONbored merged commit 031be7f into main Jul 6, 2026
10 checks passed
@JSONbored
JSONbored deleted the fix/zero-trace-webhook-relay-loss branch July 6, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

1 participant