Skip to content

Commit 29754cb

Browse files
feat(observability): WS4 funnel custom events → New Relic (InstantFunnel)
Wire common/analyticsevent into the api so the conversion funnel (anonymous→provision→claim→paid) is recorded as a per-entity New Relic custom event (InstantFunnel) alongside the existing aggregate Prometheus counter instant_conversion_funnel_total. Closes the WS4 gap: backend→NR custom-event bridge now emits at the funnel points. Emitter: - Package-level analyticsevent.Emitter in handlers (atomic, boxed for type-stable atomic.Value), default = noop. Router builds it once at boot (wireAnalyticsEmitter) from ANALYTICS_BACKEND (default "noop" = INERT; "newrelic" reuses the api's existing *newrelic.Application). Fail-open: the analyticsevent wrapper swallows panics + sanitizes PII (allowlist). noop-default is the flag protection — no separate feature flag. Emit sites (10, alongside — not replacing — the Prom counter): - provision: db/cache/nosql/vector/queue/storage/webhook NewX (anon path) - claim: onboarding.Claim (anon→claimed) - landing: onboarding.StartLanding (top of funnel) - paid: billing.handleSubscriptionCharged (claimed→paid) Attributes are PII-safe + low-cardinality: funnelStep, service, tier, env, hashed fingerprint, opaque teamId. No raw email/token/connection string. Observability (rule 25): new Prom counter instant_analytics_emit_failed_total{reason} (nil_app = NR unconfigured) via the nr failure hook; docs/OBSERVABILITY-FUNNEL-EVENTS.md documents the InstantFunnel event + NRQL + the cohort='synthetic' exclusion. Alert + dashboard tile live in the infra repo (no auto-apply). Tests: recording-emitter assertions per step+attrs, noop-default no-error, PII-not-emitted, registry-iterating allowlist guard, wire-contract step guard, NR nil-app failure-hook → counter, and a DB-backed paid-funnel event test through the real webhook path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 32ee67f commit 29754cb

20 files changed

Lines changed: 748 additions & 1 deletion

.env.example

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,3 +50,9 @@ GITHUB_CLIENT_ID=
5050
GITHUB_CLIENT_SECRET=
5151
GOOGLE_CLIENT_ID=
5252
GOOGLE_CLIENT_SECRET=
53+
54+
# Analytics (WS4 behavioral-intelligence funnel events → New Relic custom events)
55+
# "noop" (default) = inert; drops every event, zero deps, never errors. Set to
56+
# "newrelic" to emit InstantFunnel custom events via the existing New Relic app
57+
# (requires NEW_RELIC_LICENSE_KEY). Leaving it unset/noop is the safe default.
58+
ANALYTICS_BACKEND=noop
Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Observability — WS4 Behavioral-Intelligence Funnel Events
2+
3+
This note documents the New Relic custom event the api emits at the conversion
4+
funnel points, and the Prometheus counter that backs the bridge's failure mode.
5+
It satisfies CLAUDE.md rule 25 (every observability signal ships with its
6+
documentation). The NR **alert + dashboard tile** for these signals live in the
7+
separate `infra` repo (`infra/newrelic/alerts/`, `infra/newrelic/dashboards/`,
8+
`infra/observability/METRICS-CATALOG.md`) — that repo has no auto-apply, so the
9+
operator wires the tiles/alerts there; this note is the source-of-truth for the
10+
event/attribute contract the dashboards FACET on.
11+
12+
## Why this exists
13+
14+
`instant_conversion_funnel_total{step}` (Prometheus) is an **aggregate count**
15+
it answers "how many provisions today" but cannot be keyed on a stable entity
16+
(team / anonymous fingerprint bucket / cohort), so it cannot compute the per-
17+
entity / cohorted funnel KPIs the WS4 plan needs:
18+
19+
- anonymous → claimed (target **> 2%**)
20+
- claimed → paid (target **> 20%**)
21+
22+
The `InstantFunnel` New Relic custom event is the **per-entity** companion. Both
23+
are emitted at every funnel point — the Prometheus counter is **not** removed.
24+
25+
## `InstantFunnel` custom event
26+
27+
Emitted via `common/analyticsevent` (factory-wrapped: fail-open + PII-sanitized).
28+
The api wires the emitter once at boot (`router.wireAnalyticsEmitter`) from
29+
`ANALYTICS_BACKEND` (default `noop`**inert** until New Relic is configured;
30+
the noop default is the flag protection, no separate feature flag needed). The
31+
`newrelic` backend reuses the api's existing `*newrelic.Application`.
32+
33+
| Attribute | Always? | Values / notes |
34+
|----------------|---------|----------------|
35+
| `funnelStep` | yes | `landing` \| `provision` \| `claim` \| `paid` |
36+
| `serviceName` | yes | `api` (FACET to attribute a step to the emitting service) |
37+
| `tier` | most | `anonymous`/`free`/`hobby`/`pro`/… (omitted at `landing`) |
38+
| `env` | provision | `development`/`production`/… (resolved env of the provision) |
39+
| `fingerprint` | anon | **already-hashed** SHA256(/24+ASN) bucket — never a raw IP |
40+
| `teamId` | claim/paid | team UUID (opaque id, not PII) |
41+
42+
PII policy: the attribute map passes through `analyticsevent.Sanitize` (explicit
43+
allowlist + email-hashing) before any backend sees it, so no raw email / token /
44+
connection string can leak even if a future emit site passes one.
45+
46+
### Emit sites (api)
47+
48+
| Step | File:func | Trigger |
49+
|-------------|-----------|---------|
50+
| `landing` | `onboarding.go` `StartLanding` | GET `/start` (top of funnel) |
51+
| `provision` | `db.go`/`cache.go`/`nosql.go`/`vector.go`/`queue.go`/`storage.go`/`webhook.go` `New*` (anonymous path) | anonymous resource provisioned |
52+
| `claim` | `onboarding.go` `Claim` | anonymous → claimed (account created) |
53+
| `paid` | `billing.go` `handleSubscriptionCharged` | claimed → paid (subscription active) |
54+
55+
### NRQL starters
56+
57+
```sql
58+
-- anon->claimed (exclude synthetic prober traffic)
59+
SELECT uniqueCount(fingerprint) FROM InstantFunnel
60+
WHERE funnelStep = 'landing' AND cohort != 'synthetic' SINCE 1 day ago
61+
62+
SELECT uniqueCount(teamId) FROM InstantFunnel
63+
WHERE funnelStep = 'paid' AND cohort != 'synthetic' FACET tier SINCE 7 days ago
64+
```
65+
66+
> **Exclude `cohort = 'synthetic'` from all funnel analysis.** Synthetic
67+
> flow-test traffic (`InstantFlowTest`, emitted by the worker's prober) carries
68+
> `cohort='synthetic'`; the real-traffic funnel `InstantFunnel` events carry no
69+
> cohort attribute, so `WHERE cohort != 'synthetic'` keeps the two separated.
70+
71+
## `instant_analytics_emit_failed_total{reason}` (Prometheus)
72+
73+
Counts behavioral-intelligence custom events **dropped** before reaching the
74+
analytics sink, by `reason`.
75+
76+
- `reason="nil_app"` — the New Relic sink had no `*newrelic.Application` (NR not
77+
configured). This is the **expected steady state** until
78+
`ANALYTICS_BACKEND=newrelic` + a license key are wired, so a flat non-zero
79+
value in that configuration is benign.
80+
- A **sudden climb after** NR is configured means the bridge is dropping real
81+
funnel events — that is the alertable condition (suggested: P2 observability,
82+
warn on `rate(...[10m]) > 0` once `ANALYTICS_BACKEND=newrelic`).
83+
84+
Lazy `*Vec`: not visible at `/metrics` until the first dropped emit observes a
85+
label.

internal/config/config.go

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,15 @@ type Config struct {
150150
MetricsToken string // METRICS_TOKEN — if set, required as Bearer token to access /metrics
151151
DashboardBaseURL string // DASHBOARD_BASE_URL — where to redirect onboarding flows (default: http://localhost:5173)
152152

153+
// AnalyticsBackend selects the behavioral-intelligence custom-event sink
154+
// (common/analyticsevent). Read from ANALYTICS_BACKEND. One of "noop"
155+
// (default — drops every event, zero deps, never errors) or "newrelic"
156+
// (emits InstantFunnel/InstantFlowTest custom events via the existing
157+
// *newrelic.Application). Defaulting to "noop" makes funnel emission INERT
158+
// in any environment where New Relic is not configured — the safe,
159+
// fail-open default, so no separate feature flag is needed.
160+
AnalyticsBackend string
161+
153162
// APIPublicURL is the externally-routable base URL the API runs at
154163
// — used to construct fully-qualified links in outbound emails
155164
// (deletion-confirm, etc). Empty in local dev where the dashboard
@@ -421,7 +430,8 @@ func Load() *Config {
421430
cfg.DeployDomain = getenv("DEPLOY_DOMAIN", "instant.dev")
422431
cfg.ComputeProvider = getenv("COMPUTE_PROVIDER", "noop")
423432
cfg.KubeNamespaceApps = getenv("KUBE_NAMESPACE_APPS", "instant-apps")
424-
cfg.MetricsToken = os.Getenv("METRICS_TOKEN") // empty = open (local dev)
433+
cfg.MetricsToken = os.Getenv("METRICS_TOKEN") // empty = open (local dev)
434+
cfg.AnalyticsBackend = getenv("ANALYTICS_BACKEND", "noop") // noop = inert (no NR sink)
425435
cfg.DashboardBaseURL = getenv("DASHBOARD_BASE_URL", "http://localhost:5173")
426436
cfg.APIPublicURL = strings.TrimRight(getenv("API_PUBLIC_URL", ""), "/")
427437
// Parse DELETION_CONFIRMATION_TTL_MINUTES; fall back to 15 on

internal/handlers/analytics.go

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
package handlers
2+
3+
import (
4+
"context"
5+
"sync/atomic"
6+
7+
"instant.dev/common/analyticsevent"
8+
)
9+
10+
// WS4 behavioral-intelligence funnel events.
11+
//
12+
// This file is the api's bridge from the existing Prometheus conversion-funnel
13+
// counter (instant_conversion_funnel_total — an AGGREGATE count) to the
14+
// per-entity / per-cohort New Relic custom event (InstantFunnel) that the WS4
15+
// observability plan needs for funnel + retention analysis (anon→claim→
16+
// provision→paid). The Prometheus counter stays exactly where it is; every
17+
// funnel emit site now ALSO records an InstantFunnel custom event alongside it.
18+
//
19+
// Why a package-level emitter instead of a struct field on every handler: the
20+
// funnel emit sites live across nine independently-constructed handler structs
21+
// (DBHandler, CacheHandler, OnboardingHandler, BillingHandler, …), each with
22+
// its own constructor. The api already shares process-wide observability deps
23+
// (the `metrics` package globals) the same way. The router wires the concrete
24+
// emitter ONCE at boot via [SetAnalyticsEmitter]; until then — and in every
25+
// unit test that doesn't opt in — the default is the no-op emitter, so funnel
26+
// emission is INERT by default and can NEVER block, slow, or error a request.
27+
//
28+
// Fail-open + inert-by-default IS the flag protection: the analyticsevent
29+
// package wraps every backend so a panic in the sink is swallowed and a nil /
30+
// unconfigured backend is a silent drop. No separate feature flag is needed —
31+
// ANALYTICS_BACKEND defaulting to "noop" means this code path does nothing in
32+
// prod until New Relic is explicitly configured.
33+
34+
// emitterBox wraps the [analyticsevent.Emitter] interface in a single concrete
35+
// struct type so [analyticsEmitter] (an atomic.Value) always sees ONE concrete
36+
// type across Stores — atomic.Value panics if successive Store calls pass
37+
// different concrete types, which a bare interface value would (noop{} vs the
38+
// factory's wrapped{}). The box is the invariant concrete type.
39+
type emitterBox struct{ e analyticsevent.Emitter }
40+
41+
// analyticsEmitter holds the process-wide emitter (boxed). atomic.Value so
42+
// [SetAnalyticsEmitter] (called once at boot, before serving) and the per-request
43+
// reads in [recordFunnelEvent] are race-free. Defaults to the no-op emitter via
44+
// the package init below.
45+
var analyticsEmitter atomic.Value // stores emitterBox
46+
47+
func init() {
48+
// Inert default: no analytics sink until the router wires one. The no-op
49+
// emitter drops every event with zero deps and can never error.
50+
analyticsEmitter.Store(emitterBox{e: analyticsevent.NewNoop()})
51+
}
52+
53+
// SetAnalyticsEmitter installs the process-wide analytics emitter. Called once
54+
// from the router at boot with the emitter built from ANALYTICS_BACKEND (noop by
55+
// default; the New Relic sink when configured). A nil emitter is ignored so a
56+
// mis-wire degrades to the existing no-op rather than panicking on first emit.
57+
func SetAnalyticsEmitter(e analyticsevent.Emitter) {
58+
if e == nil {
59+
return
60+
}
61+
analyticsEmitter.Store(emitterBox{e: e})
62+
}
63+
64+
// getAnalyticsEmitter returns the current process-wide emitter, never nil.
65+
func getAnalyticsEmitter() analyticsevent.Emitter {
66+
if box, ok := analyticsEmitter.Load().(emitterBox); ok && box.e != nil {
67+
return box.e
68+
}
69+
return analyticsevent.NewNoop()
70+
}
71+
72+
// serviceNameAPI is the AttrServiceName value every funnel event from this
73+
// service carries, so a dashboard can FACET by which service emitted the step.
74+
const serviceNameAPI = "api"
75+
76+
// Funnel-step values re-exported from analyticsevent so the per-handler emit
77+
// sites (db/cache/nosql/…/onboarding/billing) reference one in-package constant
78+
// and don't each need to import common/analyticsevent. These MUST stay equal to
79+
// the analyticsevent constants — funnelStepsMatchCanonical (in the test) asserts
80+
// it, and the wire contract (dashboards FACET on these exact strings) depends on
81+
// it.
82+
const (
83+
funnelStepProvision = analyticsevent.FunnelStepProvision
84+
funnelStepClaim = analyticsevent.FunnelStepClaim
85+
funnelStepPaid = analyticsevent.FunnelStepPaid
86+
funnelStepLanding = analyticsevent.FunnelStepLanding
87+
)
88+
89+
// recordFunnelEvent emits one [analyticsevent.EventFunnel] custom event for the
90+
// given funnel step alongside the existing Prometheus counter. It is the single
91+
// chokepoint every funnel emit site routes through so the attribute set stays
92+
// uniform and PII-safe.
93+
//
94+
// Attributes are intentionally low-cardinality and allowlisted (the
95+
// analyticsevent wrapper drops anything not on the PII allowlist before the
96+
// event leaves the process): step, tier, env, service, and — when known — the
97+
// already-hashed fingerprint bucket (SHA256(/24+ASN), never a raw IP) and team
98+
// id (an opaque UUID, not PII). Empty values are omitted so an absent field
99+
// reads as "missing" in NRQL rather than "".
100+
//
101+
// FAIL-OPEN: this never returns an error and the wrapper swallows any panic, so
102+
// a funnel emit can never affect the request path. Callers MUST NOT wrap it in
103+
// error handling.
104+
func recordFunnelEvent(ctx context.Context, step string, attrs funnelAttrs) {
105+
getAnalyticsEmitter().Record(ctx, analyticsevent.EventFunnel, attrs.toMap(step))
106+
}
107+
108+
// funnelAttrs is the typed, PII-safe attribute payload for a funnel event. Only
109+
// these fields can reach an event; the package allowlist is the backstop.
110+
type funnelAttrs struct {
111+
// Tier is the plan tier the funnel step occurred at ("anonymous", "free",
112+
// "pro", …). Low cardinality.
113+
Tier string
114+
// Env is the resolved environment ("development", "production", …).
115+
Env string
116+
// Fingerprint is the already-hashed SHA256(/24+ASN) anonymous bucket, or ""
117+
// for an authenticated step. Never a raw IP.
118+
Fingerprint string
119+
// TeamID is the owning team UUID (opaque id, not PII), or "" when unknown
120+
// (e.g. anonymous provisions before a claim).
121+
TeamID string
122+
}
123+
124+
// toMap renders funnelAttrs + the step into the flat attribute map the emitter
125+
// consumes, omitting empty values so NRQL facets stay clean.
126+
func (a funnelAttrs) toMap(step string) map[string]any {
127+
out := map[string]any{
128+
analyticsevent.AttrFunnelStep: step,
129+
analyticsevent.AttrServiceName: serviceNameAPI,
130+
}
131+
if a.Tier != "" {
132+
out[analyticsevent.AttrTier] = a.Tier
133+
}
134+
if a.Env != "" {
135+
out[analyticsevent.AttrEnv] = a.Env
136+
}
137+
if a.Fingerprint != "" {
138+
out[analyticsevent.AttrFingerprint] = a.Fingerprint
139+
}
140+
if a.TeamID != "" {
141+
out[analyticsevent.AttrTeamID] = a.TeamID
142+
}
143+
return out
144+
}

0 commit comments

Comments
 (0)