[Big Pickle][perf] Overall Verdict on System Design & Performance (Latency Tiering + HLD + Core Web Vitals) — 11 dimensions scored 2–4/5, 13 findings indexed (P1–P3), full remediation plan

[teetangh]

Big Pickle — Overall Verdict on System Design & Performance (Latency Tiering + HLD + Core Web Vitals)

Standalone verdict in the Big Pickle review series (see #1421 finance, #1433 booking). This one audits the whole platform across three categories: 1) Subsystem Latency Tiering & SLA/SLO Alignment, 2) Core High-Level Design (HLD) fundamentals, 3) Frontend Performance & Core Web Vitals. 11 dimensions, each scored 1–5 against the current dev head (369f24bc0).

Parents/umbrellas referenced: #1120 + #1124 (Netlify cold-start event-loop stall — OPEN), #1446 (post-commit after() work starves the single-connection pool — OPEN, P1), #1148 (keep-warm/warm-deploy workflows), #932 (root layout shell stay-static rule), #866 (ADR 22 — GH Actions cron drift).

Scope rule (per this audit): raw Lighthouse/CrUX/RUM numbers are not derivable from a static read — every CWV claim below is a code-level risk assessment, and the first remediation item schedules the real measurement. Performance budgets (LCP/INP budgets, TTFB SLOs) are proposed, not yet adopted.

---

1. Executive verdict

The money path (booking → checkout → allocation → payments → payouts) is the most disciplined subsystem in this codebase — distributed Lua-CAS locks with a provable global order, Serializable-txn retries, fail-closed crons, per-service circuit breakers, and hard DB connect/query budgets. That layer is genuinely near-production-grade. The platform layer is not: every route and every scheduled sweep executes inside one Netlify serverless function (___netlify-server-handler), there is no task/event queue (no qstash/inngest/bullmq/sqs/kafka — verified absent; middleware.ts:65 still lists a vestigial /api/inngest/ prefix), background work is a GitHub-Actions cron fleet plus a 5-minute ticker that fires 10 cleanup routes at the same function, and the landing page — the LCP page — hydrates 15 distinct client components that each import framer-motion. Combined with a documented ~24.9–31.5s cold-start event-loop stall under concurrent load (#1120/#1124, re-measured 2026-08-23) and a cross-region Supabase pooler in the request path, TTFB on dynamic routes is effectively unbounded today.

The pattern is consistent across the audit: correct, disciplined, and well-documented at the seams (locks, timeouts, CAS, ADRs) but with no queuing layer, no read/write separation, no origin TTFB contract, and a render pipeline that pays full client hydration on the public marketing path. Nothing here is a rebuild — it is a set of quick-win optimizations and a short list of architectural refactors (queue, function split, motion/queue hydration de-risking) before this scales.

---

2. Scorecard (11 dimensions, 1–5)

#	Category	Dimension	Score	Status / one-line evidence
1	Latency Tiering	Critical Path Isolation	3	Webhooks ack-200 + after() deferral, but deferred work shares the request function (#1446 proves the risk)
2	Latency Tiering	Asynchronous Decoupling	2	No task queue; GH-Actions crons (ADR 22: sub-hourly → ~every 100 min) + 5-min ticker into the same function
3	Latency Tiering	Fault Isolation & Degradation	3	Robust breakers/timeouts; breaker state is per-instance (close to inert) + single-function coalescing
4	Latency Tiering	Caching & Read/Write Segregation	3	Good ISR/tag layering; no read replicas, cross-region pooler, orphaned programs tag (zero revalidateTag("programs") sites)
5	HLD	Latency vs Throughput	3	Bounded/quota'd queries everywhere; one bundled function; 6-level relation include on a list read
6	HLD	Concurrency & Thread-Pool Safety	3	Money-path locks exemplary; ~25s event-loop stall needs no app code to bite; no job/read concurrency isolation
7	HLD	Scalability & Statelessness	4	Stateless + externalized sessions; horizontal scale blocked by single-function bundle + per-instance breaker memory
8	Frontend	Loading (FCP/LCP)	3	ISR-prerendered CDN landing + self-hosted Sora are strong; LCP above the fold is a hydrated framer-motion client component
9	Frontend	Interactivity (INP/TBT)	3	15 client framer-motion sections on /; 33ms counter setInterval for 2s; Intl.NumberFormat per render
10	Frontend	Visual Stability (CLS)	4	Mostly dimensioned/aspect-video; fill avatars lack sizes; raw <img>; /feed 404 avatar asset
11	Frontend	Network (TTFB)	3	Only 1 API route sets Cache-Control; dynamic routes inherit the cold-start stall; middleware hits Upstash per request

Overall: 3.2 / 5 — launch-viable, scale-blocked; the two lowest scores (async decoupling, LCP hydration) are the real debt.

---

3. Verified-correct posture (do NOT regress)

These were re-confirmed against current code and are considered correct by design:

• Webhook ack-200 + idempotency: Razorpay returns a fast 200 and defers via after() to fit the 5s webhook timeout (app/api/webhooks/razorpay/route.ts:113,187–189); the idempotency key is built from signed material only (lines 158–166). Stripe mirrors it.
• Emails are non-fatal off-path: confirmation email failures never block payment processing (lib/payments/webhooks/handlers.ts:2186–2190, 2277–2281).
• Lock architecture: atom-keyed slot-booking:{profile}:{atomISO} covering [start,end), global deadlock-free acquisition order (event → consultee → slot), ownership-checked Lua PEXPIRE renewal, per-flow TTLs, full error taxonomy with retryAfter, and fail-closed acquisition (Redis down → 503, never an unlocked booking) — utils/appointmentlock.ts:266–305,441,493,583–586,671,771,859,924,980.
• DB budgets that actually work through Supavisor: 3s connect / 6s query runtime, 30s connect at build, and the crucial insight that only client-side query_timeout bounds pooled traffic (Supavisor ignores statement_timeout) — lib/prisma.ts:21–68,141–149.
• Fail-open vs fail-closed split: rate limits + maintenance gate fail open with a finite cache (middleware.ts), money crons fail closed (lib/cron/with-cron-lock.ts:39–44), Redis health probe cached 2s/instance (lib/redis.ts:315–332). The old single-breaker-for-everything coupling was already fixed (Stream production-readiness: everything found, bucketed by launch stage #1280).
• Shell stay-static rule: root layout reads no headers()/session so the shell stays static (app/layout.tsx:62–69, infra: cross-region latency (Netlify us-east-2 ↔ Supabase ap-south-1) causes Postgres pooler connection timeouts #932) — this is why ISR works at all.
• The Netlify 25s stall mitigation is final: memory raise measured dead-and-reverted (2048 MB applied 2026-08-22 → 11/12 slow + a 500), lazy-init build reproduced the stall, so "raise memory / lazy init" must not be re-proposed (netlify.toml:18–44).

---

4. Category 1 — Subsystem Latency Tiering & SLA/SLO Alignment

4.1 Critical Path Isolation — 3/5

Strengths: webhooks ack-200 (app/api/webhooks/razorpay/route.ts:187); the after() continuation is the right primitive and is used exactly where the money path cannot afford to await.

Deficiencies:

• after() still runs inside the same function instance, sharing the event loop and the pool with the next request. [payments][P1 HIGH] Post-commit work in after() starves the single-connection pool: two unawaited ~39 s Novu triggers overlap the chat-channel step and it dies at the 3 s connect timeout #1446 is the live proof: two unawaited ~39s Novu triggers overlapped the chat-channel step inside the single-connection pool and it died at the 3s connect timeout. Every new after() job is a new opportunity for this class of starvation.
• No latency tier exists in the deployment model: /api/health, /api/auth, /api/checkout (money), /api/explore/*, and the /api/cleanup/* sweep all resolve to the one ___netlify-server-handler (netlify.toml:34–38).

4.2 Asynchronous Decoupling — 2/5

• No queue exists. No bullmq/amqp/sqs/kafka/inngest/trigger.dev/qstash in package.json. middleware.ts:65 still authed-prefixes /api/inngest/, but Inngest is not installed (the string's only repo occurrence).
• Background work = GH Actions cron fleet (62 workflows) + Netlify scheduled ticker */5 * * * * (netlify/functions/cron-tick.mts:19). ADR 22 measured sub-hourly GH Actions cron drifting to ~once per hundred minutes (Cron architecture: measured GitHub Actions throttling breaks the sub-hourly fleet — move ~10 event-shaped jobs to QStash, keep business crons on GA #866); ADR 27 added the ticker as the workaround.
• The ticker is not a queue — it is a poller that re-enters the same serverless function 10× every 5 minutes, with no backpressure, no durable queue of work items, and loss tolerance only because "failed sweeps get picked up by the Actions backstop".
• Consequence: event → effect latency is bounded by the sweep cadence (up to hours), and every sweep competes with user traffic on the same function. This is a correctness and latency tax.

4.3 Fault Isolation & Degradation — 3/5

Strengths: listed in §3 — the fail-open/fail-closed contract, per-service breakers, cached probes.

Deficiencies:

• Breaker + health-cache state is per-instance memory in a short-lived serverless world — the Redis breaker comment itself admits it is "close to inert" because instances rarely accumulate 5 failures before dying (lib/redis.ts:115–119). Under constant new instances, the breaker effectively never trips.
• The A newly created function instance stalls its event loop for ~24s before any application work #1124 stall cannot be mitigated by any timeout — connect_timeout/setTimeout cannot fire while the loop is blocked.

4.4 Caching & Read/Write Segregation — 3/5

Strengths:

• ISR on the public marketing surface: / revalidate=3600 (app/page.tsx:48, with an explicit comment that force-dynamic was uncacheable-harmful, lines 27–33, 46–47), explore pages revalidate=300 (app/explore/experts/page.tsx:30).
• Data-layer caching via unstable_cache with tags + the hard-won contract that "tag alone never clears Full Route Cache" (lib/data/public-cache.ts:10, purgeExpertSurfaces at 39–43, purgeReviewSurfaces at 53).
• One API route is a model citizen: Cache-Control: public, s-maxage=60, stale-while-revalidate=300 (app/api/explore/recordings/route.ts:37).

Deficiencies:

• No CQRS / read-replica layer. Every read — including hot public browse — traverses the pooled Supabase (ap-south-1) from Netlify (ap-southeast-1); cross-region latency is in the request path with no facade.
• Orphaned cache tag: app/explore/programs/page.tsx:114,144 register tags: ["programs"], but a repo-wide grep shows zero call sites of revalidateTag("programs") (only announcements, experts, reviews are ever purged). Program counts/level aggregates can serve stale for the full 3600s window with no on-demand bust.
• Offset pagination dominates: only lib/data/org-workspace.ts:256–263 uses cursor-on-id; the public directories use offset (lib/data/explore-experts.ts:338/379/408, recordings listing).

---

5. Category 2 — Core High-Level Design

5.1 Latency vs Throughput — 3/5

Strengths: every findMany seen is bounded (take/limit/BATCH_SIZE); a slow-query hook warns >500ms to Sentry (lib/prisma.ts:120–127); retry budgets are explicitly sized against the ~26s function ceiling (REQUEST_PATH_RETRY_CONFIG 5 attempts, CHECKOUT_WAIT_RETRY_CONFIG ~7s ceiling — utils/appointmentlock.ts:108–117).

Deficiencies:

• lib/data/recordings-explore.ts nests a 6-level relation include (meetingSession → slotOfAppointment → appointment → webinar → webinarPlan → consultantProfile) on an offset-paginated list — join cost per page on the hot explore path.
• 392 API route.ts files all live behind the one handler bundle; there is no route-level throughput isolation or independent autoscaling of hot routes (netlify.toml:34–38).

5.2 Concurrency & Thread-Pool Safety — 3/5

Strengths: the money-path lock suite (§3) is the reference implementation: bounded pg.Pool via PG_POOL_MAX (docs say "set to 1 or 2 in serverless", lib/prisma.ts:61–67), adapter reused through globalThis.

Deficiencies:

• The A newly created function instance stalls its event loop for ~24s before any application work #1124 event-loop stall is the throughput multiplier: a burst of new instances under concurrent load blocks the loop ~25–31.5s before any app work. No pool/timer config fixes it; only ISR cache hits reliably dodge it.
• Breaker/probe state is per-instance (see 4.3) — protection is probabilistic at scale.

5.3 Scalability & Statelessness — 4/5

Strengths: stateless serverless; sessions externalized (BetterAuth DB sessions + cookie-presence edge gate in middleware.ts, no DB hit at the edge by design); all mutable coordination lives in Redis/Postgres; horizontal scale via Netlify's autoscaling instances.

Deficiencies: the single-function bundle blocks split-function scaling; per-instance breaker/health-cache memory is lost in churn; the giant handler bundle repeatedly flirted with Netlify's function size cap (mitigated via outputFileTracingExcludes, next.config.mjs:194–210).

---

6. Category 3 — Frontend Performance & Core Web Vitals

All items below are code-level risk assessments — real LCP/INP/TTFB numbers must come from the measurement sweep in §7.

6.1 Loading (FCP/LCP) — 3/5

Strengths:

• The LCP page is ISR-prerendered and served off the CDN with no function invocation (app/page.tsx:27–48).
• Single self-hosted font via next/font with display: swap (lib/fonts.ts:12–16, with a good comment on the per-callsite re-hash trap).
• next/image globally configured for AVIF+WebP (next.config.mjs:228–230); svg/webp/avif optimized assets exist under public/.
• <Suspense> skeletons around the three DB-backed home sections (app/page.tsx:1,103–121).

Deficiencies:

• The LCP hero is "use client" with framer-motion + an animated counter (components/home/HeroSection.tsx:4,20,47,78,89): hydration and motion JS are on the critical path above the fold.
• All 15/15 landing sections import framer-motion (app/page.tsx:3–17, verified grep -rl framer-motion components/home = 15 of 18 files; 15 of 18 are "use client"). framer-motion is in the initial bundle with no lazy/tiered loading.
• No priority/fetchpriority="high" on above-the-fold imagery (hero uses CSS orbs; apply to whatever becomes LCP once measured).

6.2 Interactivity (INP/TBT) — 3/5

• AnimatedNumber spins a 33ms setInterval updating state for 2000ms on the landing hero (HeroSection counter, 60 steps: components/home/HeroSection.tsx:24–36) — main-thread churn that should be requestAnimationFrame-driven or off-thread.
• Intl.NumberFormat is constructed per render in hooks/useCurrency.ts:132 instead of being module-cached.
• Good: Stream scopes, the unified calendar, onboarding steps, and earnings tabs are next/dynamic (components/stream/StreamChatScope.tsx:3, StreamVideoScope.tsx:3, components/scheduling/SafeUnifiedCalendar.tsx:3, app/form/onboarding/page.tsx:35, app/dashboard/consultant/[consultantId]/(features)/earnings/EarningsTabs.tsx:3).

6.3 Visual Stability (CLS) — 4/5

Strengths: dimensioned next/image predominates; recordings thumbnail surfaces are already inside aspect-video containers.

Deficiencies (all low-effort):

• fill next/image avatars/thumbnails with no sizes (wasted bytes + async-decode layout risk):
  • app/explore/programs/plans/consultations/[consultationPlanId]/components/ConsultationDetails.tsx:138
  • app/explore/programs/plans/subscriptions/[subscriptionPlanId]/components/SubscriptionDetails.tsx:160
  • app/explore/programs/plans/webinars/[webinarPlanId]/components/WebinarDetails.tsx:228,280
  • app/explore/programs/plans/classes/[classPlanId]/components/ClassDetails.tsx:266,319
  • app/explore/experts/[consultantId]/components/ProfileHeader.tsx:41
  • app/dashboard/consultant/[consultantId]/(features)/recordings/components/RecordingCard.tsx:110
  • app/explore/enterprise/organisations/[orgSlug]/page.tsx:232,305
• Raw <img> with eslint-disable @next/next/no-img-element at app/explore/recordings/page.tsx:53 and its [slug] page.
• 404 asset: app/feed/page.tsx:39,78 reference /placeholder-avatar.jpg, which does not exist in public/ (only placeholder-user.jpg, placeholder.svg) → real image miss renders a 404 in dev and prod.

6.4 Network (TTFB) — 3/5

Strengths: edge middleware rate limiting before functions run (DDoS containment, table-driven RATE_LIMIT_RULES); HSTS + CSP + security headers on every response; compression + CDN by Netlify.

Deficiencies:

• Only one API route sets a Cache-Control header (app/api/explore/recordings/route.ts:37). All other public GET endpoints are uncached by default.
• Dynamic routes inherit the A newly created function instance stalls its event loop for ~24s before any application work #1124 cold-start stall with no origin watchdog or TTFB contract.
• Middleware performs an Upstash rate-limit round-trip per matching edge request (fail-open by design, but it is an edge dependency on every guarded request).

---

7. Indexed backlog (new findings from this audit)

No new P0s. These are the findings this audit adds, indexed P1→P3. Severity is performance/architecture impact, not money-path impact (none of these touch the payment invariants).

#	Sev	Layer	Finding
A1	P1	observability	Dynamic-route TTFB is unbounded: the #1124 stall (~25–31.5s burst) has no origin SLO, watchdog, or alert wired to /api/health/cron heartbeats
A2	P1	architecture	No task/event queue: cron sweeps + the 5-min ticker are the only background mechanism (ADR 22 drift, #1446 starvation risk)
A3	P2	data-cache	programs tag is registered (app/explore/programs/page.tsx:114,144) but never revalidated — 0 call sites
A4	P2	caching	Only 1 of 392 API routes sets an HTTP cache header; remaining public GETs are uncacheable-by-default
A5	P2	loader	/ hydrates 15 client framer-motion sections; framer-motion is in the initial bundle on the LCP page
A6	P2	faults	Breaker + health-cache state is per-instance and "close to inert" (lib/redis.ts:115–119) — probabilistic under churn
A7	P2	HLD	Single ___netlify-server-handler bundle hosts all 392 routes — no throughput isolation, no split-function scaling
A8	P2	query	6-level relation include on an offset-paginated list (recordings explore)
A9	P2	render	/feed references missing /placeholder-avatar.jpg (404 asset)
A10	P3	page-glue	fill images without sizes (8 sites listed in 6.3)
A11	P3	mark-up	Raw <img> + eslint-disable on recordings pages
A12	P3	JS	33ms counter setInterval (2s) on the hero; Intl.NumberFormat per render
A13	P3	migration	unstable_cache is fully replaced in Next 16 by use cache/Cache Components — plan the migration before next major bump

---

8. Prioritized remediation index

Quick Wins (days, low risk, no architecture change)

1. Fix the /feed 404 asset — point app/feed/page.tsx:39,78 at the existing /placeholder-user.jpg (or ship the file). (A9)
2. Cache-header pass — extend the app/api/explore/recordings/route.ts:37 pattern to all remaining public GET endpoints. (A4)
3. Crank the programs tag — add a revalidateTag("programs") call on the program-count/level mutations, mirroring the purgeExpertSurfaces contract in lib/data/public-cache.ts. (A3)
4. Module-cache Intl.NumberFormat in hooks/useCurrency.ts:132; switch AnimatedNumber to requestAnimationFrame. (A12)
5. sizes on the 8 fill images; swap the two raw <img> to next/image. (A10, A11)
6. Sweep the vestigial /api/inngest/ middleware prefix or adopt the tool. (A2-adjacent)
7. Measure first — Lighthouse on /, an explore page, and checkout; record idle-20-min TTFB on a dynamic route; attach CrUX. Validates or disputes the stall on today's deploy and sets the budgets for A1.

Architecture (weeks, high impact)

1. Adopt a durable queue (QStash or Inngest) so webhooks/email/reminders/no-show care become event-driven rather than reconciliations. Directly repairs the 2/5 async-decoupling score and de-risks the [payments][P1 HIGH] Post-commit work in after() starves the single-connection pool: two unawaited ~39 s Novu triggers overlap the chat-channel step and it dies at the 3 s connect timeout #1446 after()-starvation class. (A2)
2. Split the server-handler function — carve heavy/background/ticker and public GET surfaces into separate Netlify functions so a sweep or a slow explore query can't stall /api/health, /api/auth, or booking reads. (A7) Re-enumerate function names after any adapter bump (the v1-name gotcha in netlify.toml:34–38 costs a day).
3. De-risk the landing hydration — a shared lightweight motion wrapper or next/dynamic around the 15 sections cuts framer-motion off the initial bundle; then re-measure. (A5)
4. Cursor pagination + flattened joins for the deep directories; evaluate a read replica or cached facade so cross-region DB latency leaves the hot read path. (A8)
5. Plan unstable_cache → use cache for the Next 16 bump; keep the "tag alone never clears Full Route Cache" contract (lib/data/public-cache.ts:10). (A13)
6. Adopt SLAs/SLOs + origin watchdog — alert on dynamic-route P95 TTFB (target: warm <800ms, cold <4s) using the existing /api/health stall signals and cron:heartbeat:last; tie ticket A1 to The 3s Prisma connect budget is not enforced — connect timeout fires at t+29.8s and pins the function for 30s #1120/A newly created function instance stalls its event loop for ~24s before any application work #1124 instead of leaving TTFB unbounded.

---

9. Verification posture

Per the series convention: no db push against the shared Supabase project; validate changes against the seeded dev-server with mock payments and the jest suites under __tests__/ relevant to the touched module. For this audit specifically each Quick Win is verified by (a) the measurement sweep in §8.QW7, (b) a Lighthouse pass on the affected surface, and (c) the existing jest suites where cache/tag behavior is touched (e.g. purgeExpertSurfaces contract tests). The architectural items each need a written design note (ADR) before build, matching how ADR 22/27 already document the cron trade-offs this audit builds on.