feat: implement deterministic stale-audit detection and re-audit lifecycle

[TarunyaProgrammer]

Round 2 PR - Vyay Stale Audit Detection and Notification Engine

Overview

This PR implements the complete stale-audit and re-audit lifecycle for Vyay. It ensures that when SaaS vendor pricing changes, historically generated audits are checked for correctness and updated without breaking older results.

Here is the core question we set out to solve:

If a founder ran an AI infrastructure audit three months ago, how do we tell them - precisely, with real numbers rather than vague hand-waving - that new vendor pricing has changed their optimal setup and potential savings?

We solved this using immutable snapshots, deterministic recomputation, and user-level batching.

---

The Re-Audit Lifecycle

Here is a visual breakdown of how the entire detection, batching, and notification flow works:

graph TD
    A[User creates Audit V1] --> B[Capture Immutable Pricing Snapshot in DB]
    B --> C[Developer edits pricing.ts]
    C --> D[Daily cron triggers detect-changes API]
    D --> E[Recompute V1 inputs against live pricing]
    E --> F{"Is savings shift material?<br>$5/mo or 5% change"}
    F -- "No" --> G[Ignore change - no notification]
    F -- "Yes" --> H[Group stale audits by email into batches]
    H --> I{"Is email the developer's?<br>Resend Sandbox check"}
    I -- "Yes" --> J[Send consolidated email via Resend]
    I -- "No" --> K[Show fallback warning banner & email HTML preview in UI]
    J --> L[User clicks compare link]
    K --> L
    L --> M[Render ReauditDiffPage side-by-side]

Loading

---

What Was Actually Built

1. Historical Pricing Snapshots

Every time a user runs an audit, we freeze the entire pricing catalog at that exact millisecond.

• We validate the snapshot using a Zod schema before saving.
• We deduplicate snapshots in a pricing_snapshots table using a hash of the content.
• We embed the full snapshot JSON directly inside the audit row as a backup.
• Why this matters: Without this snapshot, we cannot replay the history. If vendor prices change, reloading an old audit would display current prices instead of the rates the user actually saw.

2. Deterministic Recomputation (diffEngine.ts)

A completely pure, stateless TypeScript module that does the heavy lifting:

• Reruns the original, frozen user inputs through today's live pricing catalog.
• Compares the new recommendations against the old ones.
• Determines if the shift is material using a threshold of $5/month or 5% relative difference.
• Why this matters: We intentionally avoided using an LLM to compare pricing. In financial audits, calculations must be reproducible to the penny. No hallucinations, just pure math.

3. Consolidated Notifications and the Sandbox Workaround

When pricing changes affect multiple audits, we group notifications by email to avoid spamming the user.

• The Sandbox Challenge: Since this project runs on a free Resend plan without a custom domain, we are restricted to sending emails only to my developer address (tarunya.programmer@gmail.com).
• The Solution: If a recruiter or reviewer enters their own email, Resend rejects the dispatch. Instead of letting this fail silently or forcing you to buy a domain, the app detects this limitation. It logs the notification batch, saves it to Supabase, and displays an prominent notice banner on the audit page. It then renders a pixel-perfect HTML preview of the email that would have been sent, letting you verify the flow instantly.

4. Stale Banner and Side-by-Side Comparison UI

• StaleBanner.tsx: An amber warning banner that slides in at the top of the audit page if the system detects the pricing has shifted. It explains why the audit is stale and links to the comparison view.
• ReauditDiffPage.tsx: A dashboard comparing V1 and V2. It collapses unchanged vendor recommendations to keep the view clean, highlights plan changes in red/green, and shows the updated monthly savings.

5. API Trigger and GitHub Actions Scheduler

• POST /api/detect-changes: An API endpoint that runs the detection run. You can call it with {"dryRun": true} to preview changes without dispatching emails.
• scheduled-detect-changes.yml: Since Vercel Hobby accounts do not support free cron schedules easily, I set up a free GitHub Actions workflow that pings this API endpoint daily.

---

Honest Tradeoffs and Compromises

1. Manual pricing updates over web scrapers

Writing web scrapers to parse pricing pages of Cursor, OpenAI, or Vercel sounds cool, but it is incredibly brittle. A small CSS tweak on their website breaks the scraper. We opted for a manually maintained catalog in src/data/pricing.ts that compiles cleanly. It is simple, dependable, and easy to review.

2. Simple flat user batching over tool-level nesting

Initially, I spent hours trying to build a complex nesting parser that grouped notifications by user, then by tool, then by audit. The code was a nightmare to read and the email template looked cluttered. I scrapped it at midnight and went with a flat model: group by email, list the affected audits, and show a total savings delta. The simpler model is cleaner and far less buggy.

3. Caching and DB reads

Right now, the comparison page re-evaluates the diff on every single page load. While this is fine for a demo or an MVP, at production scale we would want to cache the generated diff in Supabase to avoid hitting the database and re-running calculations unnecessarily.

---

How to Test the Re-Audit Flow

To verify this behaves exactly as described:

1. Create an audit: Go to the /audit page, enter a stack (e.g. Cursor Pro with 10 seats at $20/month). Enter your email address.
2. Confirm database snapshot: Check your database or query SELECT pricing_snapshot_json FROM audits ORDER BY created_at DESC LIMIT 1;. You will see the Cursor Pro price recorded at $20.
3. Trigger a price change: Open src/data/pricing.ts, find Cursor Pro, and change monthlyPrice from 20 to 30.
4. Run detection: Trigger a POST request to /api/detect-changes with body {"dryRun": false} (or run the endpoint in your browser/postman).
5. Check notifications:
   • If you used tarunya.programmer@gmail.com, check the inbox (or Resend dashboard) for a consolidated summary email.
   • If you used a different email, open the audit result page. You will see the amber warning banner and a button to view the simulated email HTML preview.
6. Open the compare view: Click the compare link. You will see the side-by-side breakdown showing the $10 Cursor price hike and the updated savings delta.
7. Verify historical integrity: Reload the original audit page. The original savings ($20 based) are still frozen and display exactly what was shown during creation.

---

Test Suite Status

We wrote 94 tests across 18 files. They cover everything from serialization to staleness classification:

Test File	Focus Area	Number of Tests
audit-engine.test.ts	Audit generation logic	6
audit.spec.ts	Base calculation rules	6
audit-persistence.test.ts	Snapshot database writes	2
pricing-hash.test.ts	Deduplication hash logic	3
snapshot-serialization.test.ts	Serializing pricing catalog	2
pricing-change.test.ts	Price comparison functions	6
stale-classification.test.ts	Materiality thresholds ($5/5%)	14
diff-engine.test.ts	Diff creation and outcomes	19
recomputation.test.ts	Rerunning inputs against new prices	8
replay-consistency.test.ts	Frozen historical integrity	1
notification-grouping.test.ts	User-level batching	9
notification-sender.test.ts	Resend dispatch logic	4
diff-view-model.test.ts	Mapping diffs to UI states	3
recompute-diff-flow.test.ts	E2E recomputation integration	1
reaudit.test.ts	Supabase re-audit service	5
ai-fallback.test.ts	AI brief fallback safety	1
routing.test.tsx	Front-end page routes	2
LandingPage.spec.tsx	Hero section rendering	2
Total		94

---

Known Limitations

• Lack of Route Auth: The /api/detect-changes route is currently public for reviewer ease. In a real product, this would be guarded by a secure API key or signature header.
• Audit Age: The engine alerts users on any stale audit, even if it was ran a year ago. In production, we should filter out audits older than a few months to avoid annoying users.
• Mobile Responsiveness in Emails: The HTML tables in the Resend emails look clean on desktop but can wrap poorly on narrow mobile screens.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
vyay-ai-credits-auditer	Ready	Preview, Comment	May 21, 2026 7:52am