feat(pdf): watermark estimates and invoices - #408
Open
Clarion1631 wants to merge 10 commits into
Open
Conversation
DEPLOYED LIVE via clasp (verified by re-pulling from the server).
WHAT WAS BROKEN
The bot read nothing for 9 days. gemini-2.5-pro was retired (404) at the
same moment the Cloud project was blocked for billing (403). The chain had
only those two models, so both died with no survivor. Worse, 404/403 were
classified as DECISIVE ("this document is unreadable"), so every receipt
burned all 3 attempts against an API that never read it and parked forever.
The only signal was mail to an inbox nobody watches.
FIXES
1. Model chain -> [gemini-flash-latest, gemini-2.5-flash, gemini-pro-latest].
Leading with a "-latest" alias means Google repoints it as models retire,
so one retirement cannot take the pipeline down. Each verified live.
2. Classifier: 404, 401/403, empty-200 and invalid-JSON-200 now fall through
to the next model WITHOUT charging the document an attempt. Only HTTP 400
(oversized/undecodable payload) stays decisive, because that genuinely is
about the document. The old 403 comment argued a retry would "hide the
outage" - wrong twice: the outage was hidden anyway, and surfacing one is
the watchdog's job, not the retry counter's.
3. Dedup: the non_receipt branch returned BEFORE claiming a dedup key (the
claim sits ~60 lines further down). So a payroll screenshot looked brand
new every 10 minutes - the same $973.25 CJ Havens PDF alerted at 2:59pm
and 3:10pm and landed in _Needs Review twice, byte-identical. It now
claims a weak key (vendor+date+amount) before parking, and a second copy
recognises the first as owner and stays silent.
4. selfHeal.gs (new): pipelineSelfHeal() every 10 min health-checks each
model with a REAL generateContent call - the /models LISTING endpoint
returned both dead models throughout the outage and is useless - then
auto-requeues ONLY files parked by unavailability. Bounded: max 2
auto-requeues per file, 10 files per pass, never requeues while all
models are down. Files the bot judged (nonReceipt/duplicateOf/
amazonAppOwned/emailed) are never touched, so a requeue cannot
double-book. Plus an hourly canary, a daily report, and one alert channel
with a 2h cooldown.
5. requeueParkedReceipts.gs: manual un-park, same skip rules.
OUTSIDE WATCHDOG
scripts/receipt_pipeline_watch.py (Hermes cron 2cf5cb1c5e3c, every 10 min,
Telegram) watches from outside Apps Script, so it still fires when Apps
Script itself is what broke - deleted trigger, revoked auth, disabled
project. The inside layer cannot report its own death. It checks real model
reads, backlog, staleness and duplicate breeding. 6 tests, all passing; it
caught the duplicate bug on its first live run.
VERIFIED: receipts started reading again 1 minute after the push (11:08am
"could not process" -> 2:49pm "payroll / payment-app item", a business
decision that requires having READ the file).
Committed to git only. clasp push is blocked by a Google Workspace reauth policy (invalid_grant / rapt_required) that needs a fresh password + 2FA in a real browser session. The live script still runs the previous deploy and is reading receipts normally - these are hardening, not a restore. 1. BLOCKER - namespace the non-receipt dedup key. The previous commit's dedup fix was worse than the bug it fixed. It claimed the ordinary weak key vendor|date|amount, so a payroll advance to "Charles Havens" for $973.25 on 08-19 would permanently quarantine a GENUINE receipt sharing those three values - and this branch never calls releaseDedupClaims_, so the block never lifts. A silently unbooked expense is far worse than the duplicate email the fix set out to stop. Now prefixed "nonreceipt|" so the two namespaces cannot collide. 2. Identity guard on that claim. A null/garbage vendor or a 0.00 amount collapses every unreadable non-receipt onto ONE key, so the second payroll screenshot of the day gets swallowed as a duplicate of an unrelated one. Claim only when vendor, date and amount are all real. 3. Align 403 handling in analyzeMultiPageMapWithGemini. The main classifier treats 401/403 as a service failure, but the split-map path still treated it as fatal - so during the billing outage a multi-page PDF would have parked as unsplittable while single-page reads correctly retried. Two classifiers disagreeing about one HTTP code is how the original bug survived review in the first place. 4. selfHeal requeues ONLY outage victims. It was requeueing every parked file, so a $0.00 receipt would burn both of its auto-requeues re-deciding what was already decided on content. Now requires parkReason of aiUnavailable, or gaveUp with busyPasses > 0. 5. Alerts go to Telegram, not just email. Kimi's sharpest point: the 9-day outage happened because the ONLY signal was mail to ALERT_EMAIL, an inbox nobody watches - and selfHeal was alerting to that same inbox. Telegram first, email second. Credentials read from Script Properties (TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID), never in code; skipped silently if unset, and a Telegram failure never blocks the email. Both files syntax-checked with node. Backup of all 14 live files is in %LOCALAPPDATA%\Temp\qbo-safe\ (an expired token killed a push mid-flight earlier and the chained rm wiped the working dir - never chain a destructive rm behind a network call). TO DEPLOY: clasp login in a browser where the Workspace reauth can complete, then: cd workspaces/golden-touch/qbo-clasp && clasp push -f Then re-pull and verify before trusting it.
clasp push succeeded after the Workspace RAPT reauth completed. Verified by pulling into a SEPARATE directory and checking the server's own copy: 11/11 markers present, both files syntax-clean. Live on the server now: 1. namespaced non-receipt dedup key (was a BLOCKER in live code) 2. identity guard on that claim 3. split-map 403 aligned with the main classifier 4. selfHeal requeues only true outage victims 5. alerts route to Telegram before email Safety invariants re-checked on the server copy, not just locally: - AMAZON_APP_OWNS_BOOKING guard intact (single writer for Amazon in QBO) - model chain leads with gemini-flash-latest - selfHeal still skips any file with state.emailed (no double-booking) setupTelegramAlerts.gs is new: stores TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID in Script Properties and sends a test message. The token field ships EMPTY and must be blanked again after running - this file lives in a git repo. checkTelegramSetup() reports whether creds are present without printing the token.
Justin: 'my AI always gets confused about how to connect with Google's chat.' Root cause found and recorded. COMPOSIO CANNOT CONNECT GOOGLE CHAT. It lists the toolkit and exposes 8 GOOGLE_CHAT_* tools, but verified live against the API today: toolkits/google_chat -> composio_managed_auth_schemes: [] auth_configs?toolkit_slug=google_chat -> items: [] (compare gmail -> ['OAUTH2']) There is no OAuth flow behind it and no auth config can be created, so those tools return 'No connected account found' forever regardless of what the dashboard shows. My earlier claim that Chat 'just needs allowlisting' was wrong - the allowlist was already correct; the gap is one layer deeper and is not fixable from the Composio side. NATIVE PATH INSTEAD. Apps Script already runs AS jadkins@, so connecting is just declaring scopes - no service account, no domain-wide delegation. googleChat.gs: postToChatWebhook_ (zero-auth, per-space, threaded), chatListSpaces + chatPostMessage (real Chat API via ScriptApp.getOAuthToken), setChatWebhook / checkChatSetup. appsscript.json: chat.spaces.readonly + chat.messages added. NOTE that declaring oauthScopes disables auto-inference, so every scope the script already relied on is now listed explicitly (drive, gmail.send, gmail.modify, script.scriptapp, script.send_mail, spreadsheets, external_request, userinfo.email). Omitting one would silently strip access on re-auth. selfHeal.gs: alerts now fan out to Chat -> Telegram -> email, each wrapped independently so a broken channel cannot suppress the others or break the pipeline that is trying to report a failure. Verified by pulling into a separate directory and checking the SERVER copy: googleChat.js present, chat scope in manifest, selfHeal calls the webhook, Telegram intact, Amazon single-writer guard intact. Skill 'google-chat-integration' records the trap, the three native paths, and the scope-inference gotcha so this does not get re-derived.
Justin asked to message Marge 'directly'. That is a DIFFERENT mechanism from
the webhook path, and the difference is the thing every naive attempt misses:
An incoming webhook is registered INSIDE one specific space through that
space's own UI. There is no way to create one for a 1:1 with another
person. Google's docs: 'Each webhook only works in the Chat space in which
it's registered.'
So a DM needs the Chat API with user auth:
spaces:findDirectMessage?name=users/<email> -> spaces/XXXX
spaces/XXXX/messages -> post
Added chatFindDm(), chatSendDm(), and dmMargeReceiptUpdate() - the last one
has the recipient and the full message baked in so it is a single click.
The message is openly attributed to AI because Justin asked for that
explicitly. It matters beyond politeness: a process change that lands with no
named author reads as a decree from nowhere, and Marge is the person whose job
it changes.
Error paths name their own cause rather than failing blank:
403 -> either the scope is missing OR the Chat API is not enabled on the
script's Cloud project; the error does not distinguish them, so the
log spells out both.
404 -> no DM exists yet; Google will not manufacture a conversation. Send
one message by hand and it exists forever after.
Skill 'google-chat-integration' updated with a When-to-Use table that picks
the mechanism BEFORE any code, the webhook-cannot-DM rule, and the fact that
Chat uses *bold* not **bold**.
…t for WTB 0723 Report-only replacement input after the WTB password rotation killed the browser CSV export. Pulls the GL register via the existing OAuth client, prints per-day txns/in/out/net plus books-derived open/close balances that must prove themselves (exact-cents running-balance chain) or print n/a. --post is deliberately refused: QBO book balances are not the bank's OPENING/CLOSING LEDGER, and STATEMENT days for WTB-0723 are owned by parse-wtb-daily-csv.mjs; the sanctioned QBO->ledger path remains post-qbo-register.mjs (QBO_REGISTER observations).
An inbound WTB line says only "OTHER DEPOSITS DEPOSIT - DDA/MMKT". The check/deposit image is the only evidence naming the payer, and the memo line often names the job. This lets Beverly attribute deposits to clients and projects. Schema (scripts/apply-check-payer-extraction.mjs — NOT prisma migrate, per CLAUDE.md; NOT YET APPLIED to prod, additive nullable columns only): - BankImage.payerName, memoText, extractedAt, extractionModel - CHECK BankImage_extraction_pair: extractedAt and extractionModel are stamped together or not at all - --dry-run prints the SQL; --yes --expect-db --expect-host applies Extractor (scripts/extract-check-payers.mjs): - Gemini Vision (repo's REST pattern from aiGeneratePunchlist), temp 0 - HARD PRIVACY RULE, enforced twice: the prompt forbids reading the MICR strip / routing / account numbers, and scrubField() drops ANY field carrying an 8+ consecutive-digit run (9+ across separators) that is not the known check number/amount, logging a warning. MICR transit glyphs are dropped outright. Nothing bank-account-shaped can reach the DB or even stdout. - Idempotent: skips extractedAt NOT NULL; --dry-run default; --limit N; --from-manifest tests pre-DDL straight from the Drive manifest - --report / post-run REVIEW REPORT: fuzzy payerName→Client and memoText→Project suggestions, print-only. BankImageMatch stays human-confirmation-only; the matcher NEVER writes it. Verified: 12 unit tests green (scrub guard, fuzzy match, manifest derivation), typecheck clean, real dry-run against check #1027 read payer + memo with zero MICR leakage and suggested the Hoppe Project. Blocked until the DDL is applied: --commit and --report.
… caps, needsReview NULL path Kimi review fixes — all pre-commit privacy gaps closed: - scrubField: ban any token run whose TOTAL digit count >= 8 after stripping ALL separators (space/dash/dot/slash/comma/parens) AND interleaved letters. Catches spaced 8-digit accounts, dot/slash- separated routing numbers, and letter-mixed runs (A1B2C3D4E5F6G7H8). Valid ISO/US calendar dates exempted (month/day-range validated). Allow-list now matches the separator-stripped digit string. - parseModelJson: tolerate ```json fences around the model response. - Length caps: payerName 120 / memoText 200, truncate + flag. - needsReview: any drop/truncation flips it; --commit then stores NULL payer/memo (still stamps extractedAt/extractionModel for replay-skip) and logs the row for human review. No schema change; DDL untouched. tests: 12 -> 32, all green; typecheck clean; test:bank-ledger 491 pass / 0 fail.
…vidence on invoices Geordi work order (check-payer UI): - Automation page 'Check images' panel: BankImage worklist with extracted payer/memo (null-safe: 'not yet extracted' when extractedAt IS NULL), fuzzy payer->Client / memo->Project suggestions (lib/check-payer-match.ts, TS port of extract-check-payers.mjs --report with a parity test so the two can't drift), bank-line proposals via lib/bank-image.ts, and a per-image human CONFIRM action. - confirmBankImageMatch (src/lib/actions.ts): the ONE writer of BankImageMatch, gated on financialReports (internal roles only; portal sessions rejected by assertActiveStaff). P2002 loses the double-confirm race honestly. - Invoice detail: paid-by-check milestones with a human-confirmed BankImageMatch show 'Paid by <payer>, chk#<n>' + Drive image link. Number AND amount must both agree (check numbers collide across payers); ambiguity or missing corroboration renders nothing, never a guess. Display-only; no payment notifiers touched. - Both new fetches degrade independently (existing page pattern): a failure shows an honest 'unavailable' card, never takes the page down. Tests: tests/check-payer-match.test.ts, tests/check-evidence.test.ts (added to test:bank-ledger). tsc --noEmit clean, test:unit 481 pass, test:bank-ledger 508 pass, npm run build clean.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a GOLDEN TOUCH REMODELING watermark to every generated estimate and invoice page. Captured estimate PDFs are stamped before they are emailed or filed; if stamping fails, the server rebuilds the estimate instead of sending the original bytes.\n\nTests add two-page and captured-PDF coverage. Release proof and approval are still pending.