perf(sync): bound family-level transfer matching to 90-day window#2444
perf(sync): bound family-level transfer matching to 90-day window#2444JSONbored wants to merge 1 commit into
Conversation
Family::Syncer#perform_post_sync called auto_match_transfers! with no date filter, causing a full cartesian scan across ALL entries in a family's accounts on every sync. For families with 10,000+ entries this produced a slow self-join query that Sentry flagged 18,894 times (SURE-APP-WX), last seen today. The fix adds a since_date parameter to transfer_match_candidates and auto_match_transfers!. At the family level (nightly post-sync), it now limits the scan to entries dated within the last 90 days. Account-level syncs (account.family.auto_match_transfers!(account: account)) keep since_date: nil for full-history matching on a specific account. Transfer pairs where one side falls outside the 90-day window are still covered by the account-level sync path that fires when individual provider accounts sync, which already scopes by account_id. Fixes SURE-APP-WX
📝 WalkthroughWalkthrough
ChangesDate-scoped auto-transfer matching
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
jjmata
left a comment
There was a problem hiding this comment.
Clean and well-scoped. The since_date filter is correctly applied in both arms of the UNION ALL in the SQL, so the window is symmetric.
One thing to be aware of: any unmatched transfers older than 90 days that exist at the time this merges will never be auto-matched going forward. Users who had a sync gap (e.g., accounts briefly disconnected) with transfers during that gap will need to match them manually. That may be acceptable, but it's worth a note in the PR description or a brief callout in release notes so support is aware.
Also worth confirming: the since_date filter is based on inflow_candidates.date in both UNION branches. If a transfer's two sides have dates 1 day apart straddling the 90-day boundary, one candidate could be excluded. Given date_window is typically ≤7 days, this edge case would require a transfer to be submitted right at the 90-day mark, so practically it's a non-issue — just good to be aware of.
Generated by Claude Code
|
Thanks @jjmata — both are awareness items rather than code changes, so I added a Behavioral note (for support / release notes) section to the PR description capturing them:
No code change needed for either. Repo has no CHANGELOG file, so the PR description is the home for the note (happy to mirror it into release notes if there's a process for that). |
Closes #2447
Problem
Family::Syncer#perform_post_synccalledauto_match_transfers!with no date filter, causing a full cartesian scan across all entries in a family's accounts on every sync.For families with 10,000+ entries, this is an O(N²) self-join on the entries table. Sentry reports this as SURE-APP-WX: 18,894 occurrences, still firing as of today, only affecting 5 accounts (users with large transaction histories).
The offending query scans every
Transactionentry for the family, finds every pair that could be a transfer (matching amount, opposite sign, within 4-day window), and sorts them — with no upper bound on how many entries it examines.Fix
Add a
since_date:parameter totransfer_match_candidatesandauto_match_transfers!, and apply it asAND (:since_date IS NULL OR inflow_candidates.date >= :since_date)in both UNION halves of the SQL.Family::Syncer#perform_post_syncnow passessince_date: 90.days.ago.to_date, limiting each family-level post-sync scan to entries from the last 90 days.Why 90 days is safe:
since_date: nil(full history for that specific account)Callers unchanged:
account.family.auto_match_transfers!(account: account)— account-level path, nosince_date→ still full history for that accounttransfer_match_candidates(...)calls —since_datedefaults tonil→ backward compatibleBehavioral note (for support / release notes)
This bounds family-level post-sync auto-matching to the last 90 days. One consequence worth flagging: any cross-account transfers that are still unmatched and older than 90 days at the time this ships won't be auto-matched going forward — e.g. a user whose accounts were briefly disconnected, with transfers during that gap, would need to match those manually. The account-level path keeps full history (
since_date: nil), so new connections and historical imports are unaffected; this only concerns already-existing, still-unmatched transfers older than 90 days. Worth a line in release notes so support is aware.Edge case (practically a non-issue): the
since_datefilter keys offinflow_candidates.date, so a transfer whose two sides straddle the exact 90-day boundary could have one side excluded. Since the match window is ≤7 days, this only occurs for a transfer landing right at the boundary.Test plan
test/models/family/auto_transfer_matchable_test.rb— 19 tests, 0 failurestest/models/family_test.rb— 17 tests, 0 failurestest/models/family/syncer_test.rb— 3 tests, 0 failures