fix(google): bound per-site fan-out so large portfolios don't time out - #1
Open
KuznetsovRA wants to merge 1 commit into
Open
fix(google): bound per-site fan-out so large portfolios don't time out#1KuznetsovRA wants to merge 1 commit into
KuznetsovRA wants to merge 1 commit into
Conversation
Every per-site fetch was issued with a single `Promise.allSettled(pairs.map(...))`,
i.e. one concurrent HTTP request per property. On an account with ~200 verified
sites that saturates the local socket pool — under Docker Desktop the entire batch
fails with UND_ERR_CONNECT_TIMEOUT. Because allSettled swallows rejections, the
Sites table silently renders zeros and dashes instead of surfacing an error, so the
app looks broken with no clue why. `fetchDailyBreakdown` doubles it: two requests
per site, ~400 sockets at once.
Add `allSettledLimit()` — a worker-pool variant of Promise.allSettled that keeps
results positionally aligned and preserves the `{status, value|reason}` shape, so
call sites keep their existing handling. Apply it to the seven per-site/per-URL
fan-outs (listSitesWithSummary, fetchPerSiteQueries, fetchPerSitePages,
fetchDailyBreakdown, fetchQueryHistory, fetchPortfolioDecay, bulkInspect).
Per-account fan-outs are left alone — those are a handful of calls at most.
Default limit is 8. Google's per-user Search Console quota is far higher; the cap
is there to protect the socket pool, not the quota.
Measured on a 199-site account (Docker Desktop, macOS):
before — 0 impressions / 0 clicks, every row dashed
after — 7d: 16,109 impressions / 1,788 clicks
28d: 72,932 impressions / 7,078 clicks, page renders in ~8s
Independently confirmed against the API: the same 199 requests run in batches of
10 return 186×200 and 13×403 (unverified sites), 865 data rows — the data was
always there, only the fan-out was dropping it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every per-site fetch was issued with a single
Promise.allSettled(pairs.map(...)), i.e. one concurrent HTTP request per property. On an account with ~200 verified sites that saturates the local socket pool — under Docker Desktop the entire batch fails with UND_ERR_CONNECT_TIMEOUT. Because allSettled swallows rejections, the Sites table silently renders zeros and dashes instead of surfacing an error, so the app looks broken with no clue why.fetchDailyBreakdowndoubles it: two requests per site, ~400 sockets at once.Add
allSettledLimit()— a worker-pool variant of Promise.allSettled that keeps results positionally aligned and preserves the{status, value|reason}shape, so call sites keep their existing handling. Apply it to the seven per-site/per-URL fan-outs (listSitesWithSummary, fetchPerSiteQueries, fetchPerSitePages, fetchDailyBreakdown, fetchQueryHistory, fetchPortfolioDecay, bulkInspect). Per-account fan-outs are left alone — those are a handful of calls at most.Default limit is 8. Google's per-user Search Console quota is far higher; the cap is there to protect the socket pool, not the quota.
Measured on a 199-site account (Docker Desktop, macOS):
before — 0 impressions / 0 clicks, every row dashed
after — 7d: 16,109 impressions / 1,788 clicks
28d: 72,932 impressions / 7,078 clicks, page renders in ~8s
Independently confirmed against the API: the same 199 requests run in batches of 10 return 186×200 and 13×403 (unverified sites), 865 data rows — the data was always there, only the fan-out was dropping it.