Skip to content

fix(google): bound per-site fan-out so large portfolios don't time out - #1

Open
KuznetsovRA wants to merge 1 commit into
izzipizzy:mainfrom
KuznetsovRA:fix/bounded-fanout-concurrency
Open

fix(google): bound per-site fan-out so large portfolios don't time out#1
KuznetsovRA wants to merge 1 commit into
izzipizzy:mainfrom
KuznetsovRA:fix/bounded-fanout-concurrency

Conversation

@KuznetsovRA

Copy link
Copy Markdown

Every per-site fetch was issued with a single Promise.allSettled(pairs.map(...)), i.e. one concurrent HTTP request per property. On an account with ~200 verified sites that saturates the local socket pool — under Docker Desktop the entire batch fails with UND_ERR_CONNECT_TIMEOUT. Because allSettled swallows rejections, the Sites table silently renders zeros and dashes instead of surfacing an error, so the app looks broken with no clue why. fetchDailyBreakdown doubles it: two requests per site, ~400 sockets at once.

Add allSettledLimit() — a worker-pool variant of Promise.allSettled that keeps results positionally aligned and preserves the {status, value|reason} shape, so call sites keep their existing handling. Apply it to the seven per-site/per-URL fan-outs (listSitesWithSummary, fetchPerSiteQueries, fetchPerSitePages, fetchDailyBreakdown, fetchQueryHistory, fetchPortfolioDecay, bulkInspect). Per-account fan-outs are left alone — those are a handful of calls at most.

Default limit is 8. Google's per-user Search Console quota is far higher; the cap is there to protect the socket pool, not the quota.

Measured on a 199-site account (Docker Desktop, macOS):
before — 0 impressions / 0 clicks, every row dashed
after — 7d: 16,109 impressions / 1,788 clicks
28d: 72,932 impressions / 7,078 clicks, page renders in ~8s
Independently confirmed against the API: the same 199 requests run in batches of 10 return 186×200 and 13×403 (unverified sites), 865 data rows — the data was always there, only the fan-out was dropping it.

Every per-site fetch was issued with a single `Promise.allSettled(pairs.map(...))`,
i.e. one concurrent HTTP request per property. On an account with ~200 verified
sites that saturates the local socket pool — under Docker Desktop the entire batch
fails with UND_ERR_CONNECT_TIMEOUT. Because allSettled swallows rejections, the
Sites table silently renders zeros and dashes instead of surfacing an error, so the
app looks broken with no clue why. `fetchDailyBreakdown` doubles it: two requests
per site, ~400 sockets at once.

Add `allSettledLimit()` — a worker-pool variant of Promise.allSettled that keeps
results positionally aligned and preserves the `{status, value|reason}` shape, so
call sites keep their existing handling. Apply it to the seven per-site/per-URL
fan-outs (listSitesWithSummary, fetchPerSiteQueries, fetchPerSitePages,
fetchDailyBreakdown, fetchQueryHistory, fetchPortfolioDecay, bulkInspect).
Per-account fan-outs are left alone — those are a handful of calls at most.

Default limit is 8. Google's per-user Search Console quota is far higher; the cap
is there to protect the socket pool, not the quota.

Measured on a 199-site account (Docker Desktop, macOS):
  before — 0 impressions / 0 clicks, every row dashed
  after  — 7d: 16,109 impressions / 1,788 clicks
           28d: 72,932 impressions / 7,078 clicks, page renders in ~8s
Independently confirmed against the API: the same 199 requests run in batches of
10 return 186×200 and 13×403 (unverified sites), 865 data rows — the data was
always there, only the fan-out was dropping it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant