Skip to content

feat(mirror-client): page miner pulls/issues fetches via next_cursor#1487

Closed
anderdc wants to merge 1 commit into
testfrom
feat/paginate-mirror-miner-fetches
Closed

feat(mirror-client): page miner pulls/issues fetches via next_cursor#1487
anderdc wants to merge 1 commit into
testfrom
feat/paginate-mirror-miner-fetches

Conversation

@anderdc

@anderdc anderdc commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Why

The validator fetches a miner's PRs/issues via the per-repo windowed POST /api/v1/miners/:id/pulls (and /issues). That query runs the inlined per-row subqueries (labels, linked issues) over the miner's entire PR set in one shot, so for a high-volume miner it exceeds MIRROR_HTTP_TIMEOUT_SECONDS (30s × 3 attempts). load.py then catches the error, flags the fetch failed, and falls the miner back to a stale cached evaluation — their new PRs never ingest.

Observed live on gt-vali: POST /miners/156195510/pulls (kiannidev) and the same for UIDs 29 and 212 (the three highest-volume miners) read-time-out every cycle.

What

MirrorClient now follows the mirror's cursor pagination (added to the windowed POST paths in das-github-mirror#191):

  • _fetch_paginated sends limit (MIRROR_PAGE_LIMIT=100) on every request and cursor once the mirror returns a next_cursor, concatenating each page's rows into one response dict before parsing. Each page bounds the server-side subquery cost well under the 30s timeout.
  • MIRROR_MAX_PAGES=100 caps runaway paging (and logs if hit) — 10k rows headroom.
  • The windowed since_by_repo body rides along on every page; cursor/limit go as query params.

Compatibility

A mirror that predates windowed pagination ignores the params and returns the full list with no next_cursor → the loop completes in a single request, identical to today's behavior. So this is safe to merge/release independently, but only helps once #191 is live.

Tests

tests/utils/test_mirror_client.py::TestPagination — multi-page cursor following (GET + windowed POST), limit/cursor params per page, body persistence across pages, single-page degradation when next_cursor is absent/null, and the page-cap halt. Full file: 35 passed. ruff/format/pyright clean on touched files.

High-volume miners' windowed POST /miners/:id/pulls (and /issues) fetch
runs the inlined per-row subqueries over the miner's whole PR set in one
shot, exceeding MIRROR_HTTP_TIMEOUT_SECONDS; the miner is then skipped
onto a stale cached evaluation and new PRs never ingest.

Follow the mirror's cursor pagination (now on the windowed POST paths,
das-github-mirror#191) so each page bounds the server-side query cost.
_fetch_paginated sends limit on every request and cursor once a
next_cursor comes back, concatenating each page's rows; a mirror that
predates windowed pagination ignores the params and returns no
next_cursor, so the loop completes in one request and behavior is
unchanged. MIRROR_MAX_PAGES caps runaway paging and logs if hit.

Depends on das-github-mirror#191 being deployed to mirror.gittensor.io
first; until then this is a safe no-op (single unbounded request).
@anderdc

anderdc commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing — superseded by das-github-mirror#192.

This PR paginated the windowed pulls/issues fetches to keep each request under the 30s read timeout. #192 removed the cause of that latency — the per-row contributor_repo_roles re-derivation — by repointing the label/review views onto an indexed maintainers table. The timeout is now gone at the root: the heaviest miners' full unbounded windowed POST returns in 0.5–1.9s (was ~12s+/timeout), so the validator no longer needs to page to stay under the limit.

Pagination remains valid as defense against response size (e.g. UID 29's ~4.6 MB payload) rather than latency; the branch is preserved if that's ever needed.

Closing.

@anderdc anderdc closed this Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant