feat(mirror-client): page miner pulls/issues fetches via next_cursor#1487
Closed
anderdc wants to merge 1 commit into
Closed
feat(mirror-client): page miner pulls/issues fetches via next_cursor#1487anderdc wants to merge 1 commit into
anderdc wants to merge 1 commit into
Conversation
High-volume miners' windowed POST /miners/:id/pulls (and /issues) fetch runs the inlined per-row subqueries over the miner's whole PR set in one shot, exceeding MIRROR_HTTP_TIMEOUT_SECONDS; the miner is then skipped onto a stale cached evaluation and new PRs never ingest. Follow the mirror's cursor pagination (now on the windowed POST paths, das-github-mirror#191) so each page bounds the server-side query cost. _fetch_paginated sends limit on every request and cursor once a next_cursor comes back, concatenating each page's rows; a mirror that predates windowed pagination ignores the params and returns no next_cursor, so the loop completes in one request and behavior is unchanged. MIRROR_MAX_PAGES caps runaway paging and logs if hit. Depends on das-github-mirror#191 being deployed to mirror.gittensor.io first; until then this is a safe no-op (single unbounded request).
Collaborator
Author
|
Closing — superseded by das-github-mirror#192. This PR paginated the windowed pulls/issues fetches to keep each request under the 30s read timeout. #192 removed the cause of that latency — the per-row Pagination remains valid as defense against response size (e.g. UID 29's ~4.6 MB payload) rather than latency; the branch is preserved if that's ever needed. Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The validator fetches a miner's PRs/issues via the per-repo windowed
POST /api/v1/miners/:id/pulls(and/issues). That query runs the inlined per-row subqueries (labels, linked issues) over the miner's entire PR set in one shot, so for a high-volume miner it exceedsMIRROR_HTTP_TIMEOUT_SECONDS(30s × 3 attempts).load.pythen catches the error, flags the fetch failed, and falls the miner back to a stale cached evaluation — their new PRs never ingest.Observed live on
gt-vali:POST /miners/156195510/pulls(kiannidev) and the same for UIDs 29 and 212 (the three highest-volume miners) read-time-out every cycle.What
MirrorClientnow follows the mirror's cursor pagination (added to the windowed POST paths in das-github-mirror#191):_fetch_paginatedsendslimit(MIRROR_PAGE_LIMIT=100) on every request andcursoronce the mirror returns anext_cursor, concatenating each page's rows into one response dict before parsing. Each page bounds the server-side subquery cost well under the 30s timeout.MIRROR_MAX_PAGES=100caps runaway paging (and logs if hit) — 10k rows headroom.since_by_repobody rides along on every page;cursor/limitgo as query params.Compatibility
A mirror that predates windowed pagination ignores the params and returns the full list with no
next_cursor→ the loop completes in a single request, identical to today's behavior. So this is safe to merge/release independently, but only helps once #191 is live.Tests
tests/utils/test_mirror_client.py::TestPagination— multi-page cursor following (GET + windowed POST),limit/cursorparams per page, body persistence across pages, single-page degradation whennext_cursoris absent/null, and the page-cap halt. Full file: 35 passed. ruff/format/pyright clean on touched files.