Skip to content

feat(miners): add cursor pagination to windowed POST pulls/issues endpoints#191

Merged
entrius merged 1 commit into
testfrom
feat/paginate-windowed-pulls-issues
Jun 16, 2026
Merged

feat(miners): add cursor pagination to windowed POST pulls/issues endpoints#191
entrius merged 1 commit into
testfrom
feat/paginate-windowed-pulls-issues

Conversation

@anderdc

@anderdc anderdc commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Why

The per-repo windowed POST /api/v1/miners/:id/pulls and /issues endpoints run unbounded: every matched PR/issue executes the inlined labels + linked_issues subqueries, so a high-volume author's request exceeds the validator's 30s read timeout. The validator then catches the error, falls the miner back to a stale cached evaluation, and the miner's new PRs/issues never get ingested.

Observed live: POST /miners/156195510/pulls (kiannidev, 621 PRs) reliably read-times-out at 30s × 3 attempts; same for the other two highest-volume miners (UIDs 29, 212). The mirror has the data — it's purely query latency on the unbounded windowed read.

What

Mirror the cursor pagination already shipped for the GET paths (#126) onto the windowed POST variants:

  • getPullRequestsByRepo / getIssuesByRepo accept an optional pagination arg: keyset on (created_at, LOWER(repo_full_name), number), deterministic matching ORDER BY, LIMIT, and a next_cursor in the response.
  • POST /pulls and /issues accept optional ?cursor= / ?limit= query params (body stays the since_by_repo map), routed through the existing parsePaginationQuery.

Compatibility

Default responses (no cursor/limit) stay unbounded and unchanged, so validators that have not yet adopted paging are unaffected. The companion gittensor change (validator follows next_cursor) ships only after this deploys.

Verify

curl -s -X POST "https://mirror.gittensor.io/api/v1/miners/156195510/pulls?limit=2" \
  -H "Content-Type: application/json" \
  -d '{"since_by_repo":{"phase-rs/phase":"2026-05-12T00:00:00Z"}}' | jq '.next_cursor, (.pull_requests|length)'

format:check, lint, and nest build all pass.

…points

The per-repo windowed POST /miners/:id/pulls and /issues endpoints ran
unbounded: every matched PR/issue executes the inlined labels and
linked-issues subqueries, so a high-volume author's request exceeds the
validator's 30s read timeout and the miner is skipped onto a stale
cached evaluation.

Mirror the cursor pagination from the GET paths (#126) onto the windowed
variants: optional cursor/limit query params, keyset on
(created_at, lower(repo_full_name), number), and a next_cursor in the
response. Default (no cursor/limit) responses stay unbounded, so callers
that have not adopted pagination are unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants