Skip to content

/rest/v1/tags and db.get_by_tags() return unbounded results — pagination TODO never implemented #1087

Description

@prajakta128

What is the issue?

GET /rest/v1/tags (find_document_by_tag in application/web/web_main.py) calls db.get_by_tags(tags), which runs two unbounded queries:

Node.query.filter(*nodes_where_clause).all()
CRE.query.filter(*cre_where_clause).all()

Neither has a LIMIT. Both the route and the DB method carry an explicit TODO from the pagination never having been implemented:

  • application/web/web_main.py:300 — # TODO: (spyros) paginate
  • application/database/db.py:1615 — # TODO: (spyros), when we have useful tags this needs to be refactored so both standards and CREs become the same query and it gets paginated

A tag matched by many nodes/CREs (tag matching is a LIKE "%tag%" substring match, so even a short/common tag string can match broadly) returns every matching row in one response — unbounded response size on a public endpoint.

Expected behaviour

/rest/v1/tags should support the same page / items_per_page query params as the sibling /rest/v1/id/... route, bounded by the existing
ITEMS_PER_PAGE (20) / MAX_ITEMS_PER_PAGE (100) constants.

Proposed solution

Add get_by_tags_with_pagination() alongside the existing get_by_tags() (same pattern as get_nodes() / get_nodes_with_pagination() already in db.py), so non-paginated callers (e.g. line 109's internal tag-linking use) are unaffected.

Design decision: get_by_tags combines two separate queries (Node, CRE) into one list, so a single .paginate() call (as used in get_nodes_with_pagination) doesn't map directly onto it. Proposed v1: paginate the Node and CRE queries independently with the same
page/items_per_page, and return them as two labeled lists rather than one merged list:

{"nodes": {...}, "cres": {...}, "page": ..., "total_pages": ...}

This keeps each query's pagination correct and avoids manual offset math across heterogeneous result sets. Happy to adjust if a merged/interleaved result is preferred.

CSV/Markdown/OSCAL export formats (opt_format) would keep using the existing unpaginated get_by_tags(), matching how /rest/v1/id/... already treats exports as an unpaginated special case.

Acceptance criteria

  • /rest/v1/tags accepts page / items_per_page, bounded by MAX_ITEMS_PER_PAGE.
  • Export formats (CSV/Markdown/OSCAL) unaffected.
  • New tests in db_test.py / web_main_test.py covering pagination boundaries (page 1, last page, page beyond range, custom items_per_page).
  • Existing test_get_by_tags / test_find_document_by_tag still pass unmodified (backward compatible for non-paginated callers).
  • make lint / make mypy / make test green.

I'd like to work on this

Activity

  1. Anuj192006 commented on Oct 2, 2026

    @Anuj192006

    Hi! I'd like to work on this issue. I'll add pagination to the /rest/v1/tags endpoint and db.get_by_tags(), following the pagination pattern already used elsewhere in the codebase, and add tests for it.

    Could you assign it to me? Happy to adjust the approach if you have something specific in mind.

  2. prajakta128 commented on Oct 4, 2026

    @prajakta128
    Author

    Hi @Anuj192006, thanks for picking this up! Just a heads up: I already opened a PR for this issue about 3 weeks ago (#1089), which adds pagination to /rest/v1/tags with deterministic ordering. Your PR (#1123) overlaps with it, so it ends up being a duplicate.

  3. Anuj192006 commented on Oct 6, 2026

    @Anuj192006

    Hi @prajakta128, thanks for the heads-up and for being upfront about it! I didn't see #1089 when I claimed the issue—that's on me for not checking thoroughly enough.

    I've reviewed your PR and see you took a similar approach with deterministic ordering. Since you got there first, your PR should take priority. I'll leave #1123 open for now in case northdpole wants to compare or cherry-pick anything, but I'm happy to close it if that's cleaner.

    Thanks for the collaboration and for the reminder to check existing work before claiming an issue next time. Good luck with the merge!

  4. added 2 commits that reference this issue on Oct 6, 2026
    3fb0575
    c8294f7
  5. prajakta128 commented on Oct 6, 2026

    @prajakta128
    Author

    Thanks @Anuj192006 — really appreciate the grace about it. Good catch on the same TODO independently; always a good sign when two people spot the same gap. Good luck with your next one!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions