Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken pagination function of Stash provider #166

Open
Porsh33 opened this issue Apr 11, 2022 · 4 comments
Open

Broken pagination function of Stash provider #166

Porsh33 opened this issue Apr 11, 2022 · 4 comments

Comments

@Porsh33
Copy link

Porsh33 commented Apr 11, 2022

func encodeListOptions(opts scm.ListOptions) string {
params := url.Values{}
if opts.Page > 1 {
params.Set("start", strconv.Itoa(
(opts.Page-1)*opts.Size),
)
}
if opts.Size != 0 {
params.Set("limit", strconv.Itoa(opts.Size))
}
return params.Encode()
}

Reference to docs https://docs.atlassian.com/bitbucket-server/rest/5.16.0/bitbucket-rest.html

Important: If more than one page exists (i.e. the response contains "isLastPage": false), the response object will also contain a nextPageStart attribute which must be used by the client as the start parameter on the next request. Identifiers of adjacent objects in a page may not be contiguous, so the start of the next page is not necessarily the start of the last page plus the last page's size. A client should always use nextPageStart to avoid unexpected results from a paged API.

@bradrydzewski
Copy link
Member

bradrydzewski commented Apr 11, 2022

We have not experienced any issues with the below logic, so we would kindly ask that you provide more details. For example, can you narrow it down to a subset of endpoints and can you describe in more detail the error message or problems you are experiencing as a result of the below logic? Also, are you experiencing this problem through your use of Drone, or are you using this library separately?

	if !out.pagination.LastPage.Bool {
		res.Page.First = 1
		res.Page.Next = opts.Page + 1
	}

@Porsh33
Copy link
Author

Porsh33 commented Apr 14, 2022

I use the Abstruse CI, which has code like this test: https://github.com/drone/go-scm/blob/master/scm/driver/stash/repo_test.go#L257

// ListRepos returns list of repositories.
func (s SCM) ListRepos(page, size int) ([]*scm.Repository, error) {
	repos, _, err := s.client.Repositories.List(s.ctx, scm.ListOptions{Page: page, Size: size})
	return repos, err
}

But i think this test is incorrect (reference data). Because in real Bitbucket Enterprise the API answer with pagination looks like this:

porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "12341234",
      "id": 5063,
      "name": "12341234",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "~RUDSKOY",
        "id": 2809,
        "name": "Vladislav Rudskoy",
        "type": "PERSONAL",
        "owner": {
          "name": "rudskoy",
          "emailAddress": "[email protected]",
          "id": 6612,
          "displayName": "Vladislav Rudskoy",
          "active": true,
          "slug": "rudskoy",
          "type": "NORMAL",
          "links": {
            "self": [
              {
                "href": "https://domain.name/users/rudskoy"
              }
            ]
          }
        },
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/users/rudskoy"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "https://bb.domain.name/scm/~rudskoy/12341234.git",
            "name": "http"
          },
          {
            "href": "ssh://[email protected]/~rudskoy/12341234.git",
            "name": "ssh"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/users/rudskoy/repos/12341234/browse"
          }
        ]
      }
    }
  ],
  "start": 0,
  "nextPageStart": 3
}

if we use the result of func encodeListOptions(Page: 2, Size: 1), the request will be

porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=1" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "12341234",
      "id": 5063,
      "name": "12341234",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "~RUDSKOY",
        "id": 2809,
        "name": "Vladislav Rudskoy",
        "type": "PERSONAL",
        "owner": {
          "name": "rudskoy",
          "emailAddress": "[email protected]",
          "id": 6612,
          "displayName": "Vladislav Rudskoy",
          "active": true,
          "slug": "rudskoy",
          "type": "NORMAL",
          "links": {
            "self": [
              {
                "href": "https://domain.name/users/rudskoy"
              }
            ]
          }
        },
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/users/rudskoy"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "https://bb.domain.name/scm/~rudskoy/12341234.git",
            "name": "http"
          },
          {
            "href": "ssh://[email protected]/~rudskoy/12341234.git",
            "name": "ssh"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/users/rudskoy/repos/12341234/browse"
          }
        ]
      }
    }
  ],
  "start": 1,
  "nextPageStart": 3
}

As you can see, server returns the same data as the first time. This is due to the fact that the value of the "nextPageStart" key is not taken into account when forming the URL. And we can't provide this value "as is" using this function

A new data can be received using request:

porsh@i109885904 ~/work/github [15:31] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=3" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "2021-09-homework-01",
      "id": 16135,
      "name": "2021-09-homework-01",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "BACKENDSCHOOL",
        "id": 24862,
        "name": "Backend School",
        "description": "https://domain.name/services/backendschool",
        "public": false,
        "type": "NORMAL",
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/projects/BACKENDSCHOOL"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "ssh://[email protected]/backendschool/2021-09-homework-01.git",
            "name": "ssh"
          },
          {
            "href": "https://bb.domain.name/scm/backendschool/2021-09-homework-01.git",
            "name": "http"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/projects/BACKENDSCHOOL/repos/2021-09-homework-01/browse"
          }
        ]
      }
    }
  ],
  "start": 3,
  "nextPageStart": 5
}

@bradrydzewski
Copy link
Member

bradrydzewski commented Apr 14, 2022

if we use the result of func encodeListOptions(Page: 2, Size: 1), the request will be
porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer " > "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=1" | jq -r

Just a note that in your example the curl statement still has start=1 and not start=2 which could explain why the contents are the same in both requests.

@Porsh33
Copy link
Author

Porsh33 commented Apr 14, 2022

if we use the result of func encodeListOptions(Page: 2, Size: 1), the request will be
porsh@i109885904 ~/work/github [15:16] $ curl -sH "Authorization: Bearer " > "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=1" | jq -r

Just a note that in your example the curl statement still has start=1 and not start=2 which could explain why the contents are the same in both requests.

It's ok because strconv.Itoa((opts.Page-1)*opts.Size) is equal (2-1)*1 = 1. But even if I try to get data with start=2 I'll get same result as first one (because the new data providing by server starts from "nextPageStart": 3)

porsh@i109885904 ~/work [15:49] $ curl -sH "Authorization: Bearer <token>" "https://bb.domain.name/rest/api/1.0/repos?limit=1&start=2" | jq -r
{
  "size": 1,
  "limit": 1,
  "isLastPage": false,
  "values": [
    {
      "slug": "12341234",
      "id": 5063,
      "name": "12341234",
      "scmId": "git",
      "state": "AVAILABLE",
      "statusMessage": "Available",
      "forkable": true,
      "project": {
        "key": "~RUDSKOY",
        "id": 2809,
        "name": "Vladislav Rudskoy",
        "type": "PERSONAL",
        "owner": {
          "name": "rudskoy",
          "emailAddress": "[email protected]",
          "id": 6612,
          "displayName": "Vladislav Rudskoy",
          "active": true,
          "slug": "rudskoy",
          "type": "NORMAL",
          "links": {
            "self": [
              {
                "href": "https://domain.name/users/rudskoy"
              }
            ]
          }
        },
        "links": {
          "self": [
            {
              "href": "https://bb.domain.name/users/rudskoy"
            }
          ]
        }
      },
      "public": false,
      "links": {
        "clone": [
          {
            "href": "https://bb.domain.name/scm/~rudskoy/12341234.git",
            "name": "http"
          },
          {
            "href": "ssh://[email protected]/~rudskoy/12341234.git",
            "name": "ssh"
          }
        ],
        "self": [
          {
            "href": "https://bb.domain.name/users/rudskoy/repos/12341234/browse"
          }
        ]
      }
    }
  ],
  "start": 1,
  "nextPageStart": 3
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants