Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP requests and response body validation don't work with non-object content #262

Open
hawkeyexl opened this issue Jan 13, 2025 · 1 comment
Open

HTTP requests and response body validation don't work with non-object content #262

hawkeyexl opened this issue Jan 13, 2025 · 1 comment
Assignees
@hawkeyexl
Labels
bug Something isn't working

Comments

@hawkeyexl
Copy link
Contributor

Original report

If the request body is something other than a JSON object (even just an array), the test refuses to run.

If a response body is something other than a JSON object (even just an array), response validation fails.
@hawkeyexl hawkeyexl added the bug Something isn't working label Jan 13, 2025
@hawkeyexl hawkeyexl self-assigned this Jan 13, 2025
@hawkeyexl hawkeyexl added this to 2.19.0 Jan 13, 2025
@github-project-automation github-project-automation bot moved this to Todo in 2.19.0 Jan 13, 2025
@anaxite
Copy link

anaxite commented Jan 13, 2025

Here is an example request that causes issues.

1. Create a request with the GitHub Commits API

The following request gets the last two commits containing src/config.js in this repository.

curl --request GET \
  --url 'https://api.github.com/repos/doc-detective/doc-detective-core/commits?path=src%2Fconfig.js&per_page=2' \
  --header 'accept: application/vnd.github+json' \
  --header 'x-github-api-version: 2022-11-28'

2. Analyze the response

[
  {
    "sha": "3b40e4448e8db95a1f84f960264a4d0586bb1aff",
    "node_id": "C_kwDOI1jjuNoAKDNiNDBlNDQ0OGU4ZGI5NWExZjg0Zjk2MDI2NGE0ZDA1ODZiYjFhZmY",
    "commit": {
      "author": {
        "name": "hawkeyexl",
        "email": "[email protected]",
        "date": "2024-10-11T02:18:20Z"
      },
      "committer": {
        "name": "hawkeyexl",
        "email": "[email protected]",
        "date": "2024-10-11T02:18:20Z"
      },
      "message": "Add error handling for loading OpenAPI descriptions and remove failed configurations",
      "tree": {
        "sha": "1ad1dfea9a42fa4e7f63f335c230f6541dc35296",
        "url": "https://api.github.com/repos/doc-detective/doc-detective-core/git/trees/1ad1dfea9a42fa4e7f63f335c230f6541dc35296"
      },
      "url": "https://api.github.com/repos/doc-detective/doc-detective-core/git/commits/3b40e4448e8db95a1f84f960264a4d0586bb1aff",
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null,
        "verified_at": null
      }
    },
    "url": "https://api.github.com/repos/doc-detective/doc-detective-core/commits/3b40e4448e8db95a1f84f960264a4d0586bb1aff",
    "html_url": "https://github.com/doc-detective/doc-detective-core/commit/3b40e4448e8db95a1f84f960264a4d0586bb1aff",
    "comments_url": "https://api.github.com/repos/doc-detective/doc-detective-core/commits/3b40e4448e8db95a1f84f960264a4d0586bb1aff/comments",
    "author": {
      "login": "hawkeyexl",
      "id": 5209367,
      "node_id": "MDQ6VXNlcjUyMDkzNjc=",
      "avatar_url": "https://avatars.githubusercontent.com/u/5209367?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/hawkeyexl",
      "html_url": "https://github.com/hawkeyexl",
      "followers_url": "https://api.github.com/users/hawkeyexl/followers",
      "following_url": "https://api.github.com/users/hawkeyexl/following{/other_user}",
      "gists_url": "https://api.github.com/users/hawkeyexl/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/hawkeyexl/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/hawkeyexl/subscriptions",
      "organizations_url": "https://api.github.com/users/hawkeyexl/orgs",
      "repos_url": "https://api.github.com/users/hawkeyexl/repos",
      "events_url": "https://api.github.com/users/hawkeyexl/events{/privacy}",
      "received_events_url": "https://api.github.com/users/hawkeyexl/received_events",
      "type": "User",
      "user_view_type": "public",
      "site_admin": false
    },
    "committer": {
      "login": "hawkeyexl",
      "id": 5209367,
      "node_id": "MDQ6VXNlcjUyMDkzNjc=",
      "avatar_url": "https://avatars.githubusercontent.com/u/5209367?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/hawkeyexl",
      "html_url": "https://github.com/hawkeyexl",
      "followers_url": "https://api.github.com/users/hawkeyexl/followers",
      "following_url": "https://api.github.com/users/hawkeyexl/following{/other_user}",
      "gists_url": "https://api.github.com/users/hawkeyexl/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/hawkeyexl/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/hawkeyexl/subscriptions",
      "organizations_url": "https://api.github.com/users/hawkeyexl/orgs",
      "repos_url": "https://api.github.com/users/hawkeyexl/repos",
      "events_url": "https://api.github.com/users/hawkeyexl/events{/privacy}",
      "received_events_url": "https://api.github.com/users/hawkeyexl/received_events",
      "type": "User",
      "user_view_type": "public",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "e401838f45491b43ecd4a5a902e9a60336c0f082",
        "url": "https://api.github.com/repos/doc-detective/doc-detective-core/commits/e401838f45491b43ecd4a5a902e9a60336c0f082",
        "html_url": "https://github.com/doc-detective/doc-detective-core/commit/e401838f45491b43ecd4a5a902e9a60336c0f082"
      }
    ]
  },
  {
    "sha": "8c1bf56ccdca8c2870ee23503b6174a6a3b1ca56",
    "node_id": "C_kwDOI1jjuNoAKDhjMWJmNTZjY2RjYThjMjg3MGVlMjM1MDNiNjE3NGE2YTNiMWNhNTY",
    "commit": {
      "author": {
        "name": "hawkeyexl",
        "email": "[email protected]",
        "date": "2024-10-09T02:40:23Z"
      },
      "committer": {
        "name": "hawkeyexl",
        "email": "[email protected]",
        "date": "2024-10-09T02:40:23Z"
      },
      "message": "Update for proper terminology",
      "tree": {
        "sha": "fd12c87a627ba6b5c422ec097e023a333df7f5c0",
        "url": "https://api.github.com/repos/doc-detective/doc-detective-core/git/trees/fd12c87a627ba6b5c422ec097e023a333df7f5c0"
      },
      "url": "https://api.github.com/repos/doc-detective/doc-detective-core/git/commits/8c1bf56ccdca8c2870ee23503b6174a6a3b1ca56",
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null,
        "verified_at": null
      }
    },
    "url": "https://api.github.com/repos/doc-detective/doc-detective-core/commits/8c1bf56ccdca8c2870ee23503b6174a6a3b1ca56",
    "html_url": "https://github.com/doc-detective/doc-detective-core/commit/8c1bf56ccdca8c2870ee23503b6174a6a3b1ca56",
    "comments_url": "https://api.github.com/repos/doc-detective/doc-detective-core/commits/8c1bf56ccdca8c2870ee23503b6174a6a3b1ca56/comments",
    "author": {
      "login": "hawkeyexl",
      "id": 5209367,
      "node_id": "MDQ6VXNlcjUyMDkzNjc=",
      "avatar_url": "https://avatars.githubusercontent.com/u/5209367?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/hawkeyexl",
      "html_url": "https://github.com/hawkeyexl",
      "followers_url": "https://api.github.com/users/hawkeyexl/followers",
      "following_url": "https://api.github.com/users/hawkeyexl/following{/other_user}",
      "gists_url": "https://api.github.com/users/hawkeyexl/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/hawkeyexl/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/hawkeyexl/subscriptions",
      "organizations_url": "https://api.github.com/users/hawkeyexl/orgs",
      "repos_url": "https://api.github.com/users/hawkeyexl/repos",
      "events_url": "https://api.github.com/users/hawkeyexl/events{/privacy}",
      "received_events_url": "https://api.github.com/users/hawkeyexl/received_events",
      "type": "User",
      "user_view_type": "public",
      "site_admin": false
    },
    "committer": {
      "login": "hawkeyexl",
      "id": 5209367,
      "node_id": "MDQ6VXNlcjUyMDkzNjc=",
      "avatar_url": "https://avatars.githubusercontent.com/u/5209367?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/hawkeyexl",
      "html_url": "https://github.com/hawkeyexl",
      "followers_url": "https://api.github.com/users/hawkeyexl/followers",
      "following_url": "https://api.github.com/users/hawkeyexl/following{/other_user}",
      "gists_url": "https://api.github.com/users/hawkeyexl/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/hawkeyexl/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/hawkeyexl/subscriptions",
      "organizations_url": "https://api.github.com/users/hawkeyexl/orgs",
      "repos_url": "https://api.github.com/users/hawkeyexl/repos",
      "events_url": "https://api.github.com/users/hawkeyexl/events{/privacy}",
      "received_events_url": "https://api.github.com/users/hawkeyexl/received_events",
      "type": "User",
      "user_view_type": "public",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "ed6298e3515ecd608d74cbd4d8142561f269c9a3",
        "url": "https://api.github.com/repos/doc-detective/doc-detective-core/commits/ed6298e3515ecd608d74cbd4d8142561f269c9a3",
        "html_url": "https://github.com/doc-detective/doc-detective-core/commit/ed6298e3515ecd608d74cbd4d8142561f269c9a3"
      }
    ]
  }
]

The JSON response for this API endpoint is always contained in a top-level array. Let's try to get a value.

3a. Create a test with the GitHub Commits API.

We try to test the "sha" value of the first entry.

{
  "tests": [
    {
      "id": "check-file-sha",
      "description": "Check file SHA",
      "steps": [
        {
          "action": "httpRequest",
          "url": "https://api.github.com/repos/doc-detective/doc-detective-core/commits",
          "method": "get",
          "requestHeaders": {
            "Accept": "application/vnd.github+json",
            "X-GitHub-Api-Version": "2022-11-28"
          },
          "requestParams": {
            "path": "src/config.js",
            "per_page": "2"
          },
          "responseData": {
            "sha": "3b40e4448e8db95a1f84f960264a4d0586bb1aff"
          },
          "statusCodes": [200]
        }
      ]
    }
  ]
}

3b. Test failure

The test fails with an error because the top-level data structure is an array that doesn't contain a "sha" key:

"result": "FAIL",
"resultDescription": "Returned 200. The 'sha' key did't exist in returned JSON.",

4a. Create a second test with an array object

Take the previous test, and encapsulate responseData in square brackets:

"responseData": [{
  "sha": "3b40e4448e8db95a1f84f960264a4d0586bb1aff"
}],

4b. Test 2

The test fails because Doc Detective thinks there are no tests.

{
  "summary": {
    "specs": {
      "pass": 0,
      "fail": 0,
      "warning": 0,
      "skipped": 0
    },
    "tests": {
      "pass": 0,
      "fail": 0,
      "warning": 0,
      "skipped": 0
    },
    "contexts": {
      "pass": 0,
      "fail": 0,
      "warning": 0,
      "skipped": 0
    },
    "steps": {
      "pass": 0,
      "fail": 0,
      "warning": 0,
      "skipped": 0
    }
  },
  "specs": []
}

Thoughts

Because responseData needs its content defined inline, it's not clear how to reference specific parts of a JSON data structure. Adding characters like square brackets have a bad interaction with the key.

It would be useful to accept special syntax to test a specific key in the response.
In Bruno, for example, the following selector string works: res.body[0].sha

@hawkeyexl hawkeyexl removed this from 2.19.0 Feb 1, 2025
@hawkeyexl hawkeyexl added this to v3.0.0 Feb 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hawkeyexl hawkeyexl

Labels
bug Something isn't working
Projects
v3.0.0
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hawkeyexl @anaxite