Support jq filters directly?

[baodrate]

Currently, hurl supports capturing responses by transforming them with jsonpath expressions. It would be nice if hurl could additionally support jq filters. jq provides much more powerful transformations and its syntax is well known among developers.

jaq is a rust implementation of the jq language, and could be integrated into hurl relatively easily.

I would like to work on this, if this feature would be desired by the project

[jcamiel]

Hi @baod-rate

Thanks for the idea. After discussion, whereas it can be useful, we're not going to include it:

• we have a lot of issues with JSONPath: as there is discrepancy between the "reference" document (https://goessner.net/articles/JsonPath/) and various implementations of JSONPath, we have issues about why queries are not working in Hurl, whereas they're working with https://jsonpath.com. Having multiple implementations, between what we think is the spec and other implementations raises issues about what's "the correct behaviour". If the crate jaq decides to have slightly different behaviour compared to jq, we'll have the same type of issues
• speaking of jaq, the crates has a lot of other dependencies and we're always reluctant to increment the number of crates dependencies,
• we have now filters that works on all Hurl queries (jsonpath, xpath etc...) so we prefer to favor the usage of filter instead of adding another way of querying JSON.

That said, I confess don't know all the powers of jq. Do you have examples of what is difficult to do with our actual jsonpath / filters combination and would be easy with a jq filter ?

[baodrate]

If the crate jaq decides to have slightly different behaviour compared to jq, we'll have the same type of issues

That's fair. And though jaq has been good at maintaining consistency with jq where possible, there are certain irreconcilable (albeit minor) differences that arise because of jaq's more strict language design

speaking of jaq, the crates has a lot of other dependencies and we're always reluctant to increment the number of crates dependencies

totally fair

we have now filters that works on all Hurl queries (jsonpath, xpath etc...) so we prefer to favor the usage of filter instead of adding another way of querying JSON

The issue I was having was it wasn't possible (AFAIK) to produce the filter I needed with the expressions that jsonpath allows.

Do you have examples of what is difficult to do with our actual jsonpath / filters combination and would be easy with a jq filter?

Admittedly, my use-case might be uncommon so I'm totally open to it simply being out-of-scope. But I find myself reaching for jq when I need anything more sophisticated than extracting a simple path. To use an example from MDN

GET https://heroes.org/api/squads
[QueryStringParams]
name: "Super hero squad"

HTTP 200
[Captures]
# get the name of the oldest member that has either 'Radiation resistance' or 'Immortality'
member-name: jq ".members | map(select(.powers[]|.=='Radiation resistance' or .=='Immortality')) | sort_by(.age)[-1] | .name"
# 

Although jq is many orders of magnitude more expressive than this example, this is the simplest one I can think of that (AFAICT) is not possible with only Hurl filters. I sometimes make involved hurl request chains and the lack of expressiveness makes crafting them a bit of a headache.

I do recognize that jq's filters overlap a great deal with hurl's. Design wise I have to admit that it's probably better to work on extending hurl's language-agnostic filters. Still, a lot of the time I wish I could just fall back to jq. I wish it were possible to simply pipe the result into a command and capture the output, allowing unbounded expressiveness in the query syntax.

[Captures]
foobar: sh "jq -ecr '.authors | map(select(.books|map(select(.published_year<1990))|length>3)) | sort_by(.last_name)[0]'"
"

although I assume the maintainers have already considered that design and dismissed it

[luisbelloch]

One possible use case that is easy with jq is to aggregate values for a later use.

Let me put an example, let's say we do a first request that returns a list of instruments, we want to extract the name of it and use it in a second request:

GET https://example.com
HTTP/2 200
[Captures]
instruments: jsonpath "$.instruments[*].name"

This first endpoint would return something like:

{"instruments":[{"name":"TRY_JPY"},{"name":"USD_GBP"}]}

The instruments captured variable is:

* Captures:
* instruments: [TRY_JPY,USD_GBP]

And the second request we want to do is:

GET https://example.com/pricing?instruments={{instruments}}

Which... gives an error because it's a collection instead of an string.

We could solve this particular case by adding a join or reduce filter, but that would introduce many other corner cases to maintain in the future. This would be handled with jq by [.instruments[].name] | join(",") | @uri, no need to replicate that functionality in filters.

I see the point on bringing many dependencies into the project — maybe you can consider assert/captures/filters using plain shell commands?