[BUG] Delete PIT and clear scroll request fails with AwsSdk2Transport & ApacheHttpClient

[graytaylor0]

What is the bug?

deletePit request to opensearch client fails with sigv4 signing mismatch

org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'DELETE
/_search/point_in_time

... remaining request info...

How can one reproduce the bug?

Create an OpenSearchClient following instructions from here (https://opensearch.org/docs/2.7/clients/java/#connecting-to-amazon-opensearch-service) and then attempt to use the client's deletePit API.

I believe this is because it is a DELETE request with a body. The createPit method works as intended. Sending the request results in a sigv4 error

What is the expected behavior?

The deletePit API call does not have a signature mismatch

What is your host/environment?

Mac OS

implementation 'org.opensearch.client:opensearch-java:2.5.0'
implementation 'org.opensearch.client:opensearch-rest-client:2.7.0'

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[VachaShah]

Hi @graytaylor0, thank you for raising this issue! Would you be up to PR a fix?

[graytaylor0]

Hi @VachaShah,

I'd be happy to make a PR, but I am unsure of how to fix it or debug what is wrong with the current client code

[dblock]

Start by adding a test (unit or integration) that reproduces the issue. I don't see anything method-specific in https://github.com/opensearch-project/opensearch-java/blob/ae4ac0e08e0b60236805fd75e334aa4e05a8c304/java-client/src/main/java/org/opensearch/client/transport/aws/AwsSdk2Transport.java, so we'll need to look into what headers are generated, whether the body is or isn't getting signed, etc.

[graytaylor0]

Created this integration test (#522) which is currently failing with the sigv4 signature error. What is the best way to compare the request that is being sent by the client to what is expected?

[dblock]

I think it's pretty difficult to know what's expected, so from here I'd debug the code path for a DELETE vs. a POST. I suspect it skips signing the body or something like that.

[graytaylor0]

I did some stepping through with the debugger, and it looked like no different between handling of a DELETE and POST request.

The only thing I do notice is that the Content-Length header is added with a value of the request body length, and the error message says that content length is expected to be empty.

The Canonical String for this request should have been
'DELETE
/_search/point_in_time

accept-encoding:gzip
content-length:
content-type:application/json

[graytaylor0]

Also the header for x-amz-content-sha256 is just given a value of "required". I'm guessing this is expected though

[graytaylor0]

Also found a similar past issue that may be helpful (aws/aws-sdk-js#1733)

[harshavamsi]

@graytaylor0 did some initial digging around and it doesn't look like it's an issue with SigV4 signing. I suspect the issue is either with the ApacheHttpClient. You can verify this by setting the first parameter in #522 for the test case to be true. This will use the async client and the test does pass. Let me do some more in-depth research and post my findings here, but in the meanwhile you can try using the Async client and work around this issue

[graytaylor0]

Hi @harshavamsi. Thanks for the information there. That is very helpful. However, I would like to understand the implications of using an async client instead of the non-async client. How will this impact the other API calls that I am currently making successfully with the non-async client? I'm not sure it makes sense to use the async client just for the sake of this failed delete PIT request.