Skip to content

Regression in AWS CLI v2 CRT transfer client for http connectors #10748

Closed
#10753
Closed
Regression in AWS CLI v2 CRT transfer client for http connectors#10748
#10753
Labels
bugThis issue is a bug.dependenciesThis issue is a problem in a dependency.p1This is a high priority issueregression

Description

@pguinard-public-com
pguinard-public-com
opened on Oct 12, 2026

Describe the bug

We first noticed these errors on circleci while upgrading from cimg/aws:2026.09 to cimg/aws:2026.10 which is an upgrade from aws-cli/2.36.36 to aws-cli/2.37.9.

We're using localstack/localstack:4 as the endpoint which does not support TLS by default and have configured AWS with the following credentials:

AWS_DEFAULT_REGION=us-east-1
AWS_SECRET_ACCESS_KEY=thisisalsoforlocalstack
AWS_ACCESS_KEY_ID=thisisforlocalstack
AWS_ENDPOINT_URL=http://localhost:4566

As suggested by #10703, the current workaround is aws configure set s3.preferred_transfer_client classic

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

When AWS_ENDPOINT_URL=http://localhost:4566 is set to a non HTTP endpoint a connection with TLS should not be attempted.

Current Behavior

Here's a sample of the logs for aws --debug s3 cp "${LOCAL_FILE}" "${S3_DESTINATION}":

[DEBUG] [2026-10-12T15:21:14Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data.
[DEBUG] [2026-10-12T15:21:14Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 s_s3_client_process_work_default - Processing any new meta requests.
[DEBUG] [2026-10-12T15:21:14Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 Updating meta requests.
[DEBUG] [2026-10-12T15:21:14Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 Updating connections, assigning requests where possible.
[INFO] [2026-10-12T15:21:14Z] [00007f8aebfff640] [S3ClientStats] - id=0x2c060d20 Requests-in-flight(approx/exact):1/1  Requests-preparing:0  Requests-queued:0  Requests-network(get/put/default/total):0/1/0/1  Requests-streaming-waiting:0  Requests-streaming-response:0  Writes-pending:0 Endpoints(in-table/allocated):1/1
[DEBUG] [2026-10-12T15:21:15Z] [00007f8a2e7fc640] [dns] - static: resolving host localhost
[DEBUG] [2026-10-12T15:21:15Z] [00007f8a2e7fc640] [dns] - static: resolved record: 127.0.0.1
[DEBUG] [2026-10-12T15:21:15Z] [00007f8a2e7fc640] [dns] - static, resolving host localhost successful, returned 1 addresses
[DEBUG] [2026-10-12T15:21:16Z] [00007f8a2e7fc640] [dns] - static: resolving host localhost
[DEBUG] [2026-10-12T15:21:16Z] [00007f8a2e7fc640] [dns] - static: resolved record: 127.0.0.1
[DEBUG] [2026-10-12T15:21:16Z] [00007f8a2e7fc640] [dns] - static, resolving host localhost successful, returned 1 addresses
[DEBUG] [2026-10-12T15:21:17Z] [00007f8a2e7fc640] [dns] - static: resolving host localhost
[DEBUG] [2026-10-12T15:21:17Z] [00007f8a2e7fc640] [dns] - static: resolved record: 127.0.0.1
[DEBUG] [2026-10-12T15:21:17Z] [00007f8a2e7fc640] [dns] - static, resolving host localhost successful, returned 1 addresses
[DEBUG] [2026-10-12T15:21:18Z] [00007f8a2e7fc640] [dns] - static: resolving host localhost
[DEBUG] [2026-10-12T15:21:18Z] [00007f8a2e7fc640] [dns] - static: resolved record: 127.0.0.1
[DEBUG] [2026-10-12T15:21:18Z] [00007f8a2e7fc640] [dns] - static, resolving host localhost successful, returned 1 addresses
[DEBUG] [2026-10-12T15:21:19Z] [00007f8a2e7fc640] [dns] - static: resolving host localhost
[DEBUG] [2026-10-12T15:21:19Z] [00007f8a2e7fc640] [dns] - static: resolved record: 127.0.0.1
[DEBUG] [2026-10-12T15:21:19Z] [00007f8a2e7fc640] [dns] - static, resolving host localhost successful, returned 1 addresses
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 s_s3_client_process_work_default - Processing any new meta requests.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 Updating meta requests.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 Updating connections, assigning requests where possible.
[INFO] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3ClientStats] - id=0x2c060d20 Requests-in-flight(approx/exact):1/1  Requests-preparing:0  Requests-queued:0  Requests-network(get/put/default/total):0/1/0/1  Requests-streaming-waiting:0  Requests-streaming-response:0  Writes-pending:0 Endpoints(in-table/allocated):1/1
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [tls-handler] - id=0x7f8af40127c0: Delayed shut down in write direction
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [channel] - id=0x7f8af4001690: handler 0x7f8af40127c0 shutdown in write dir completed.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [socket] - id=0x7f8af4000e60 fd=114: closing
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [channel] - id=0x7f8af4001690: handler 0x7f8af4012580 shutdown in write dir completed.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [channel-bootstrap] - id=0x2be9d060: channel 0x7f8af4001690 shutdown with error 1029.
[ERROR] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [http-connection] - static: Client connection failed with error 1029 (AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE).
[WARN] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [connection-manager] - id=0x2befbf00: Failed to obtain new connection from http layer, error 1029(TLS (SSL) negotiation failed)
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [connection-manager] - id=0x2befbf00: Failing connection acquisition with error code 1029
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [connection-manager] - id=0x2befbf00: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=1
[WARN] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [connection-manager] - id=0x2befbf00: Failed to complete connection acquisition with error_code 1029(TLS (SSL) negotiation failed)
[ERROR] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [S3Endpoint] - id=0x2bef7f60: Could not acquire connection due to error code 1029 (TLS (SSL) negotiation failed)
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [S3Client] - id=0x2c060d20 Client scheduling retry of request 0x7f8ae4001060 for meta request 0x2bef6e20 with token 0x7f8ae4000f90 with error code 1029 (TLS (SSL) negotiation failed).
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [standard-retry-strategy] - token_id=0x7f8ae4000f90: reducing retry capacity by 10 from 480 and scheduling retry.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [exp-backoff-strategy] - id=0x2c064a60: Attempting retry on token 0x7f8ae4001be0 with error type 0
[WARN] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [exp-backoff-strategy] - id=0x2c064a60: token 0x7f8ae4001be0 has exhausted allowed retries. Retry count 2 max retries 2
[ERROR] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [standard-retry-strategy] - token_id=0x7f8ae4000f90: error occurred while scheduling retry: aws-c-io: AWS_IO_MAX_RETRIES_EXCEEDED, Retry cannot be attempted because the maximum number of retries has been exceeded..
[ERROR] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [S3Client] - id=0x2c060d20 Client could not retry request 0x7f8ae4001060 for meta request 0x2bef6e20 with token 0x7f8ae4000f90 due to error 1069 (Retry cannot be attempted because the maximum number of retries has been exceeded.)
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [channel] - id=0x7f8af4001690: destroying channel.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 s_s3_client_process_work_default - Processing any new meta requests.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 Updating meta requests.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3MetaRequest] - id=0x2bef6e20 Meta request finished with error code 1029 (TLS (SSL) negotiation failed)
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b0e7fc640] [channel-bootstrap] - id=0x2be9d060: releasing bootstrap reference
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3MetaRequest] - id=0x2bef6e20 Cleaning up meta request
upload failed: build/terraform_module.zip to s3://localstack/a/b/b/1.0.0/1.0.0.zip AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE: TLS (SSL) negotiation failed
2026-10-12 15:21:19,427 - Thread-2 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 Updating connections, assigning requests where possible.
[INFO] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3ClientStats] - id=0x2c060d20 Requests-in-flight(approx/exact):0/0  Requests-preparing:0  Requests-queued:0  Requests-network(get/put/default/total):0/0/0/0  Requests-streaming-waiting:0  Requests-streaming-response:0  Writes-pending:0 Endpoints(in-table/allocated):1/1
[DEBUG] [2026-10-12T15:21:19Z] [00007f8b23ffa000] [S3Client] - id=0x2c060d20 Client starting destruction.
[INFO] [2026-10-12T15:21:19Z] [00007f8a2effd640] [event-loop] - id=0x2c06a480: Destroying event_loop
[INFO] [2026-10-12T15:21:19Z] [00007f8a2effd640] [event-loop] - id=0x2c06a480: Stopping event-loop thread.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8a8dffb640] [event-loop] - id=0x2c06a480: exiting main loop
[INFO] [2026-10-12T15:21:19Z] [00007f8a2dffb640] [event-loop] - id=0x2bf11400: Destroying event_loop
[INFO] [2026-10-12T15:21:19Z] [00007f8a2effd640] [event-loop] - id=0x2c069cd0: Destroying event_loop
[INFO] [2026-10-12T15:21:19Z] [00007f8a2dffb640] [event-loop] - id=0x2bf11400: Stopping event-loop thread.
[INFO] [2026-10-12T15:21:19Z] [00007f8a2effd640] [event-loop] - id=0x2c069cd0: Stopping event-loop thread.
[DEBUG] [2026-10-12T15:21:19Z] [00007f8aebfff640] [S3Client] - id=0x2c060d20 s_s3_client_process_work_default - Moving relevant synced_data into threaded_data.
[INFO] [2026-10-12T15:21:19Z] [00007f8aebfff640] [connection-manager] - id=0x2befbf00: release

Reproduction Steps

  1. Create a sample network: docker network create sample-app
  2. Run localstack: docker run --network sample-app --name localstack --rm -it -p 4566:4566 localstack/localstack:4
  3. Run the AWSCLI on a known broken container: docker run --network sample-app --rm -it --env AWS_ENDPOINT_URL=http://localstack:4566 --env AWS_ACCESS_KEY_ID=thisisforlocalstack --env AWS_SECRET_ACCESS_KEY=thisisalsoforlocalstack --env AWS_DEFAULT_REGION=us-east-1 cimg/aws:2026.10
  4. Inside the AWS CLI container, verify localstack works aws sts get-caller-identity, should return valid json
  5. Inside the AWS CLI container, make s3 bucket: aws s3 mb s3://localstack
  6. Inside the AWS CLI container, configure the classic behavior: aws configure set s3.preferred_transfer_client classic
  7. Inside the AWS CLI container, verify working behaivor: echo "" | aws --debug s3 cp - s3://localstack/test
  8. Inside the AWS CLI container, setup the new transfer type: aws configure set s3.preferred_transfer_client crt
  9. Inside the AWS CLI container, verify broken behavior echo "" | aws --debug s3 cp - s3://localstack/test

Possible Solution

Testing connections via HTTP endpoints for local development is a valid use case for every other AWS service using the AWS CLI, the new crt client for s3 should also support plaintext transfers to localstack.

Additional Information/Context

No response

CLI version used

2.37.9

Environment details (OS name and version, etc.)

22.04.5 LTS (Jammy Jellyfish)

Activity

  1. pguinard-public-com
    added
    bugThis issue is a bug.
    needs-triageThis issue or PR still needs to be triaged.
    on Oct 12, 2026
  2. github-actions
    added
    potential-regressionMarking this issue as a potential regression to be checked by team member
    on Oct 12, 2026
  3. ashovlin
    removed
    needs-triageThis issue or PR still needs to be triaged.
    on Oct 13, 2026
  4. hssyoo
    closed this as completedin #10753on Oct 13, 2026

  5. github-actions commented on Oct 13, 2026

    @github-actions
    github-actionsBot
    on Oct 13, 2026 – with GitHub Actions
    Contributor

    This issue is now closed. Comments on closed issues are hard for our team to see.
    If you need more assistance, please open a new issue that references this one.

  6. hssyoo
    reopened this on Oct 13, 2026

  7. hssyoo commented on Oct 13, 2026

    @hssyoo
    hssyoo
    on Oct 13, 2026
    Contributor

    Thanks for the report @pguinard-public-com. We merged a fix via #10753 that was released today in version 2.38.0. The AWS CLI will use the resolved endpoint to determine whether or not TLS should be enabled when creating CRT clients. Upgrading to the latest version should resolve your issue.

  8. hssyoo
    added
    closing-soonThis issue will automatically close in 4 days unless further comments are made.
    and removed
    potential-regressionMarking this issue as a potential regression to be checked by team member
    on Oct 13, 2026

  9. pguinard-public-com commented on Oct 13, 2026

    @pguinard-public-com
    pguinard-public-com
    on Oct 13, 2026
    Author

    Looks great to me! Thank you for the fast turnaround time!

  10. pguinard-public-com
    closed this as completedon Oct 13, 2026

  11. github-actions commented on Oct 13, 2026

    @github-actions
    github-actionsBot
    on Oct 13, 2026 – with GitHub Actions
    Contributor

    This issue is now closed. Comments on closed issues are hard for our team to see.
    If you need more assistance, please open a new issue that references this one.

  12. RyanFitzSimmonsAK
    added
    p1This is a high priority issue
    dependenciesThis issue is a problem in a dependency.
    regression
    and removed
    closing-soonThis issue will automatically close in 4 days unless further comments are made.
    on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.dependenciesThis issue is a problem in a dependency.p1This is a high priority issueregression

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions