credential_process doesn't work for federated roles

Confirm by changing [ ] to [x] below to ensure that it's a bug:

[ x ] I've gone though the User Guide and the API reference
[ x ] I've searched for previous similar issues and didn't find any solution

Describe the bug
It's impossible to use credential_process with federated credentials without manually editing the credentials file every time I want to use the cli.

I use federated roles. Federated roles require constant updating of STS tokens. I have profile [profile user] and federated creds under [user]. I have an internal tool which updates creds for me. It's also a daemon so you run tool& and it will refresh them for you.

If I completely delete [user] in .aws/credentials, the credential_process will execute because the CLI looks for creds in this order:

2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2020-04-03 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file

If it finds no creds, it will run credential_process. But if it DOES, and those creds are invalid, it just fails and says "invalid creds" instead of running credential_process again.

SDK version number
aws-cli/1.18.36 botocore/1.15.36

Platform/OS/Hardware/Device
What are you running the cli on?
Python/3.7.5 Darwin/18.7.0

To Reproduce (observed behavior)

Get temp federated credentials
run cli command, eg aws s3 ls
invalidate the session token by changing some letters
run cli command, eg aws s3 ls

It will fail instead of re-running the credential_process.

Expected behavior
CLI should re-run the credential_process if session tokens are expired or invalid

Logs/output
Get full traceback and error logs by adding --debug to the command.

...
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2020-04-03 14:22:24,020 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2020-04-03 14:22:24,021 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
...
2020-04-03 14:22:24,149 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '6E9775F8FE088D43', 'x-amz-id-2': '123456789', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 03 Apr 2020 21:22:23 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}
2020-04-03 14:22:24,149 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidToken</Code><Message>The provided token is malformed or otherwise invalid.</Message><Token-0>IQo...asdfasdfasdfasdf....UE=</Token-0><RequestId>123456789</RequestId><HostId>123456789</HostId></Error>'
2020-04-03 14:22:24,150 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <botocore.retryhandler.RetryHandler object at 0x109e2e410>
2020-04-03 14:22:24,150 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2020-04-03 14:22:24,150 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x109e2e110>>
2020-04-03 14:22:24,150 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListBuckets: calling handler <function enhance_error_msg at 0x10b418200>
2020-04-03 14:22:24,152 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/clidriver.py", line 217, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/commands.py", line 190, in __call__
    parsed_globals)
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/s3/subcommands.py", line 478, in _run_main
    self._list_all_buckets()
  File "/user/Library/Python/3.7/lib/python/site-packages/awscli/customizations/s3/subcommands.py", line 543, in _list_all_buckets
    response_data = self.client.list_buckets()
  File "/user/Library/Python/3.7/lib/python/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/user/Library/Python/3.7/lib/python/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.
2020-04-03 14:22:24,153 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.

Additional context
Add any other context about the problem here.

hi @clhefton ,
thanks for pointing this out, I'll need a bit more help from you because it's hard for me to test this but, could you try adding the profile name and using a region dependent service like:
aws dynamodb list-tables --profile user

I'm expecting this to have the same behavior, but want to be sure before I go further into this.

I do get the same error.

Also, I have verified that the script I have for "credential_process" (/usr/local/bin/getcreds) does work if run on its own outside the context of the CLI. So the only issue is re-running it upon finding expired tokens

What is the format of the getcreds output? The process creds expect an expiry time, if the creds are present and not expired the CLI will assume they are valid and try to use them. You didn't mention that they were expired or not, just that they were invalid, in which case the CLI behavior seems correct to me unless I am misunderstanding the scenario.

Can you provide a full sanitized version of your config file with the profiles you metnioned, as well as provide the output format of the process called by the process provider.

IMO, it's not "correct" if it doesn't find new creds for you if the ones you have don't work. That's what credential_process is for. But I tested that it doesn't work on legitimately expired creds either; here are some screenshots. It shows last written date to ~/.aws/credentials yesterday, shows creds existing under my profile, and shows the "expired token" error when running a command.

As for the getcreds question, it does the following: starts a daemon that updates the ~/.aws/credentials file, sleep(5) to give the daemon time to update the first time, pull the new creds from credentials, output them again in correct JSON format so credential_process doesn't error out.

I.e.,

[profile]
credential_helper=/bin/getcreds
    -> call getcreds, which:
        ->starts my_daemon
        ->reads updated credentials file
        ->outputs correct json based on values above
    -> reads JSON from get_creds (or should)

But it never reaches step 1, which is calling getcreds

Hi, any progress on this issue as I'm getting exactly the same issue using gsts? I need to delete the credentials from the credential file before it updates them. Thanks

So this is by design and an issue with saml2aws? Solution is to change where the aws credentials are stored by saml2aws like this? (will confirm)

credential_process = saml2aws login -a saml --skip-prompt --quiet --credential-process --role arn:aws:iam::01234555654:role/my-role-read --region us-east-1 --profile dev --credentials-file=~/.aws/saml2aws.creds

I'm of the opinion that this is a bug. The documentation for credential_process specify that if an expiration is provided the credentials are treated as temporary. But then when they expire, the cli forgets that they are temporary and doesnt call the credential_process to fetch more. It treats them as long lasting credentials.

The CLI follows the credential provider chain in order. The two relevant providers here are:

Credentials file (~/.aws/credentials)
Process provider (credential_process)

The process provider is expected to return credentials to stdout. In this issue, it looks like a tool is being used and it does not conform to stdout behavior. It also seems like the tool also writes credentials to the credentials file. The next time the CLI goes through the chain, it finds those credentials in the file and tries to use them instead of reaching the process provider. For the process provider to be invoked, credentials should not be present in the credentials file for that profile.

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

Navigation Menu

Repository navigation

credential_process doesn't work for federated roles #5109

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

credential_process doesn't work for federated roles #5109

Description

Activity

KaibaLopez commented on Apr 6, 2020

m3ldis commented on Apr 6, 2020

stealthycoin commented on Apr 8, 2020

m3ldis commented on Apr 15, 2020

m3ldis commented on Apr 15, 2020

jamessoubry commented on Mar 8, 2021

8 remaining items

BEllis commented on Oct 13, 2022

lumattr commented on Dec 11, 2024

adev-code commented on Apr 13, 2026

github-actions commented on May 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions