Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ecs): Implementing backoff and retry for registerScalableTarget request #6215

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

christosarvanitis
Copy link
Member

According to AWS docs:

If you call the RegisterScalableTarget API operation to create a scalable target, there might be a brief delay until the operation achieves eventual consistency . You might become aware of this brief delay if you get unexpected errors when performing sequential operations. The typical strategy is to retry the request, and some Amazon Web Services SDKs include automatic backoff and retry logic.

The backoff and retry logic was implemented for the CreateServerGroupAtomicOperation but it was not for the other operations that heavily depend on the RegisterScalableTarget API request. We have seen random failed operations that fail with errors like:

Orchestration failed: DisableServiceAtomicOperation | ConcurrentUpdateException: [You already have a pending update to an Auto Scaling resource. (Service: AWSApplicationAutoScaling; Status Code: 400; Error Code: ConcurrentUpdateException; Request ID: uuid Proxy: null)]"
Although no action was pending to the ASGs.

This PR is following the same logic as the CreateServerGroupAtomicOperation for the rest of the AtomicOperations that involve RegisterScalableTarget API calls to AWS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant