Skip to content

Commit fc0c006

Browse files
mpdudempdude
authored
Mitigate race condition (#3)
Try to ensure in-order execution when different workflows race for deployments.
1 parent 62d30ec commit fc0c006

File tree

5 files changed

+88
-5
lines changed

5 files changed

+88
-5
lines changed

‎README.md

+11
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,7 @@ The IAM User that is used to run the action requires the following IAM permissio
134134
"codedeploy:CreateDeployment",
135135
"codedeploy:RegisterApplicationRevision",
136136
"codedeploy:GetDeploymentConfig",
137+
"codedeploy:GetDeploymentGroup",
137138
"codedeploy:UpdateDeploymentGroup",
138139
"codedeploy:CreateDeploymentGroup"
139140
],
@@ -148,6 +149,16 @@ The IAM User that is used to run the action requires the following IAM permissio
148149
}
149150
```
150151

152+
## Race Conditions
153+
154+
As of writing, the AWS CodeDeploy API does not accept new deployment requests for an application and deployment group as long as another deployment is still in progress. So, this action will retry a few times and eventually (hopefully) succeed.
155+
156+
There might be situations where several workflow runs are triggered in quick succession - for example, when merging several approved pull requests in a short time. Since your test suites or workflow runs might take a varying amount of time to finish and to reach the deployment phase (_this_ action), you cannot be sure that the triggered deployments will happen in the order you merged the pull requests (to stick with the example). You could not even be sure that the last deployment made was based on the last commit in your repository.
157+
158+
To work around this, this action includes the GitHub Actions "[run id](https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context)" in the `description` field for created deployments. Before creating a new deployment, it will fetch the _last attempted deployment_ from the AWS API and compare its run id with the current run. If the current run has a _lower_ id than the last attempted deployment, the deployment will be aborted.
159+
160+
This workaround should catch a good share of possible out-of-order deployments. There is a slight chance for mishaps, however: If a _newer_ deployment happens to start _after_ we checked the run id and finishes _before_ we commence our own deployment (just a few lines of code later), this might go unnoticed. To really prevent this from happening, ordering deployments probably needs to be supported on the AWS API side, see https://github.com/aws/aws-codedeploy-agent/issues/248.
161+
151162
## Action Input and Output Parameters
152163

153164
### Input

‎cli.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@
7979

8080
const action = require('./create-deployment');
8181
try {
82-
await action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
82+
await action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, null, core);
8383
} catch (e) {
8484
console.log(`👉🏻 ${e.message}`);
8585
process.exit(1);

‎create-deployment.js

+35-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ function fetchBranchConfig(branchName) {
2323
process.exit();
2424
}
2525

26-
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, core) {
26+
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, runNumber, core) {
2727
const branchConfig = fetchBranchConfig(branchName);
2828
const safeBranchName = branchName.replace(/[^a-z0-9-/]+/gi, '-').replace(/\/+/, '--');
2929
const deploymentGroupName = branchConfig.deploymentGroupName ? branchConfig.deploymentGroupName.replace('$BRANCH', safeBranchName) : safeBranchName;
@@ -63,19 +63,53 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
6363
}
6464

6565
let tries = 0;
66+
const description = runNumber ? `Created by webfactory/create-aws-codedeploy-deployment (run_number=${runNumber})` : '';
67+
6668
while (true) {
6769

6870
if (++tries > 5) {
6971
core.setFailed('🤥 Unable to create a new deployment (too much concurrency?)');
7072
return;
7173
}
7274

75+
if (runNumber) {
76+
var {deploymentGroupInfo: {lastAttemptedDeployment: {deploymentId: lastAttemptedDeploymentId}}} = await codeDeploy.getDeploymentGroup({
77+
applicationName: applicationName,
78+
deploymentGroupName: deploymentGroupName,
79+
}).promise();
80+
81+
var {deploymentInfo: {description: lastAttemptedDeploymentDescription}} = await codeDeploy.getDeployment({
82+
deploymentId: lastAttemptedDeploymentId,
83+
}).promise();
84+
85+
var matches, lastAttemptedDeploymentRunNumber;
86+
87+
if (matches = lastAttemptedDeploymentDescription.match(/run_number=(\d+)/)) {
88+
lastAttemptedDeploymentRunNumber = matches[1];
89+
if (parseInt(lastAttemptedDeploymentRunNumber) > parseInt(runNumber)) {
90+
core.setFailed(`🙅‍♂️ The last attempted deployment as returned by the AWS API has been created by a higher run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber}. Aborting.`);
91+
return;
92+
} else {
93+
console.log(`🔎 Last attempted deployment was from run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber} - proceeding.`);
94+
}
95+
}
96+
97+
/*
98+
There's a slight remaining chance that the above check does not suffice: If we just
99+
passed the check, but another (newer) build creates AND finishes a deployment
100+
BEFORE we reach the next lines, an out-of-order deployment might happen. This is a
101+
race condition that requires an extension on the AWS API side in order to be resolved,
102+
see https://github.com/aws/aws-codedeploy-agent/issues/248.
103+
*/
104+
}
105+
73106
try {
74107
var {deploymentId: deploymentId} = await codeDeploy.createDeployment({
75108
...deploymentConfig,
76109
...{
77110
applicationName: applicationName,
78111
deploymentGroupName: deploymentGroupName,
112+
description: description,
79113
revision: {
80114
revisionType: 'GitHub',
81115
gitHubLocation: {

‎dist/index.js

+38-2
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ function fetchBranchConfig(branchName) {
3939
process.exit();
4040
}
4141

42-
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, core) {
42+
exports.createDeployment = async function(applicationName, fullRepositoryName, branchName, commitId, runNumber, core) {
4343
const branchConfig = fetchBranchConfig(branchName);
4444
const safeBranchName = branchName.replace(/[^a-z0-9-/]+/gi, '-').replace(/\/+/, '--');
4545
const deploymentGroupName = branchConfig.deploymentGroupName ? branchConfig.deploymentGroupName.replace('$BRANCH', safeBranchName) : safeBranchName;
@@ -79,19 +79,53 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
7979
}
8080

8181
let tries = 0;
82+
const description = runNumber ? `Created by webfactory/create-aws-codedeploy-deployment (run_number=${runNumber})` : '';
83+
8284
while (true) {
8385

8486
if (++tries > 5) {
8587
core.setFailed('🤥 Unable to create a new deployment (too much concurrency?)');
8688
return;
8789
}
8890

91+
if (runNumber) {
92+
var {deploymentGroupInfo: {lastAttemptedDeployment: {deploymentId: lastAttemptedDeploymentId}}} = await codeDeploy.getDeploymentGroup({
93+
applicationName: applicationName,
94+
deploymentGroupName: deploymentGroupName,
95+
}).promise();
96+
97+
var {deploymentInfo: {description: lastAttemptedDeploymentDescription}} = await codeDeploy.getDeployment({
98+
deploymentId: lastAttemptedDeploymentId,
99+
}).promise();
100+
101+
var matches, lastAttemptedDeploymentRunNumber;
102+
103+
if (matches = lastAttemptedDeploymentDescription.match(/run_number=(\d+)/)) {
104+
lastAttemptedDeploymentRunNumber = matches[1];
105+
if (parseInt(lastAttemptedDeploymentRunNumber) > parseInt(runNumber)) {
106+
core.setFailed(`🙅‍♂️ The last attempted deployment as returned by the AWS API has been created by a higher run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber}. Aborting.`);
107+
return;
108+
} else {
109+
console.log(`🔎 Last attempted deployment was from run number ${lastAttemptedDeploymentRunNumber}, this is run number ${runNumber} - proceeding.`);
110+
}
111+
}
112+
113+
/*
114+
There's a slight remaining chance that the above check does not suffice: If we just
115+
passed the check, but another (newer) build creates AND finishes a deployment
116+
BEFORE we reach the next lines, an out-of-order deployment might happen. This is a
117+
race condition that requires an extension on the AWS API side in order to be resolved,
118+
see https://github.com/aws/aws-codedeploy-agent/issues/248.
119+
*/
120+
}
121+
89122
try {
90123
var {deploymentId: deploymentId} = await codeDeploy.createDeployment({
91124
...deploymentConfig,
92125
...{
93126
applicationName: applicationName,
94127
deploymentGroupName: deploymentGroupName,
128+
description: description,
95129
revision: {
96130
revisionType: 'GitHub',
97131
gitHubLocation: {
@@ -157,8 +191,10 @@ exports.createDeployment = async function(applicationName, fullRepositoryName, b
157191
const branchName = isPullRequest ? payload.pull_request.head.ref : payload.ref.replace(/^refs\/heads\//, ''); // like "my/branch_name"
158192
console.log(`🎋 On branch '${branchName}', head commit ${commitId}`);
159193

194+
const runNumber = process.env['github_run_number'] || process.env['GITHUB_RUN_NUMBER'];
195+
160196
try {
161-
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
197+
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, runNumber, core);
162198
} catch (e) {}
163199
})();
164200

‎index.js

+3-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,9 @@
1414
const branchName = isPullRequest ? payload.pull_request.head.ref : payload.ref.replace(/^refs\/heads\//, ''); // like "my/branch_name"
1515
console.log(`🎋 On branch '${branchName}', head commit ${commitId}`);
1616

17+
const runNumber = process.env['github_run_number'] || process.env['GITHUB_RUN_NUMBER'];
18+
1719
try {
18-
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, core);
20+
action.createDeployment(applicationName, fullRepositoryName, branchName, commitId, runNumber, core);
1921
} catch (e) {}
2022
})();

0 commit comments

Comments
 (0)