Repository navigation
ci: fix the deploy-beta SSM parameter collision and retry tool downloads - #843
Merged
Merged
Conversation
Every merge to main has failed at deploy-beta since the v1.1.0 revert
(2026-07-10) with:
Resource of type 'AWS::SSM::Parameter' with identifier
'/lambda-web-adapter/layer/x86_64/1.0.1' already exists. (AlreadyExists)
The parameter's Name embedded ${CargoPkgVersion}, so a version change forces
CloudFormation to replace the resource, and UpdateReplacePolicy: Retain left
the old name behind un-managed. Releasing 1.1.0 orphaned /x86_64/1.0.1;
reverting to 1.0.1 then tried to create that exact name and hit the orphan.
Rollback restores the 1.1.0 name, so the next deploy repeats it — a permanent
wedge, not a flake. It also took e2e-test-zip and e2e-test-oci down with it
(both `needs: deploy-beta`), so main had no e2e coverage for two months.
Removing the resource is the fix rather than dropping UpdateReplacePolicy:
with only the current version ever tracked, the per-version parameter holds
the same value as the /latest parameter beside it, so it earns nothing. The
alternative — writing it with `aws ssm put-parameter --overwrite` from the
workflow — needs ssm:PutParameter added to the PipelineExecutionRole in every
beta, gamma, prod, and China account, which today grants only ssm:GetParameters
on /lambda-web-adapter/e2e/*.
Nothing reads a per-version parameter: the only consumer of either parameter is
tests/e2e_tests/fixtures/go-httpbin-zip/template.yaml, which reads
/lambda-web-adapter/layer/x86_64/latest. That one keeps its static name, so it
updates in place and can never collide.
No parameter is deleted by this change. DeletionPolicy: Retain applies when a
resource is removed from the template, so the currently-tracked names are kept
as the historical pointers, and the layer Description still records the version
for `aws lambda list-layer-versions`.
e2e-test-zip failed on the first green deploy-beta in two months, six steps before it ran anything: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ... curl: (35) Recv failure: Connection reset by peer tar: Error is not recoverable: exiting now Two problems. There was no retry, so one reset from get.nexte.st failed the job. And piping into tar discarded curl's exit status — the default shell for `run` is `bash -e`, without pipefail — so the diagnosis surfaced as a tar error rather than the network failure it was. Both downloads now retry and land in a file before extraction: cargo-nextest in all five places it is installed (merge test, merge e2e-test-zip, merge e2e-test-oci, pr, release) and mdBook in the docs workflow, which has the same pattern and would have been the next one to flake.
![aws-sam-tooling-bot[bot]](https://avatars.githubusercontent.com/b/1290?s=60&v=4){kind=small}
The mdBook step kept `-sSL` while the cargo-nextest steps use `-LsSf`, so it still had the failure mode this branch set out to remove. Without `--fail`, curl treats an HTTP error as a successful transfer: it writes the error page to /tmp/mdbook.tar.gz and exits 0, and `tar` then fails with "not in gzip format" instead of reporting the 403 or 404 that actually happened. `--retry` cannot help either, because curl never classifies the response as an error to retry. Reproduced against a non-existent release asset: without -f, curl exits 0 having written a 9-byte "Not Found" body and tar exits 2 with no usable message; with -f, curl exits 22 and says "The requested URL returned error: 404". The real asset still downloads and extracts (mdbook v0.4.40). Reported by aws-sam-tooling-bot on #843.
vicheey
approved these changes
Sep 14, 2026
bnusunny
added a commit
that referenced
this pull request
Sep 18, 2026
* ci: publish the version-named layer parameter from the pipeline The second half of #843. That change removed the version-named SSM pointer from the layer stacks, because a parameter whose Name contains the package version is replaced when the version changes and — retained — collides with its own orphan when a version is re-used, which wedged every merge to main for two months. The pointer itself is still wanted, so it is now written after deployment with `aws ssm put-parameter --overwrite`, which is idempotent and cannot collide. Two scripts: publish-layer-parameter.sh reads the layer ARN from the stack's own output and writes /lambda-web-adapter/layer/<arch>/<version>. It refuses a version containing a slash before making any AWS call, since such a name would fall outside the granted path and could address a `latest` pointer. configure-layer-parameter-iam.sh grants the PipelineExecutionRole ssm:PutParameter on that path. Run once per role, with administrator credentials in the role's account. This is a prerequisite: SAM-managed pipeline roles do not have the permission, and until the bootstrap is run the publish steps fail with AccessDenied. 69 roles are in scope — 32 gamma, 32 prod, 2 China gamma, 2 China prod, 1 beta, all distinct. Three things about the grant are deliberate: * Region-agnostic. `put-role-policy` replaces the whole named policy, so a per-region document meant a second invocation for another region silently revoked the first. The Resource wildcards the region instead, so one run per role covers every region and re-running is idempotent. * No region argument. With the region wildcarded there is nothing for the partition parsed from the role ARN to disagree with; an `aws` role paired with `cn-north-1` previously produced a policy matching nothing while reporting success. * An explicit Deny on `/latest`. The Allow has to wildcard the version segment, which also covers the `latest` pointer CloudFormation owns — and which the e2e fixture resolves at deploy time. Without the Deny, a mistyped version could overwrite stack-managed state out of band, where nobody would look for drift. In the workflows, both architectures deploy before either pointer is written. Ordered the other way — publish x86, deploy arm64 — a missed bootstrap left the region with a new x86 layer and no arm64 one, failed deploy-prod, skipped publish-to-public-ecr, and stopped the release after prod had been partially mutated. Ordered this way the same failure leaves both layers published and the pointers stale. Each stack name is now defined once per job and consumed by both the deploy and the publish step, rather than written twice in ten places with nothing asserting the two agree. Exercised: the publish script's argument, architecture and slash-in-version guards; the IAM script's ARN validation against a non-ARN, a short account id, a user ARN and a nested role path; the rendered policy document; and both workflows parse. * ci: take the SSM grant from the infrastructure package, not a bootstrap script The PipelineExecutionRole is managed in LambdaWebAdapterInfrastructureCDK, so the grant belongs there rather than in an imperative script run by hand against 69 roles. lib/infra-template.yaml now carries it in PipelineExecutionRolePermissions, allowing ssm:PutParameter on parameter/lambda-web-adapter/layer/{x86_64,arm64}/* and denying .../latest, which CloudFormation owns. Deletes .github/scripts/configure-layer-parameter-iam.sh. Keeping it would have been worse than redundant: it wrote a separate inline policy under its own name, so the two would have coexisted on every role with no indication which was authoritative, and its region-wildcarded Resource was broader than the per-region scope the infra template can express — one pipeline-resources stack exists per region and its role only deploys in that region. The workflow comments and the publish script's header now say where the permission comes from, so the AccessDenied case points at a region whose pipeline-resources stack has not been updated rather than at a script nobody ran.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two failures seen on the last merge to main: a permanent
deploy-betawedge, and a flaky tool download that tooke2e-test-zipwith it.1.
deploy-betahas been wedged since the v1.1.0 revertEvery merge to main has failed since 2026-07-10 (run 29062863936, run 33441923283) with:
The parameter's
Nameembedded${CargoPkgVersion}, so changing the version forces CloudFormation to replace the resource, andUpdateReplacePolicy: Retainleft the old name behind un-managed:87f0c87) created/x86_64/1.1.0and orphaned/x86_64/1.0.1.3fc665d) tried to create that exact name → collided with the orphan.e2e-test-zipande2e-test-ociareneeds: [deploy-beta], so main had no e2e coverage for two months.Why removal rather than a different policy
UpdateReplacePolicy: Retainwould fix the collision, but then only the current version is ever tracked — and that parameter holds the identical value to the/latestparameter beside it. It earns nothing.aws ssm put-parameter --overwritefrom the workflow needsssm:PutParameteron thePipelineExecutionRolein every beta/gamma/prod/China account. Today that role grants onlyssm:GetParameterson/lambda-web-adapter/e2e/*, so the step would fail closed on first run.tests/e2e_tests/fixtures/go-httpbin-zip/template.yaml, which reads/lambda-web-adapter/layer/x86_64/latest. That name is static, so it updates in place and can never collide.No parameter is deleted by merging this.
DeletionPolicy: Retainapplies when a resource is removed from the template, so the currently tracked names are kept as historical pointers. The layerDescriptionstill records the version, soaws lambda list-layer-versionsmaps a version to its layer ARN.2. Tool downloads had no retry, and hid their own failure
On the first green
deploy-betain two months,e2e-test-zipdied at step 6 of 12 — beforesetup-sam, before assuming any role, before deploying or testing anything:Two problems: no retry, so a single reset from
get.nexte.stfails the job; and piping intotardiscards curl's exit status, because the default shell forrunisbash -ewithoutpipefail— so a network failure was reported as a tar error.Both downloads now retry and land in a file before extraction:
cargo-nextestin all five install sites (mergetest, mergee2e-test-zip, mergee2e-test-oci,pr,release) and mdBook indocs.yaml, which had the same pattern and would have been next to flake.Remediation already applied
The two orphaned parameters were deleted by hand in the beta account, which unwedged
deploy-betaon run 34767327920 (e2e-test-zippassed on re-run). Current beta state: all fourlambda-adapter-beta-*stacksUPDATE_COMPLETE,/x86_64/1.0.1re-created under stack management at layer:273,/latestadvanced to:273so it no longer advertises the backed-out v1.1.0 build.Gamma/prod/China accounts still hold the same class of orphan from earlier releases. They are harmless while version numbers are never reused, and this change stops new ones accruing — but a re-release of an existing version there would need the same one-time cleanup.
Validation
sam validate --lintclean on both templates.cargo-nextest 0.9.144.CargoPkgVersionis still referenced (the layerDescription), so the parameter is not orphaned in the template.