OTA-1773: manifests*/0000_90_cluster-update-keys_configmap: New release-repository-*#90
OTA-1773: manifests*/0000_90_cluster-update-keys_configmap: New release-repository-*#90wking wants to merge 2 commits into
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: wking The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
…ory-*
Preparation for Sigstore, as described in the README text I'm adding.
I backed off on some of the signature assertions in the previous
README, but checking some recent releases to see who's signing what,
OCP releases are signed by release key 2 [1]:
$ DIGEST="$(oc image info -o json quay.io/openshift-release-dev/ocp-release:4.20.6-x86_64 | jq -r .digest)"
$ echo "${DIGEST}"
sha256:a29bcbc9f286d68b394ffa0288c5de7e487c90077c06cbaf7a4cadeb0398ce28
$ curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${DIGEST/:/=}/signature-1" | gpg --decrypt | jq .
gpg: Signature made Thu 27 Nov 2025 11:27:17 AM PST
gpg: using RSA key 199E2F91FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
{
"critical": {
"image": {
"docker-manifest-digest": "sha256:a29bcbc9f286d68b394ffa0288c5de7e487c90077c06cbaf7a4cadeb0398ce28"
},
"type": "atomic container signature",
"identity": {
"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.20.6-x86_64"
}
},
"optional": {
"creator": "Red Hat OpenShift Signing Authority 0.0.1"
}
}
$ gpg --list-packets <keys/verifier-public-key-redhat-release | grep 'keyid: .*FD431D51'
keyid: 199E2F91FD431D51
OCP Engineering candidates are signed by beta 2 [2]:
$ DIGEST="$(oc image info -o json quay.io/openshift-release-dev/ocp-release:4.21.0-ec.3-x86_64 | jq -r .digest)"
$ echo "${DIGEST}"
sha256:e3a750749770f4435360852de7c566e370bef20ad90cdc1446e8ac84e4728e49
$ curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${DIGEST/:/=}/signature-1" | gpg --decrypt | jq .
gpg: Signature made Wed 19 Nov 2025 01:45:57 AM PST
gpg: using RSA key 938A80CAF21541EB
gpg: Good signature from "Red Hat, Inc. (beta key 2) <security@redhat.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: B08B 659E E86A F623 BC90 E8DB 938A 80CA F215 41EB
{
"critical": {
"image": {
"docker-manifest-digest": "sha256:e3a750749770f4435360852de7c566e370bef20ad90cdc1446e8ac84e4728e49"
},
"type": "atomic container signature",
"identity": {
"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.21.0-ec.3-x86_64"
}
},
"optional": {
"creator": "Red Hat OpenShift Signing Authority 0.0.1"
}
}
$ gpg --list-packets <keys/verifier-public-key-redhat-beta-2 | grep 'keyid: .*F21541EB'
keyid: 938A80CAF21541EB
OCP nightlies are signed by the CI key [3]:
$ DIGEST="$(oc image info -o json registry.ci.openshift.org/ocp/release:4.21.0-0.nightly-2025-12-09-062745 | jq -r .digest)"
$ echo "${DIGEST}"
sha256:80fa69fc5aaa9ac897df610817f60649d4310c2ab362bef6ed85fbff2203c441
$ curl -s "https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/${DIGEST/:/=}/signature-1" | gpg --decrypt | jq .
gpg: Signature made Mon 08 Dec 2025 10:31:15 PM PST
gpg: using RSA key 6A9588F07F2D3AE7973AD2A9193E53CCD2E2F7A4
gpg: Good signature from "openshift-ci-release-key (This key is used to sign releases from the OpenShift CI system) <brawilli@redhat.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6A95 88F0 7F2D 3AE7 973A D2A9 193E 53CC D2E2 F7A4
{
"critical": {
"type": "atomic container signature",
"image": {
"docker-manifest-digest": "sha256:80fa69fc5aaa9ac897df610817f60649d4310c2ab362bef6ed85fbff2203c441"
},
"identity": {
"docker-reference": "registry.ci.openshift.org/ocp/release:4.21.0-0.nightly-2025-12-09-062745"
}
},
"optional": {
"creator": "openshift release-controller",
"timestamp": 1765261875
}
}
$ gpg --list-packets <keys/verifier-public-key-openshift-ci-4 | grep 'keyid: .*D2E2F7A4'
keyid: 193E53CCD2E2F7A4
That does not seem all that useful, because they don't trust that key:
$ oc adm release extract --to nightly registry.ci.openshift.org/ocp/release:4.21.0-0.nightly-2025-12-09-062745
$ diff -wU3 nightly/0000_90_cluster-update-keys_configmap.yaml manifests.rhel/0000_90_cluster-update-keys_configmap.yaml
...no key difference...
But whatever, nightlies are internal and not customer facing, and they
don't support updates [4]:
Nightly (Internal) ... Red Hat internal use for inter-op and continuous QE testing...
AND OCP CI forces the updates, so it doesn't care if they're signed or not.
CI releases are not signed by anything [5], not sure why not:
$ DIGEST="$(oc image info -o json registry.ci.openshift.org/ocp/release:4.21.0-0.ci-2025-12-09-151854 | jq -r .digest)"
$ echo "${DIGEST}"
sha256:f472f3cf10e72bb2b9d7289093ee63b8d1823184f3fa996f9913afffc9fd64bc
$ curl -s "https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/${DIGEST/:/=}/signature-1" | gpg --decrypt | jq .
gpg: no valid OpenPGP data found.
gpg: decrypt_message failed: Unknown system error
A recent 4-scos-stable release is signed by the CI key [6]:
$ DIGEST="$(oc image info -o json quay.io/okd/scos-release:4.20.0-okd-scos.12 | jq -r .digest)"
$ echo "${DIGEST}"
sha256:9c0435c9552cfa98248532d3e8b0ecd7020ca332ba2a0bba21d2c8f0185d8fb5
$ curl -s "https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/${DIGEST/:/=}/signature-1" | gpg --decrypt | jq .
gpg: Signature made Sun 07 Dec 2025 09:43:21 PM PST
gpg: using RSA key 6A9588F07F2D3AE7973AD2A9193E53CCD2E2F7A4
gpg: Good signature from "openshift-ci-release-key (This key is used to sign releases from the OpenShift CI system) <brawilli@redhat.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6A95 88F0 7F2D 3AE7 973A D2A9 193E 53CC D2E2 F7A4
{
"critical": {
"type": "atomic container signature",
"image": {
"docker-manifest-digest": "sha256:9c0435c9552cfa98248532d3e8b0ecd7020ca332ba2a0bba21d2c8f0185d8fb5"
},
"identity": {
"docker-reference": "quay.io/okd/scos-release:4.20.0-okd-scos.12"
}
},
"optional": {
"creator": "openshift release-controller",
"timestamp": 1765172601
}
}
and they trust that key:
$ oc adm release extract --to okd quay.io/okd/scos-release:4.20.0-okd-scos.12
$ diff -wU3 okd/0000_90_cluster-update-keys_configmap.yaml manifests/0000_90_cluster-update-keys_configmap.yaml
...no key differences...
OKD Engineering Candidates are also signed by the CI key [7]:
$ DIGEST="$(oc image info -o json quay.io/okd/scos-release:4.21.0-okd-scos.ec.13 | jq -r .digest)"
$ echo "${DIGEST}"
sha256:a303ecbfab19d1d98d2258efde3b43cac4c308bedff49e3c10a27314b70dfe37
$ curl -s "https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/${DIGEST/:/=}/signature-1" | gpg --decrypt | jq .
gpg: Signature made Sun 07 Dec 2025 09:15:46 PM PST
gpg: using RSA key 6A9588F07F2D3AE7973AD2A9193E53CCD2E2F7A4
gpg: Good signature from "openshift-ci-release-key (This key is used to sign releases from the OpenShift CI system) <brawilli@redhat.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6A95 88F0 7F2D 3AE7 973A D2A9 193E 53CC D2E2 F7A4
{
"critical": {
"type": "atomic container signature",
"image": {
"docker-manifest-digest": "sha256:a303ecbfab19d1d98d2258efde3b43cac4c308bedff49e3c10a27314b70dfe37"
},
"identity": {
"docker-reference": "quay.io/okd/scos-release:4.21.0-okd-scos.ec.13"
}
},
"optional": {
"creator": "openshift release-controller",
"timestamp": 1765170946
}
}
[1]: https://amd64.ocp.releases.ci.openshift.org/#4-stable
[2]: https://amd64.ocp.releases.ci.openshift.org/#4-dev-preview
[3]: https://amd64.ocp.releases.ci.openshift.org/#4.21.0-0.nightly
[4]: https://mirror.openshift.com/pub/openshift-v4/OpenShift_Release_Types.pdf
[5]: https://amd64.ocp.releases.ci.openshift.org/#4.21.0-0.ci
[6]: https://amd64.origin.releases.ci.openshift.org/#4-scos-stable
[7]: https://amd64.origin.releases.ci.openshift.org/#4-scos-next
…'namespace' Cleaning up after 9af1fc2 (Add IBM Cloud managed annotations to CVO manifests, 2020-11-11, openshift#24). We only need to set the namespace key once.
wking
force-pushed
the
release-repository
branch
from
5e434bd to
a1cba4a
Compare
|
@wking: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
mtrmac
left a comment
mtrmac
left a comment
There was a problem hiding this comment.
On the principle of the thing: From an earlier discussion I understand
- We can extract / authenticate the version number from the release image of the update target (or from a verified signature of the update target?), in a way the update graph can’t affect
- CVO will refuse to downgrade to earlier (even if correctly signed) versions
LGTM assuming the above.
On form:
- If the thing we are trying to protect against is the update graph pointing at out-of-product images, that might be worth noting here.
- I’m not sure about the authentication/authorization phrasing of the concepts (I tend to think it applies to principals, not to objects), but it works well enough and I don’t have a good short replacement for the latter one.
|
ACK I agree we can keep image pullspec lockdown separate from signature verification. ClusterImagePolicy can provide scoped verification at the digest level that the CVO can consume and trust. |
Yes, that happens here, for users who set ClusterVersion's
Yes, that happens here, with a narrow carve-out for rolling back to the previous target version, if that previous target version is in the same z-stream as the current target version.
The update-service might have been the one recommending the update, but it's also possible that the cluster admin used a different pipeline, including passing around pullspecs via word-of-mouth, or whatever. This new guard is trying to stiffen cluster protection for all sources of next-hop update advice, under the assumption that no tooling in that space will ever be so reliable that we can trust it without guard-rails. |
wking
changed the title
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
openshift-ci-robot
added
the
jira/valid-reference
|
@wking: This pull request references OTA-1773 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Thanks for the references, LGTM. (I’ll leave formal / tool-visible approvals to specialist maintainers of this repo.) |
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
Bot
added
the
lifecycle/stale
|
Important Review skippedAuto reviews are limited based on label configuration. 🚫 Review skipped — only excluded labels are configured. (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
5 participants
Preparation for Sigstore, as described in the README text I'm adding:
Also some other modernization of README text; details in the commit messages :)