Digest immutability still broken – multiple tags continuing to change over time

[aydosman]

Bug Overview

Previous issue #265

I've been actively monitoring this issue using a digest tracking script across the relevant tags on quay.io/nginx/nginx-unprivileged and unfortunately I can confirm that digest immutability does not appear to be resolved.

For example, here are just a few tags where the digest has changed:

1.26.3

• 2025-04-02: sha256:ca5f66c1769ebd9658e32e70df77d347202635d1a05da41f4e68d7f7e0e3ca50
• 2025-04-07: sha256:e3bbd65240a9eafbb86c814a523cf3dec78d6a5be663f4fe8b5371a7b73fd4be
• 2025-04-14: sha256:4d35de56fa9cf7343979cb021a8750d691300eeb22fcd09fbbb97b35eb50cab5
• 2025-04-21: sha256:d8e7e962cf7cf13cbca50643c3cd3c0870d1d214d35a14c1d83e5857fc622839
• 2025-04-22: sha256:cd83270b8e0019ee7687198a473842b45c1ef31126229d511afd96c8bba625bc

1.27.4

• 2025-04-02: sha256:7f5f11aecd21f0f95267396b8e0fb839312368fdaa51b15199d28f03d91ccdc3
• 2025-04-07: sha256:8df8fee6cabcd54c545a8a28a41d04a80cdcedb28c30c068e512f5a56aee4de4
• 2025-04-14: sha256:218e25d58d22541b93613a61a04bf6e7d0e337b3384b6f890e83b9a1027480d1
• 2025-04-21: sha256:663d1f83b634d625f3c36b323b21cc5c2c4ed5ebf912a01bfe13cce2f455e92e

1.27.5

• 2025-04-26: sha256:1c1d30bc587c2351abb5457f3bbb2567074738cffafccf0603c7934cca574b1f
• 2025-04-28: sha256:666939c941fd57db2924734da8e66a75af4f660ed10cedc0fca992cb3ddf088d
• 2025-04-28: sha256:965bba109c9dd2ad90342f80653d94768e2d8b814551a4e5c9a1130f7992ee4b

1.28.0

• 2025-04-26: sha256:3e5f030818c3782a35b6f621458a21f3e526a35267b2b4505d225684d5eac7c4
• 2025-04-28: sha256:02776f71690c1a764da6fdb104ebf793d8f0064d6c2854dfce5289e301683edf
• 2025-04-28: sha256:aa538e1dc81068827c28ad5855bbd721f0f17a3d303f4b5d6737ddc219d1c8c6

Thanks again for your time and efforts, but could some one take another look at this?

Full tracking

1.25.5:
  2025-04-02T15:17:35Z -> sha256:77e4b763b46ed8be6da1e8cc6386e3965411ea8e86f1c8f254a868d66657bddc

1.26.0:
  2025-04-02T15:17:35Z -> sha256:2f9cdab1fb340a6c1908366654c4d6b19d58286a46635eb034a53fc61b44c34d

1.26.1:
  2025-04-02T15:17:35Z -> sha256:e5b7447e9937ac788d592fa65430936005f344591c6ff9ca073a1d3cb8ca82ab

1.26.2:
  2025-04-02T15:17:35Z -> sha256:3c216da7d3aa0384782e1199a7b30c8d3bc4b0288dcac32c178b0c0fb26f8059

1.26.3:
  2025-04-02T15:17:35Z -> sha256:ca5f66c1769ebd9658e32e70df77d347202635d1a05da41f4e68d7f7e0e3ca50
  2025-04-07T12:08:30Z -> sha256:e3bbd65240a9eafbb86c814a523cf3dec78d6a5be663f4fe8b5371a7b73fd4be
  2025-04-14T12:08:48Z -> sha256:4d35de56fa9cf7343979cb021a8750d691300eeb22fcd09fbbb97b35eb50cab5
  2025-04-21T12:08:13Z -> sha256:d8e7e962cf7cf13cbca50643c3cd3c0870d1d214d35a14c1d83e5857fc622839
  2025-04-22T12:08:35Z -> sha256:cd83270b8e0019ee7687198a473842b45c1ef31126229d511afd96c8bba625bc

1.27.0:
  2025-04-02T15:17:35Z -> sha256:e0b90e4ae842abc9b3eecba9344b0f7c11276346e548659a500ecea18771687c

1.27.1:
  2025-04-02T15:17:35Z -> sha256:144210beee75048a0db053eada2e4555fe70724a122a4049aac9acb8bb1de9bf

1.27.2:
  2025-04-02T15:17:35Z -> sha256:890b8283159ff3eec13ee426a2cd79d5a7d7a300457074970ea14c75865dc39c

1.27.3:
  2025-04-02T15:17:35Z -> sha256:6d51e4a8e10dfe334f8e2d15bb81b1ed2580ea9cb874b644acc720eda7022b54

1.27.4:
  2025-04-02T15:17:35Z -> sha256:7f5f11aecd21f0f95267396b8e0fb839312368fdaa51b15199d28f03d91ccdc3
  2025-04-07T12:08:30Z -> sha256:8df8fee6cabcd54c545a8a28a41d04a80cdcedb28c30c068e512f5a56aee4de4
  2025-04-14T12:08:48Z -> sha256:218e25d58d22541b93613a61a04bf6e7d0e337b3384b6f890e83b9a1027480d1
  2025-04-21T12:08:13Z -> sha256:663d1f83b634d625f3c36b323b21cc5c2c4ed5ebf912a01bfe13cce2f455e92e

1.27.5:
  2025-04-26T00:22:01Z -> sha256:1c1d30bc587c2351abb5457f3bbb2567074738cffafccf0603c7934cca574b1f
  2025-04-28T00:23:54Z -> sha256:666939c941fd57db2924734da8e66a75af4f660ed10cedc0fca992cb3ddf088d
  2025-04-28T12:08:37Z -> sha256:965bba109c9dd2ad90342f80653d94768e2d8b814551a4e5c9a1130f7992ee4b

1.28.0:
  2025-04-26T00:22:01Z -> sha256:3e5f030818c3782a35b6f621458a21f3e526a35267b2b4505d225684d5eac7c4
  2025-04-28T00:23:54Z -> sha256:02776f71690c1a764da6fdb104ebf793d8f0064d6c2854dfce5289e301683edf
  2025-04-28T12:08:37Z -> sha256:aa538e1dc81068827c28ad5855bbd721f0f17a3d303f4b5d6737ddc219d1c8c6

[alessfg]

This is a fundamental issue with Quay. It has a built in garbage collection system which removes tags when a new tag takes its place. I would suggest using any of the other three supported registries if you want to pin to a specific SHA. Creating more granular tags is also not an option at this moment for various reasons.

I could not rebuild and push images to Quay on a weekly basis, but that means that any potential CVEs will not get addressed until a new NGINX release takes place.

[aydosman]

Hey @alessfg thanks for the advice however I'm seeing the same behavior at ghcr.io/nginx/nginx-unprivileged, note the timestamps are when my action observed the different digest not when it changed. Please advise

1.26.3:
  2025-05-02T13:53:59Z -> sha256:cd83270b8e0019ee7687198a473842b45c1ef31126229d511afd96c8bba625bc

1.27.4:
  2025-05-02T13:53:59Z -> sha256:663d1f83b634d625f3c36b323b21cc5c2c4ed5ebf912a01bfe13cce2f455e92e

1.27.5:
  2025-05-02T13:53:59Z -> sha256:965bba109c9dd2ad90342f80653d94768e2d8b814551a4e5c9a1130f7992ee4b
  2025-05-05T12:01:34Z -> sha256:799d8bfce0e9a2df29181f2f01c22f2ff1282f92fde3c76daa825fa4ecd28f5e
  2025-05-12T12:01:47Z -> sha256:773b6546272c808baf4dd5f8da71f61c561adba04b1c627883d5c1da67e1f1ef
  2025-05-19T12:01:42Z -> sha256:212f7e44f95a035bc40822d4bba7c0565b6b782cc311fa423cae1568a622aac5

1.28.0:
  2025-05-02T13:53:59Z -> sha256:aa538e1dc81068827c28ad5855bbd721f0f17a3d303f4b5d6737ddc219d1c8c6
  2025-05-05T12:01:34Z -> sha256:f78874a7cd6c4dd8a5d33a287c7a09a2a4ae8717fc843500246d5b69b9dcb239
  2025-05-12T12:01:47Z -> sha256:268e2bf6771ecce36403cc783635ab2a3ca95840c4754266f853d658f98247eb
  2025-05-19T12:01:42Z -> sha256:b9ec99876f483ac7dddddba0f0272cd64f44865480001d421482d00a74d4c9a2

---

update - However it seems like all the digests are still available, great.

[alessfg]

Yup, there really isn't a simple way to fix that, but you should be able to pin your images to the digest. Eventually the goal is to try to reduce the amount of builds so that instead of being built on a weekly basis, the images only get rebuilt when a "critical" CVE is detected. There's an issue tracking this #136 but I haven't had a chance to start working on it.