[Bug] Verifying Image Signatures fails: Unable to Retrieve Public Key from Azure Key Vault

[RaphDiana]

Kyverno Version

1.12

Kubernetes Version

1.29

Kubernetes Platform

AKS

Description

I am experiencing an issue with verifyImages policy when attempting to validate image signatures created with Sigstore Cosign with a key stored in Azure Key Vault. The verification process fails because the public key cannot be retrieved from Azure Key Vault. Below is the error message encountered during the operation:

"no matching signatures: failed to get public key: public key: public key: AzureCLICredential: fork/exec /bin/sh: no such file or directory"

Steps to reproduce

1. Create a Cosign signing key in Azure Key Vault:

cosign generate-key-pair --kms "azurekms://[VAULT_NAME][VAULT_URI]/[KEY]" --output-key-prefix="key-prefix"

2. Setup Kyverno to use Workload Identity

• Create a managed identity
• Configure Kyverno to use Workload Identity:

config:
  webhookAnnotations:
    admissions.enforcer/disabled: "true"
admissionController:
  rbac:
    serviceAccount:
      annotations: 
        azure.workload.identity/client-id: <client-id>
  podLabels:
    azure.workload.identity/use: "true"

• Assigned the necessary roles to the managed identity to access the signing key in Azure Key Vault

2. Configure a verifyImage policy to verify image signatures using Sigstore Cosign.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image
spec:
  webhookConfiguration:
    failurePolicy: Fail
    timeoutSeconds: 30
  background: false
  rules:
    - name: check-image
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaceSelector:
                matchExpressions:
                  - key: toolkit.fluxcd.io/tenant
                    operator: In
                    values:
                      - dev-team
      verifyImages:
        - imageReferences:
            - "registry-url*"
          failureAction: Enforce
          attestors:
            - count: 1
              entries:
                - keys:
                    publicKeys: azurekms://[VAULT_NAME][VAULT_URI]/[KEY]
                    rekor:
                      ignoreTlog: true
                      url: https://rekor.sigstore.dev

2. Attempt to deploy a signed image to AKS cluster.
3. Observe the failure in retrieving the public key due to the error described above.

Expected behavior

Kyverno should successfully retrieve the public key from Azure Key Vault and verify the image signature as Kyverno supports Azure Key Vault KMS.

Screenshots

No response

Kyverno logs

2025-03-14T17:44:08Z TRC github.com/kyverno/kyverno/pkg/engine/internal/imageverifier.go:219 > cache entry not found imageRef=<image-reference>logger=engine.verify namespace= new.kind=Deployment new.name=<deployment-name> new.namespace=<ns-name> policy=verify-image policy.apply=All policy.name=verify-image policy.namespace= rule.name=autogen-verify-image ruleName=autogen-verify-image v=2
2025-03-14T17:44:08Z TRC github.com/kyverno/kyverno/pkg/engine/internal/imageverifier.go:274 > verifying image signatures attestations=0 attestors=1 image=<image-reference>logger=engine.verify new.kind=Deployment new.name=<deployment-name> new.namespace=<ns-name> policy.apply=All policy.name=verify-image policy.namespace= rule.name=autogen-verify-image v=2
2025-03-14T17:44:08Z INF github.com/kyverno/kyverno/pkg/cosign/cosign.go:82 > image verification failed error="no matching signatures: failed to get public key: public key: public key: AzureCLICredential: fork/exec /bin/sh: no such file or directory" logger=cosign v=0
2025-03-14T17:44:08Z INF github.com/kyverno/kyverno/pkg/engine/internal/imageverifier.go:494 > image attestors verification failed errors=".attestors[0].entries[0].keys: no matching signatures: failed to get public key: public key: public key: AzureCLICredential: fork/exec /bin/sh: no such file or directory" image=<image-reference>logger=engine.verify new.kind=Deployment new.name=<deployment-name> new.namespace=<ns-name> policy.apply=All policy.name=verify-image policy.namespace= requiredCount=1 rule.name=autogen-verify-image v=0 verifiedCount=0
2025-03-14T17:44:08Z ERR github.com/kyverno/kyverno/pkg/engine/internal/imageverifier.go:318 > failed to verify image error=".attestors[0].entries[0].keys: no matching signatures: failed to get public key: public key: public key: AzureCLICredential: fork/exec /bin/sh: no such file or directory" image=<image-reference>logger=engine.verify new.kind=Deployment new.name=<deployment-name> new.namespace=<ns-name> policy.apply=All policy.name=verify-image policy.namespace= rule.name=autogen-verify-image
2025-03-14T17:44:08Z TRC github.com/kyverno/kyverno/pkg/webhooks/utils/block.go:29 > blocking admission request URLParams= clusterroles=["cluster-admin","system:basic-user","system:discovery","system:public-info-viewer","system:service-account-issuer-discovery"] gvk="apps/v1, Kind=Deployment" gvr={"group":"apps","resource":"deployments","version":"v1"} kind=Deployment logger=webhooks/resource/mutate name=<deployment-name> namespace=<ns-name> operation=CREATE policy=verify-image resource.gvk="apps/v1, Kind=Deployment" roles=[] uid=acb3c34c-44d0-417c-ab44-42b7b3ee3247 user={"groups":["system:serviceaccounts","system:serviceaccounts:<ns-name>","system:authenticated"],"uid":"88dc20eb-d9da-4963-9059-a7214d9c9736","username":"system:serviceaccount:<sa-name>"} v=2
2025-03-14T17:44:08Z ERR github.com/kyverno/kyverno/pkg/webhooks/resource/handlers.go:235 > image verification failed error="\n\nresource Deployment/<ns-name>/<deployment-name> was blocked due to the following policies \n\nverify-image:\n  autogen-verify-image: 'failed to verify image <image-reference>:\n    .attestors[0].entries[0].keys: no matching signatures: failed to get public key:\n    public key: public key: AzureCLICredential: fork/exec /bin/sh: no such file or\n    directory'\n" URLParams= clusterroles=["cluster-admin","system:basic-user","system:discovery","system:public-info-viewer","system:service-account-issuer-discovery"] gvk="apps/v1, Kind=Deployment" gvr={"group":"apps","resource":"deployments","version":"v1"} kind=Deployment logger=webhooks/resource/mutate name=<deployment-name> namespace=<ns-name> operation=CREATE resource.gvk="apps/v1, Kind=Deployment" roles=[] uid=acb3c34c-44d0-417c-ab44-42b7b3ee3247 user={"groups":["system:serviceaccounts","system:<sa-name>","system:authenticated"],"uid":"88dc20eb-d9da-4963-9059-a7214d9c9736","username":"system:serviceaccount:<sa-name>"}

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[RaphDiana]

@snaquekiller FYI