Istio 1.21+ unable to get secrets from SPIRE

[vassilvk]

• Version: 1.11.0
• Platform: Kubernetes 1.29.6 (EKS)
• Istio: 1.22
• Subsystem: server, agent, (istio?)

Not sure if this is an Istio or a SPIRE issue, however, after following this Istio + SPIRE guide, the Istio gateway pods detect the SPIRE-provided SDS API (as expected), however, they fail to obtain certificates from their SPIRE agents.

The gateway proxy logs contain a series of the following warning message:

2024-12-06T18:23:13.287724Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:155	StreamSecrets gRPC config stream to sds-grpc closed: 3, workload is not authorized for the requested identities ["file-root:system"]	thread=28

The SPIRE agents log the following corresponding message:

time="2024-12-06T18:24:09Z" level=error msg="Error building stream secrets response" error="rpc error: code = InvalidArgument desc = workload is not authorized for the requested identities [\"file-root:system\"]" method=StreamSecrets pid=1364350 service=SDS.v3 subsystem_name=endpoints

The SPIRE infrastructure was deployed using spire helm chart version 0.24.1 (app version 1.11.0).

The ClusterSPIFFEID CRDs for the Istio gateway pods function correctly - I can see the SPIFFE registration entries for the gateway pods added to the SPIRE server.

What I don't see is an entry for identity file-root:system.

Not sure why the gateway is attempting to obtain a certificate for identity file-root:system, but looking at Istio's code this indicates that the proxy is asking for the OS root certificates.

I understand that this might be a an Istio question, but I figured someone here might have experienced the same.

[vassilvk]

Found out what causes the issue.

Istio 1.21 switched to verifying server certificates at proxies by default.

This leads to Istio proxies (both gateways and sidecars) trying to procure the operating system's CA by asking the SDS service for the file-root:system resource. When the SDS service is SPIRE, since no file-root:system registration entry exists on the SPIRE server, SPIRE agent fails to provide this secret, essentially breaking Istio's TLS verification and ability to originate HTTPS connections to non-mTLS endpoints.

Setting VERIFY_CERTIFICATE_AT_CLIENT env variable on istiod to false solves the issue for Istio 1.22 (note the docs incorrectly talk about VERIFY_CERT_AT_CLIENT).

The above workaround is not ideal as we need to have proxies verify TLS certificates coming from external servers.
Does it make sense to implement support for the file-root:system resource in SPIRE agent SDS implementation?