High CPU consumption observed in ztunnel during connection spikes

[barth12]

Description:
We are experiencing potentially high CPU consumption in the Istio ztunnel component during periods of higher connection activity. We would like to get more insights and confirmation on whether this is expected behavior or if further investigation is required.

Observations:

• During a spike in traffic, we observed up to 72,000 new connections being established within a 1-minute window in a specific ztunnel.
• The CPU usage for this ztunnel reached approximately 2 vCPUs.

Questions:

• Is this level of CPU consumption expected given the higher volume of new connections being established in such a period?
• Are there any known optimizations or configuration tweaks we could apply to mitigate CPU usage during these spikes?

Additional Context:
Istio version: 1.23.0
Environment: EKS
Overprovisioning: Some of our workloads deal with much higher traffic than others. This creates a situation where some ztunnels consume significantly more CPU than others. Because CPU consumption is controlled at the DaemonSet level, we end up heavily overprovisioning some ztunnels to handle this situtation. As a result, migrating to Ambient mode has not significantly reduced CPU consumption or costs as we initially expected.

Thank you for your assistance!

[linsun]

Hi @barth12 thanks for reporting this! Could you share a bit about your scenario? That is a ton of connections!

72,000 new connections being established within a 1-minute window in a specific ztunnel

[howardjohn]

We've been discussing this back and forth on slack a bit for higher bandwidth, I'll update (or let @barth12 ) once we have some more complete state