[Feature] Hubble in Azure CNI Powered by Cilium

[illrill]

I'd like to see support for Hubble (hubble-relay and hubble-ui) in Azure CNI Powered by Cilium

[siegenthalerroger]

Our team is also looking to have hubble enabled in Azure CNI Powered by Cilium. Less interested in hubble-ui tbh, as that is deployable in standalone mode and could easily be left to the user.

Being able to configure Cilium the same way e.g. CoreDNS is configured would be great, allow us to enable Hubble and Service Mesh as we see fit.

[mac-kul]

is there any other good solution that could help us with network observability? On the docs we can see the benefit:

• Better observability of cluster traffic

How this can be achieved? I would like to be able to monitor the traffic inside the cluster.

[PixelRobots]

is there any other good solution that could help us with network observability? On the docs we can see the benefit:

• Better observability of cluster traffic

How this can be achieved? I would like to be able to monitor the traffic inside the cluster.

Hi @mac-kul

Check out this doc's page. https://learn.microsoft.com/en-us/azure/aks/network-observability-managed-cli?tabs=cilium.

It is currently in preview but could be a good starting point for you to look at. It also supports non cilium clusters.

[siegenthalerroger]

@PixelRobots from what I understand, Network Observability only provides metrics about aggregate traffic (bandwidth, latency, etc). There's no capability for a "Flow" type view, where you can see/analyse individual requests/packets right?

[PixelRobots]

@PixelRobots from what I understand, Network Observability only provides metrics about aggregate traffic (bandwidth, latency, etc). There's no capability for a "Flow" type view, where you can see/analyse individual requests/packets right?

According to the docs it currently only supports node level metrics. So it suggests that pod level metrics will hopefully come soon.

In the meantime it could also be good to look at https://www.inspektor-gadget.io/. To fill the gap when troubleshooting.

[illrill]

Being able to configure Cilium the same way e.g. CoreDNS is configured would be great, allow us to enable Hubble and Service Mesh as we see fit.

I like this idea. By providing a way for users to override the kube-system/cilium-config CM, a lot of other Cilium-related issues (e.g. #3450 #3797) could also be closed, since we would have the ability to toggle these things on ourselves.

One approach could be to use CiliumNodeConfig as the config override mechanism. AKS team would just need to install the CRD. We could use the CR with a nodeSelector that selects all nodes.

[JitseHijlkema]

Just announced on KubeCon 2024 in Paris, which will gives us the Hubble functionality:

Advanced Networking Observability which includes Hubble (CLI+UI) and with Retina (made OpenSource) also supports other CNI's and other platforms.

[JitseHijlkema]

Just announced on KubeCon 2024 in Paris, which will gives us the Hubble functionality:

Advanced Networking Observability which includes Hubble (CLI+UI) and with Retina (made OpenSource) also supports other CNI's and other platforms.

[bartwitkowski]

@JitseHijlkema do you remember from which session is this photo?

[JitseHijlkema]

@JitseHijlkema do you remember from which session is this photo?

This is presented during Azure Day with Kubernetes on March 19 in Paris:

Link to presentation -Starting from slide 43

[PixelRobots]

@JitseHijlkema do you remember from which session is this photo?

You can find the video on the AKS youtube Channel. here is where it starts. https://youtu.be/hb__fHnx11s?si=jJderXVXRwWpHPfh&t=2013

I would advise you to watch all the sessions if you have not.

[bartwitkowski]

I was on the KubeCon in Paris but not on the Azure day but the Cilium :-).
I'm watching Azure Day sessions in the meantime. Great stuff there!

Thanks @JitseHijlkema @PixelRobots

[CalleB3]

Hey, any updates on this issue?

[illrill]

It was released in public preview in May: https://learn.microsoft.com/en-us/azure/aks/advanced-network-observability-cli

AKS bundles hubble-relay, and then you'll need to install hubble-ui yourself.

[PixelRobots]

I did a blog post on it too. It might help. https://pixelrobots.co.uk/2024/06/advanced-network-observability-supercharging-container-network-observability-in-azure-kubernetes-service-aks/

[bartwitkowski]

Well, Advanced Network Observability is a part of Advanced Container Networking Services and it is a paid feature: https://azure.microsoft.com/en-us/pricing/details/azure-container-networking-services/.
Is there any way to do "BYO Hubble" and look through the metrics/flows/networkpolicies?

[jodr5786]

Are there plans to add the ability to export the flow logs to a logging solution? Or will the Hubble exporter (link) be a feature that can be enabled? We're in need of them due to compliance requirements.

[illrill]

Are there plans to add the ability to export the flow logs to a logging solution? Or will the Hubble exporter (link) be a feature that can be enabled? We're in need of them due to compliance requirements.

Our company's security team also requires us to log all outbound connections destined towards any public internet endpoint. To solve this, we run a standalone deployment, which tails a filtered subset of all flow logs by executing hubble observe --follow directly against hubble-relay, and then prints it all to stdout. From there on, our normal container log collecting tools take care of the rest.

It's perhaps not the most scalable solution, but we've been running this in our BYOCNI clusters with self-installed Cilium for a long time, since before the Hubble Exporter even existed. Now we're running it in our clusters with Azure CNI Powered by Cilium too. I would probably want to switch to the official Hubble Exporter when (if) supported in AKS, though.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hubble-flowlogs-exporter
  namespace: kube-system
  labels:
    app.kubernetes.io/name: hubble-flowlogs-exporter
    app.kubernetes.io/part-of: cilium
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hubble-flowlogs-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hubble-flowlogs-exporter
        app.kubernetes.io/part-of: cilium
    spec:
      containers:
      - command: ["sh", "-c"]
        env:
        - name: TLS_RELAY_SERVER_NAME
          value: instance.hubble-relay.cilium.io
        - name: TLS_RELAY_CA_CERT_FILES
          value: /var/lib/hubble-relay/certs/hubble-relay-ca.crt
        - name: TLS_RELAY_CLIENT_CERT_FILE
          value: /var/lib/hubble-relay/certs/client.crt
        - name: TLS_RELAY_CLIENT_KEY_FILE
          value: /var/lib/hubble-relay/certs/client.key
        args: ["hubble observe --tls --tls-server-name $TLS_RELAY_SERVER_NAME --tls-ca-cert-files $TLS_RELAY_CA_CERT_FILES --tls-client-cert-file $TLS_RELAY_CLIENT_CERT_FILE --tls-client-key-file $TLS_RELAY_CLIENT_KEY_FILE --server hubble-relay.kube-system.svc.cluster.local.:443 --ipv4 --traffic-direction egress --to-label reserved:world --follow --output jsonpb"]
        image: quay.io/cilium/cilium:v1.15.0
        name: hubble-flowlogs-exporter
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: hubble-relay-client-certs
          mountPath: /var/lib/hubble-relay/certs
          readOnly: true
      # Mount the secret containing the client-side certificate for mTLS communication with Hubble Relay.
      # This secret is already pre-created by AKS as part of the Hubble Relay installation, we just need to mount it.
      volumes:
      - name: hubble-relay-client-certs
        projected:
          defaultMode: 0400
          sources:
          - secret:
              name: hubble-relay-client-certs
              items:
                - key: tls.crt
                  path: client.crt
                - key: tls.key
                  path: client.key
                - key: ca.crt
                  path: hubble-relay-ca.crt

[jodr5786]

Are there plans to add the ability to export the flow logs to a logging solution? Or will the Hubble exporter (link) be a feature that can be enabled? We're in need of them due to compliance requirements.

Our company's security team also requires us to log all outbound connections destined towards any public internet endpoint. To solve this, we run a standalone deployment, which tails a filtered subset of all flow logs by executing hubble observe --follow directly against hubble-relay, and then prints it all to stdout. From there on, our normal container log collecting tools take care of the rest.

It's perhaps not the most scalable solution, but we've been running this in our BYOCNI clusters with self-installed Cilium for a long time, since before the Hubble Exporter even existed. Now we're running it in our clusters with Azure CNI Powered by Cilium too. I would probably want to switch to the official Hubble Exporter when (if) supported in AKS, though.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hubble-flowlogs-exporter
  namespace: kube-system
  labels:
    app.kubernetes.io/name: hubble-flowlogs-exporter
    app.kubernetes.io/part-of: cilium
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hubble-flowlogs-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hubble-flowlogs-exporter
        app.kubernetes.io/part-of: cilium
    spec:
      containers:
      - command: ["sh", "-c"]
        env:
        - name: TLS_RELAY_SERVER_NAME
          value: instance.hubble-relay.cilium.io
        - name: TLS_RELAY_CA_CERT_FILES
          value: /var/lib/hubble-relay/certs/hubble-relay-ca.crt
        - name: TLS_RELAY_CLIENT_CERT_FILE
          value: /var/lib/hubble-relay/certs/client.crt
        - name: TLS_RELAY_CLIENT_KEY_FILE
          value: /var/lib/hubble-relay/certs/client.key
        args: ["hubble observe --tls --tls-server-name $TLS_RELAY_SERVER_NAME --tls-ca-cert-files $TLS_RELAY_CA_CERT_FILES --tls-client-cert-file $TLS_RELAY_CLIENT_CERT_FILE --tls-client-key-file $TLS_RELAY_CLIENT_KEY_FILE --server hubble-relay.kube-system.svc.cluster.local.:443 --ipv4 --traffic-direction egress --to-label reserved:world --follow --output jsonpb"]
        image: quay.io/cilium/cilium:v1.15.0
        name: hubble-flowlogs-exporter
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: hubble-relay-client-certs
          mountPath: /var/lib/hubble-relay/certs
          readOnly: true
      # Mount the secret containing the client-side certificate for mTLS communication with Hubble Relay.
      # This secret is already pre-created by AKS as part of the Hubble Relay installation, we just need to mount it.
      volumes:
      - name: hubble-relay-client-certs
        projected:
          defaultMode: 0400
          sources:
          - secret:
              name: hubble-relay-client-certs
              items:
                - key: tls.crt
                  path: client.crt
                - key: tls.key
                  path: client.key
                - key: ca.crt
                  path: hubble-relay-ca.crt

I considered doing something similar as a stop-gap, so this is good to know that someone else has already had success with it.

[vakalapa]

Is there any way to do "BYO Hubble" and look through the metrics/flows/networkpolicies?

Customization on Azure CNI powered by Cilium is not possible today. Since Hubble runs inside the Cilium agent, BYO Hubble would need Cilium CNI in BYO CNI cluster.

With Advanced Container Networking Services (https://aka.ms/acns) Observability, you will get Hubble in both Cilium and non-Cilium dataplanes. In Azure CNI powered by Cilium when ACNS is enabled, you will get Hubble enabled inside the Cilium agent, Hubble relay deployed on the cluster and support for FQDN Policies is enabled as well.

In non-Cilium Azure CNI flavors, Hubble will be running inside the Retina agent and similar observability capabilities are unlocked.
More details:

Are there plans to add the ability to export the flow logs to a logging solution?

@jodr5786 thanks for bringing this feature req, we have this feature in the roadmap. One question is where you want the flow logs to be stored, in Log Analytics workspace so that they are query able, or in Azure Storage account to be pulled later on on-demand and can be fed into splunk.

[jodr5786]

@vakalapa - my preference would be to have them output within the container environment, so that they could be picked up by something like fluentd, or other similar logging solutions. And then we'd have the flexibility to send them where we need to. If that's not possible, a storage account or Log Analytics may work as well, they just aren't a part of the K8s logging workflow for our company.

[illrill]

@vakalapa I think you should just enable use of Hubble Exporter as-is, and not tie it to any Azure specific logging or storage solutions. Hubble Exporter writes the flow logs to a file, from where they can be picked up by most (all?) Kubernetes native log collection tools, for example Opentelemetry's filelog receiver.