From 9c9f096682da384c9fe4ca2d3ffc36f7fa082453 Mon Sep 17 00:00:00 2001 From: zhangzujian Date: Wed, 5 Aug 2026 11:59:44 +0800 Subject: [PATCH 1/6] docs: add VPC egress gateway observability guide Signed-off-by: zhangzujian --- docs/reference/kube-ovn-api.en.md | 83 ++++++++++ .../vpc-egress-gateway-observability.en.md | 154 ++++++++++++++++++ docs/vpc/vpc-egress-gateway.en.md | 2 + 3 files changed, 239 insertions(+) create mode 100644 docs/vpc/vpc-egress-gateway-observability.en.md diff --git a/docs/reference/kube-ovn-api.en.md b/docs/reference/kube-ovn-api.en.md index 20497e814..3213c1dec 100644 --- a/docs/reference/kube-ovn-api.en.md +++ b/docs/reference/kube-ovn-api.en.md @@ -741,6 +741,7 @@ In each CRD definition, the Condition field in Status follows the above format, | tolerations | []Toleration | Optional. Standard Kubernetes tolerations | | resources | ResourceRequirements | Optional. Container resource limits; the controller uses defaults when unspecified | | bandwidth | BandwidthLimit | Optional. Per-replica ingress/egress bandwidth limit, expressed as integer Mbps or a supported bit-rate quantity | +| observability | VpcEgressGatewayObservability | Optional. Native interface metrics, conntrack metrics, JSON flow logs, and ServiceMonitor metadata. All capabilities default to disabled | ##### VpcEgressGatewaySelector @@ -781,6 +782,88 @@ In each CRD definition, the Condition field in Status follows the above format, | ingress | Int64 or String | Limit for traffic entering the VPC from the external network. Integers and numeric strings are Mbps; quantities use `M`, `Mi`, `G`, or `Gi` bits per second | | egress | Int64 or String | Limit for traffic leaving the VPC for the external network. Integers and numeric strings are Mbps; quantities use `M`, `Mi`, `G`, or `Gi` bits per second | +##### VpcEgressGatewayObservability + +Kubernetes 1.29 or later is required. For runtime behavior, metrics, flow-log schema, and operational limits, see [VPC Egress Gateway Observability](../vpc/vpc-egress-gateway-observability.en.md). + +| Property Name | Type | Description | +| --- | --- | --- | +| resources | ResourceRequirements | Optional. Observer sidecar resources. Defaults to requests of `20m` CPU and `64Mi` memory and limits of `200m` CPU and `256Mi` memory | +| interfaceMetrics | VpcEgressGatewayObservabilityFeature | Interface metrics collector configuration | +| conntrack | VpcEgressGatewayConntrackObservability | Conntrack metrics and flow-log configuration | +| serviceMonitor | VpcEgressGatewayServiceMonitor | Metadata merged into the per-gateway ServiceMonitor | + +##### VpcEgressGatewayObservabilityFeature + +| Property Name | Type | Description | +| --- | --- | --- | +| enabled | Boolean | Whether the feature is enabled. Defaults to false | + +##### VpcEgressGatewayConntrackObservability + +| Property Name | Type | Description | +| --- | --- | --- | +| metrics | VpcEgressGatewayObservabilityFeature | Low-cardinality NAT conntrack metrics configuration | +| log | VpcEgressGatewayConntrackLog | JSON Lines NAT flow-log configuration | + +##### VpcEgressGatewayConntrackLog + +| Property Name | Type | Description | +| --- | --- | --- | +| enabled | Boolean | Whether JSON flow logging to the observer container standard output is enabled. Defaults to false | +| events | []String | Optional. Lifecycle events to emit: `start` and/or `end`. Both are emitted when omitted | +| rateLimit | VpcEgressGatewayConntrackLogRateLimit | Per-Pod flow-log rate limit | +| filters | VpcEgressGatewayConntrackLogFilters | Optional include and exclude rules | + +##### VpcEgressGatewayConntrackLogRateLimit + +| Property Name | Type | Description | +| --- | --- | --- | +| recordsPerSecond | Int32 | Sustained records per second. Defaults to 100 | +| burst | Int32 | Maximum burst. Defaults to 1000 | + +##### VpcEgressGatewayConntrackLogFilters + +| Property Name | Type | Description | +| --- | --- | --- | +| include | []VpcEgressGatewayConntrackLogFilter | Optional include rules. An empty list includes every NAT flow | +| exclude | []VpcEgressGatewayConntrackLogFilter | Optional exclude rules. Exclude matches take precedence | + +##### VpcEgressGatewayConntrackLogFilter + +Configured fields in one rule are ANDed; values within a field and rules within one list are ORed. + +| Property Name | Type | Description | +| --- | --- | --- | +| addressFamilies | []String | Optional. `ipv4` and/or `ipv6` | +| protocols | []String | Optional. `tcp`, `udp`, `sctp`, `icmp`, `icmpv6`, and/or `other` | +| natTypes | []String | Optional. `snat`, `dnat`, and/or `snat_dnat` | +| original | VpcEgressGatewayConntrackTupleFilter | Original tuple match | +| translated | VpcEgressGatewayConntrackTupleFilter | Translated tuple match | + +##### VpcEgressGatewayConntrackTupleFilter + +| Property Name | Type | Description | +| --- | --- | --- | +| sourceCIDRs | []String | Optional source CIDRs | +| destinationCIDRs | []String | Optional destination CIDRs | +| sourcePorts | []VpcEgressGatewayPortRange | Optional inclusive source-port ranges | +| destinationPorts | []VpcEgressGatewayPortRange | Optional inclusive destination-port ranges | + +##### VpcEgressGatewayPortRange + +| Property Name | Type | Description | +| --- | --- | --- | +| start | Int32 | Inclusive first port, from 0 through 65535 | +| end | Int32 | Inclusive last port, from `start` through 65535 | + +##### VpcEgressGatewayServiceMonitor + +| Property Name | Type | Description | +| --- | --- | --- | +| labels | map[String]String | Optional ServiceMonitor labels. Controller-required selector labels cannot be overridden | +| annotations | map[String]String | Optional ServiceMonitor annotations | + #### VpcEgressGatewayStatus | Property Name | Type | Description | diff --git a/docs/vpc/vpc-egress-gateway-observability.en.md b/docs/vpc/vpc-egress-gateway-observability.en.md new file mode 100644 index 000000000..083ddba2c --- /dev/null +++ b/docs/vpc/vpc-egress-gateway-observability.en.md @@ -0,0 +1,154 @@ +# VPC Egress Gateway Observability + +VPC Egress Gateway can run one native observability sidecar in every gateway Pod. The sidecar can expose internal and external interface counters, expose low-cardinality conntrack NAT metrics, and write versioned NAT flow logs to its container standard output. All three capabilities are disabled by default and do not affect gateway data-plane readiness. + +## Requirements and Upgrade Notes + +- Kubernetes 1.29 or later is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). If the Kubernetes version is older or cannot be discovered, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. +- Upgrade the `vpc-egress-gateways.kubeovn.io` CRD explicitly before creating resources that use `spec.observability`. Helm does not upgrade CRDs that are already installed automatically. Apply the CRD delivered with the same Kube-OVN version, using your normal CRD upgrade procedure. +- Conntrack collection requires `NET_ADMIN` in the gateway Pod network namespace. The observer runs as UID and GID 65534, drops all other capabilities, disallows privilege escalation, and uses a read-only root filesystem. The observer binary has no file capability. +- Prometheus Operator is optional. The gateway works without the ServiceMonitor CRD. + +## Enabling Observability + +The following example enables both metrics collectors and JSON flow logs: + +```yaml +apiVersion: kubeovn.io/v1 +kind: VpcEgressGateway +metadata: + name: gateway1 + namespace: default +spec: + vpc: ovn-cluster + replicas: 2 + externalSubnet: macvlan1 + policies: + - snat: true + subnets: + - ovn-default + observability: + resources: + requests: + cpu: 20m + memory: 64Mi + limits: + cpu: 200m + memory: 256Mi + interfaceMetrics: + enabled: true + conntrack: + metrics: + enabled: true + log: + enabled: true + events: [start, end] + rateLimit: + recordsPerSecond: 100 + burst: 1000 + filters: + include: [] + exclude: [] + serviceMonitor: + labels: + monitoring: platform + annotations: {} +``` + +When `resources` is empty or omitted, the observer requests `20m` CPU and `64Mi` memory and is limited to `200m` CPU and `256Mi` memory. + +The controller stores the runtime configuration in a per-gateway ConfigMap. Collector switches, flow-log events, filters, and rate limits are reloaded without replacing gateway Pods. The following changes update the Deployment and replace Pods: + +- enabling observability for the first time; +- disabling all observability capabilities; +- changing `observability.resources`; +- changing the gateway workload image. + +If an explicitly selected older workload image does not contain `/kube-ovn/vpc-egress-gateway-observer`, the launcher runs `sleep infinity` instead, so the missing binary does not block the data plane. + +## Metrics + +The observer serves `/metrics` and `/healthz` on TCP port `10666`. It uses a private Prometheus registry and does not expose `go_*`, `process_*`, or `promhttp_*` metrics. Every metric includes the identity labels `namespace`, `name`, `pod`, and `node`. + +### Interface Metrics + +Interface metrics are read from `/proc/net/dev` on every scrape. The observer resolves the primary interface and the external Multus interface from the Pod `network-status` annotation on the first successful scrape and caches those names for the lifetime of the process. + +| Metric | Additional labels | Description | +| :--- | :--- | :--- | +| `kube_ovn_vpc_egress_gateway_interface_rx_bytes_total` | `interface`, `type` | Received bytes. | +| `kube_ovn_vpc_egress_gateway_interface_tx_bytes_total` | `interface`, `type` | Transmitted bytes. | +| `kube_ovn_vpc_egress_gateway_interface_rx_packets_total` | `interface`, `type` | Received packets. | +| `kube_ovn_vpc_egress_gateway_interface_tx_packets_total` | `interface`, `type` | Transmitted packets. | +| `kube_ovn_vpc_egress_gateway_interface_drops_total` | `interface`, `type`, `direction` | Dropped packets. | +| `kube_ovn_vpc_egress_gateway_interface_errors_total` | `interface`, `type`, `direction` | Packet errors. | + +The `type` label is `internal` or `external`, and `direction` is `rx` or `tx`. + +### Conntrack Metrics + +Conntrack metrics aggregate flows without using IP addresses or ports as labels: + +| Metric | Additional labels | Description | +| :--- | :--- | :--- | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_flows_active` | `address_family`, `protocol`, `nat_type` | Current NAT flows initialized from the table dump and maintained from events. | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_flows_started_total` | `address_family`, `protocol`, `nat_type` | NAT flows observed starting after the observer subscribed. | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_flows_ended_total` | `address_family`, `protocol`, `nat_type` | NAT flow end events observed. | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_packets_total` | `address_family`, `protocol`, `nat_type`, `direction` | Packets observed when conntrack accounting is available. | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_bytes_total` | `address_family`, `protocol`, `nat_type`, `direction` | Bytes observed when conntrack accounting is available. | + +`address_family` is `ipv4` or `ipv6`. Known `protocol` values are `tcp`, `udp`, `sctp`, `icmp`, and `icmpv6`; all other protocol numbers are aggregated as `other`. `nat_type` is `snat`, `dnat`, or `snat_dnat`, and `direction` is `original` or `reply`. + +Operational metrics use the `kube_ovn_vpc_egress_gateway_observability_*` prefix. They report collector availability, configuration reload results, conntrack events and errors, cache entries/capacity/evictions, emitted and dropped log records, and conntrack accounting availability. One collector failing does not make `/metrics` fail or suppress data from the other collector. + +## ServiceMonitor Discovery + +When either metrics collector is enabled, the controller creates a per-gateway headless Service with `publishNotReadyAddresses: true` and a numeric `targetPort` of `10666`. It also creates a ServiceMonitor in the gateway namespace. User labels and annotations from `observability.serviceMonitor` are merged into the ServiceMonitor metadata, but controller-required selector labels cannot be overridden. + +If the ServiceMonitor CRD is absent, the controller sets `ServiceMonitorReady=False` and retries periodically. It creates the ServiceMonitor automatically after the CRD becomes available. This condition is separate from the existing gateway `Ready` condition. + +## JSON Flow Logs + +Flow logs are JSON Lines written only to the `observability` container standard output. Diagnostics are written to standard error. Read the records with: + +```shell +kubectl logs -n default -c observability +``` + +The `v1` schema emits `start` and `end` lifecycle records. Raw conntrack update events are not logged. A record resembles: + +```json +{"schemaVersion":"v1","timestamp":"2026-08-05T03:40:00Z","event":"end","conntrackID":2471802592,"zone":0,"namespace":"default","name":"gateway1","pod":"gateway1-7c9b8f8db7-r9x8m","node":"worker-1","addressFamily":"ipv4","protocol":"tcp","protocolNumber":6,"natType":["snat"],"original":{"sourceIP":"10.16.0.25","sourcePort":42136,"destinationIP":"203.0.113.10","destinationPort":443},"translated":{"sourceIP":"172.17.0.11","sourcePort":42136,"destinationIP":"203.0.113.10","destinationPort":443},"counters":{"originalPackets":8,"originalBytes":624,"replyPackets":7,"replyBytes":591}} +``` + +For protocols aggregated as `other`, `protocolNumber` retains the original IP protocol number. `natType` is a stable sorted array and can contain both `dnat` and `snat`. The `counters` object is omitted when conntrack accounting is unavailable; the observer never changes `nf_conntrack_acct`. + +The observer subscribes to conntrack events before dumping the existing table. Existing NAT flows initialize the active-flow metric and cache, but do not produce historical `start` logs. The cache is limited to 65,536 entries. The netlink event channel and asynchronous log queue are each limited to 4,096 entries. A blocked container log consumer or full log queue drops log records without blocking conntrack collection. + +By default, all NAT flows and both lifecycle events are logged, subject to a per-Pod limit of 100 records per second with a burst of 1,000. These are operational, best-effort records and metrics; they are not intended for billing-grade accounting. + +## Flow-log Filters + +Each include or exclude rule can match address families, protocols, NAT types, and fields in the original or translated tuple. Values within one field are ORed, configured fields within one rule are ANDed, and rules in one list are ORed. Exclude rules take precedence over include rules. An empty include list matches every NAT flow. + +Ports use inclusive structured ranges. The following configuration logs TCP SNAT flows from `10.16.0.0/16` to destination ports 80 through 443, except translated destinations in `192.0.2.0/24`: + +```yaml +spec: + observability: + conntrack: + log: + enabled: true + filters: + include: + - protocols: [tcp] + natTypes: [snat] + original: + sourceCIDRs: [10.16.0.0/16] + destinationPorts: + - start: 80 + end: 443 + exclude: + - translated: + destinationCIDRs: [192.0.2.0/24] +``` diff --git a/docs/vpc/vpc-egress-gateway.en.md b/docs/vpc/vpc-egress-gateway.en.md index a52b40810..7f1c8dc50 100644 --- a/docs/vpc/vpc-egress-gateway.en.md +++ b/docs/vpc/vpc-egress-gateway.en.md @@ -20,6 +20,8 @@ VPC Egress Gateway also supports establishing dynamic routing with external networks through BGP and EVPN/VXLAN. For details, refer to [Egress Gateway BGP/EVPN Support](./vpc-egress-gateway-bgp-evpn.en.md). +For interface metrics, conntrack NAT metrics, and JSON flow logs, refer to [VPC Egress Gateway Observability](./vpc-egress-gateway-observability.en.md). + At the same time, VPC Egress Gateway has the following limitations: - Uses macvlan for underlying network connectivity, requiring [Underlay support](../start/underlay.en.md#environment-requirements) from the underlying network From a1352130f40c2749eddf5b5ba3c5d25907401a49 Mon Sep 17 00:00:00 2001 From: zhangzujian Date: Wed, 5 Aug 2026 16:58:26 +0800 Subject: [PATCH 2/6] docs: clarify observer capability requirements Signed-off-by: zhangzujian --- docs/vpc/vpc-egress-gateway-observability.en.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/vpc/vpc-egress-gateway-observability.en.md b/docs/vpc/vpc-egress-gateway-observability.en.md index 083ddba2c..f7f7cdbb4 100644 --- a/docs/vpc/vpc-egress-gateway-observability.en.md +++ b/docs/vpc/vpc-egress-gateway-observability.en.md @@ -6,7 +6,7 @@ VPC Egress Gateway can run one native observability sidecar in every gateway Pod - Kubernetes 1.29 or later is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). If the Kubernetes version is older or cannot be discovered, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. - Upgrade the `vpc-egress-gateways.kubeovn.io` CRD explicitly before creating resources that use `spec.observability`. Helm does not upgrade CRDs that are already installed automatically. Apply the CRD delivered with the same Kube-OVN version, using your normal CRD upgrade procedure. -- Conntrack collection requires `NET_ADMIN` in the gateway Pod network namespace. The observer runs as UID and GID 65534, drops all other capabilities, disallows privilege escalation, and uses a read-only root filesystem. The observer binary has no file capability. +- Conntrack collection requires `NET_ADMIN` in the gateway Pod network namespace. The official observer binary carries only the `CAP_NET_ADMIN` file capability, while the generated container security context admits only `NET_ADMIN` into the capability bounding set. `allowPrivilegeEscalation` is enabled so that this trusted file capability survives the non-root launcher `exec`; the observer still runs as UID and GID 65534, drops all other capabilities, and uses a read-only root filesystem. - Prometheus Operator is optional. The gateway works without the ServiceMonitor CRD. ## Enabling Observability @@ -64,7 +64,7 @@ The controller stores the runtime configuration in a per-gateway ConfigMap. Coll - changing `observability.resources`; - changing the gateway workload image. -If an explicitly selected older workload image does not contain `/kube-ovn/vpc-egress-gateway-observer`, the launcher runs `sleep infinity` instead, so the missing binary does not block the data plane. +If an explicitly selected older workload image does not contain `/kube-ovn/vpc-egress-gateway-observer`, the launcher runs `sleep infinity` instead, so the missing binary does not block the data plane. A custom image that does contain the observer must preserve its `CAP_NET_ADMIN` file capability; otherwise, interface metrics remain available but conntrack metrics and flow logs report collector errors. ## Metrics From 555103135bbf0bfa583ba9d2ea29a1208f47fd4c Mon Sep 17 00:00:00 2001 From: zhangzujian Date: Wed, 5 Aug 2026 18:05:18 +0800 Subject: [PATCH 3/6] docs: explain observer capability probe Signed-off-by: zhangzujian --- docs/vpc/vpc-egress-gateway-observability.en.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/vpc/vpc-egress-gateway-observability.en.md b/docs/vpc/vpc-egress-gateway-observability.en.md index f7f7cdbb4..35a8092cd 100644 --- a/docs/vpc/vpc-egress-gateway-observability.en.md +++ b/docs/vpc/vpc-egress-gateway-observability.en.md @@ -4,7 +4,7 @@ VPC Egress Gateway can run one native observability sidecar in every gateway Pod ## Requirements and Upgrade Notes -- Kubernetes 1.29 or later is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). If the Kubernetes version is older or cannot be discovered, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. +- Kubernetes 1.29 or later with the `SidecarContainers` feature enabled is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). The controller performs a server-side dry-run with a zero-replica Deployment and verifies that the API server preserves the restartable-init policy. If the version is too old, the feature is disabled, or the capability cannot be verified, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. Transient capability-probe errors are retried. - Upgrade the `vpc-egress-gateways.kubeovn.io` CRD explicitly before creating resources that use `spec.observability`. Helm does not upgrade CRDs that are already installed automatically. Apply the CRD delivered with the same Kube-OVN version, using your normal CRD upgrade procedure. - Conntrack collection requires `NET_ADMIN` in the gateway Pod network namespace. The official observer binary carries only the `CAP_NET_ADMIN` file capability, while the generated container security context admits only `NET_ADMIN` into the capability bounding set. `allowPrivilegeEscalation` is enabled so that this trusted file capability survives the non-root launcher `exec`; the observer still runs as UID and GID 65534, drops all other capabilities, and uses a read-only root filesystem. - Prometheus Operator is optional. The gateway works without the ServiceMonitor CRD. @@ -56,6 +56,7 @@ spec: ``` When `resources` is empty or omitted, the observer requests `20m` CPU and `64Mi` memory and is limited to `200m` CPU and `256Mi` memory. +The interface-only mode is continuously tested to remain at or below `20MiB` of steady-state resident memory; the larger default request leaves room for conntrack cache and log-queue growth. The controller stores the runtime configuration in a per-gateway ConfigMap. Collector switches, flow-log events, filters, and rate limits are reloaded without replacing gateway Pods. The following changes update the Deployment and replace Pods: From 4e1956b8f42cdd1d453df6c0f21e0209e59255bd Mon Sep 17 00:00:00 2001 From: zhangzujian Date: Wed, 5 Aug 2026 22:21:05 +0800 Subject: [PATCH 4/6] docs: explain observer liveness probe scope Signed-off-by: zhangzujian --- docs/vpc/vpc-egress-gateway-observability.en.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/vpc/vpc-egress-gateway-observability.en.md b/docs/vpc/vpc-egress-gateway-observability.en.md index 35a8092cd..ea9c01fc6 100644 --- a/docs/vpc/vpc-egress-gateway-observability.en.md +++ b/docs/vpc/vpc-egress-gateway-observability.en.md @@ -71,6 +71,8 @@ If an explicitly selected older workload image does not contain `/kube-ovn/vpc-e The observer serves `/metrics` and `/healthz` on TCP port `10666`. It uses a private Prometheus registry and does not expose `go_*`, `process_*`, or `promhttp_*` metrics. Every metric includes the identity labels `namespace`, `name`, `pod`, and `node`. +For a gateway in the default VPC, kubelet checks `/healthz` with an HTTP liveness probe. For a gateway in a custom VPC, the liveness probe executes the observer binary inside the container and checks the same endpoint over loopback. This avoids making observer health depend on node-to-Pod reachability into the custom VPC. + ### Interface Metrics Interface metrics are read from `/proc/net/dev` on every scrape. The observer resolves the primary interface and the external Multus interface from the Pod `network-status` annotation on the first successful scrape and caches those names for the lifetime of the process. From b7ddbd0211153224679d8c7119f47bee92964e2c Mon Sep 17 00:00:00 2001 From: zhangzujian Date: Thu, 6 Aug 2026 14:52:33 +0800 Subject: [PATCH 5/6] docs: clarify observer failure isolation Signed-off-by: zhangzujian --- docs/vpc/vpc-egress-gateway-observability.en.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/vpc/vpc-egress-gateway-observability.en.md b/docs/vpc/vpc-egress-gateway-observability.en.md index ea9c01fc6..d4aa8664d 100644 --- a/docs/vpc/vpc-egress-gateway-observability.en.md +++ b/docs/vpc/vpc-egress-gateway-observability.en.md @@ -4,7 +4,7 @@ VPC Egress Gateway can run one native observability sidecar in every gateway Pod ## Requirements and Upgrade Notes -- Kubernetes 1.29 or later with the `SidecarContainers` feature enabled is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). The controller performs a server-side dry-run with a zero-replica Deployment and verifies that the API server preserves the restartable-init policy. If the version is too old, the feature is disabled, or the capability cannot be verified, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. Transient capability-probe errors are retried. +- Kubernetes 1.29 or later with the `SidecarContainers` feature enabled is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). The controller performs a server-side dry-run with a zero-replica Deployment and verifies that the API server preserves the restartable-init policy. If the version is too old or the feature is disabled, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. Transient capability-probe errors are retried, and an existing observer remains in its last-known-good configuration while the capability cannot be verified. - Upgrade the `vpc-egress-gateways.kubeovn.io` CRD explicitly before creating resources that use `spec.observability`. Helm does not upgrade CRDs that are already installed automatically. Apply the CRD delivered with the same Kube-OVN version, using your normal CRD upgrade procedure. - Conntrack collection requires `NET_ADMIN` in the gateway Pod network namespace. The official observer binary carries only the `CAP_NET_ADMIN` file capability, while the generated container security context admits only `NET_ADMIN` into the capability bounding set. `allowPrivilegeEscalation` is enabled so that this trusted file capability survives the non-root launcher `exec`; the observer still runs as UID and GID 65534, drops all other capabilities, and uses a read-only root filesystem. - Prometheus Operator is optional. The gateway works without the ServiceMonitor CRD. @@ -65,13 +65,15 @@ The controller stores the runtime configuration in a per-gateway ConfigMap. Coll - changing `observability.resources`; - changing the gateway workload image. +If the capability probe or a ConfigMap or Service update fails transiently, the controller reports the observability condition and keeps the existing observer container, volumes, and auxiliary resources. Data-plane reconciliation continues without replacing a last-known-good gateway Pod because of an observability-only failure. + If an explicitly selected older workload image does not contain `/kube-ovn/vpc-egress-gateway-observer`, the launcher runs `sleep infinity` instead, so the missing binary does not block the data plane. A custom image that does contain the observer must preserve its `CAP_NET_ADMIN` file capability; otherwise, interface metrics remain available but conntrack metrics and flow logs report collector errors. ## Metrics The observer serves `/metrics` and `/healthz` on TCP port `10666`. It uses a private Prometheus registry and does not expose `go_*`, `process_*`, or `promhttp_*` metrics. Every metric includes the identity labels `namespace`, `name`, `pod`, and `node`. -For a gateway in the default VPC, kubelet checks `/healthz` with an HTTP liveness probe. For a gateway in a custom VPC, the liveness probe executes the observer binary inside the container and checks the same endpoint over loopback. This avoids making observer health depend on node-to-Pod reachability into the custom VPC. +For every gateway, the liveness probe executes the observer binary inside the container and checks `/healthz` over loopback. This keeps observer health independent of node-to-Pod reachability and lets the probe succeed when an older workload image uses the `sleep infinity` fallback because it does not contain the observer binary. ### Interface Metrics From a9a36d47c520daff51755bd5424c5561de74f339 Mon Sep 17 00:00:00 2001 From: zhangzujian Date: Mon, 10 Aug 2026 15:37:53 +0800 Subject: [PATCH 6/6] docs: complete VPC egress observability documentation Signed-off-by: zhangzujian --- docs/reference/kube-ovn-api.en.md | 16 +- docs/reference/kube-ovn-api.md | 83 +++++++++ .../vpc-egress-gateway-observability.en.md | 4 +- docs/vpc/vpc-egress-gateway-observability.md | 159 ++++++++++++++++++ docs/vpc/vpc-egress-gateway.md | 2 + mkdocs.yml | 2 + 6 files changed, 256 insertions(+), 10 deletions(-) create mode 100644 docs/vpc/vpc-egress-gateway-observability.md diff --git a/docs/reference/kube-ovn-api.en.md b/docs/reference/kube-ovn-api.en.md index 3213c1dec..e116c9ff0 100644 --- a/docs/reference/kube-ovn-api.en.md +++ b/docs/reference/kube-ovn-api.en.md @@ -819,15 +819,15 @@ Kubernetes 1.29 or later is required. For runtime behavior, metrics, flow-log sc | Property Name | Type | Description | | --- | --- | --- | -| recordsPerSecond | Int32 | Sustained records per second. Defaults to 100 | -| burst | Int32 | Maximum burst. Defaults to 1000 | +| recordsPerSecond | Int32 | Sustained records per second, from 1 through 100000. Defaults to 100 | +| burst | Int32 | Maximum burst, from 1 through 1000000. Defaults to 1000 | ##### VpcEgressGatewayConntrackLogFilters | Property Name | Type | Description | | --- | --- | --- | -| include | []VpcEgressGatewayConntrackLogFilter | Optional include rules. An empty list includes every NAT flow | -| exclude | []VpcEgressGatewayConntrackLogFilter | Optional exclude rules. Exclude matches take precedence | +| include | []VpcEgressGatewayConntrackLogFilter | Optional include rules, with at most 64 entries. An empty list includes every NAT flow | +| exclude | []VpcEgressGatewayConntrackLogFilter | Optional exclude rules, with at most 64 entries. Exclude matches take precedence | ##### VpcEgressGatewayConntrackLogFilter @@ -845,10 +845,10 @@ Configured fields in one rule are ANDed; values within a field and rules within | Property Name | Type | Description | | --- | --- | --- | -| sourceCIDRs | []String | Optional source CIDRs | -| destinationCIDRs | []String | Optional destination CIDRs | -| sourcePorts | []VpcEgressGatewayPortRange | Optional inclusive source-port ranges | -| destinationPorts | []VpcEgressGatewayPortRange | Optional inclusive destination-port ranges | +| sourceCIDRs | []String | Optional source CIDRs, with at most 64 entries | +| destinationCIDRs | []String | Optional destination CIDRs, with at most 64 entries | +| sourcePorts | []VpcEgressGatewayPortRange | Optional inclusive source-port ranges, with at most 64 entries | +| destinationPorts | []VpcEgressGatewayPortRange | Optional inclusive destination-port ranges, with at most 64 entries | ##### VpcEgressGatewayPortRange diff --git a/docs/reference/kube-ovn-api.md b/docs/reference/kube-ovn-api.md index bd8dba992..aad5dd2f3 100644 --- a/docs/reference/kube-ovn-api.md +++ b/docs/reference/kube-ovn-api.md @@ -741,6 +741,7 @@ | tolerations | []Toleration | 可选，标准 Kubernetes 容忍配置 | | resources | ResourceRequirements | 可选，容器资源限制；未指定时控制器使用默认值 | | bandwidth | BandwidthLimit | 可选，每个网关副本的入向/出向带宽限制，可使用整数 Mbps 或支持的 bit-rate quantity | +| observability | VpcEgressGatewayObservability | 可选，原生网卡指标、conntrack 指标、JSON 流日志和 ServiceMonitor 元数据；所有功能默认禁用 | ##### VpcEgressGatewaySelector @@ -781,6 +782,88 @@ | ingress | Int64 或 String | 从外部网络进入 VPC 的流量限制；整数和纯数字字符串的单位为 Mbps，quantity 使用 `M`、`Mi`、`G` 或 `Gi` bit/s | | egress | Int64 或 String | 从 VPC 发往外部网络的流量限制；整数和纯数字字符串的单位为 Mbps，quantity 使用 `M`、`Mi`、`G` 或 `Gi` bit/s | +##### VpcEgressGatewayObservability + +需要 Kubernetes 1.29 或更高版本。有关运行时行为、指标、流日志 schema 和运行限制，请参考 [VPC Egress Gateway 可观测性](../vpc/vpc-egress-gateway-observability.md)。 + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| resources | ResourceRequirements | 可选，observer sidecar 资源；默认 request 为 `20m` CPU 和 `64Mi` 内存，limit 为 `200m` CPU 和 `256Mi` 内存 | +| interfaceMetrics | VpcEgressGatewayObservabilityFeature | 网卡指标采集器配置 | +| conntrack | VpcEgressGatewayConntrackObservability | conntrack 指标和流日志配置 | +| serviceMonitor | VpcEgressGatewayServiceMonitor | 合并到每个网关 ServiceMonitor 的元数据 | + +##### VpcEgressGatewayObservabilityFeature + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| enabled | Boolean | 是否启用该功能，默认 false | + +##### VpcEgressGatewayConntrackObservability + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| metrics | VpcEgressGatewayObservabilityFeature | 低基数 NAT conntrack 指标配置 | +| log | VpcEgressGatewayConntrackLog | JSON Lines NAT 流日志配置 | + +##### VpcEgressGatewayConntrackLog + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| enabled | Boolean | 是否把 JSON 流日志写入 observer 容器标准输出，默认 false | +| events | []String | 可选，要输出的生命周期事件：`start` 和/或 `end`；省略时输出两者 | +| rateLimit | VpcEgressGatewayConntrackLogRateLimit | 每个 Pod 的流日志限速 | +| filters | VpcEgressGatewayConntrackLogFilters | 可选，include 和 exclude 规则 | + +##### VpcEgressGatewayConntrackLogRateLimit + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| recordsPerSecond | Int32 | 每秒持续记录数，范围为 1 至 100000，默认 100 | +| burst | Int32 | 最大突发记录数，范围为 1 至 1000000，默认 1000 | + +##### VpcEgressGatewayConntrackLogFilters + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| include | []VpcEgressGatewayConntrackLogFilter | 可选，最多 64 条 include 规则；空列表包含所有 NAT 流 | +| exclude | []VpcEgressGatewayConntrackLogFilter | 可选，最多 64 条 exclude 规则；exclude 匹配优先 | + +##### VpcEgressGatewayConntrackLogFilter + +同一规则内已配置的字段按 AND 组合；同一字段内的值以及同一列表内的规则按 OR 组合。 + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| addressFamilies | []String | 可选，`ipv4` 和/或 `ipv6` | +| protocols | []String | 可选，`tcp`、`udp`、`sctp`、`icmp`、`icmpv6` 和/或 `other` | +| natTypes | []String | 可选，`snat`、`dnat` 和/或 `snat_dnat` | +| original | VpcEgressGatewayConntrackTupleFilter | 原始 tuple 匹配 | +| translated | VpcEgressGatewayConntrackTupleFilter | 转换后 tuple 匹配 | + +##### VpcEgressGatewayConntrackTupleFilter + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| sourceCIDRs | []String | 可选，源 CIDR，最多 64 项 | +| destinationCIDRs | []String | 可选，目标 CIDR，最多 64 项 | +| sourcePorts | []VpcEgressGatewayPortRange | 可选，包含边界的源端口范围，最多 64 项 | +| destinationPorts | []VpcEgressGatewayPortRange | 可选，包含边界的目标端口范围，最多 64 项 | + +##### VpcEgressGatewayPortRange + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| start | Int32 | 包含边界的起始端口，范围为 0 至 65535 | +| end | Int32 | 包含边界的结束端口，范围为 `start` 至 65535 | + +##### VpcEgressGatewayServiceMonitor + +| 属性名称 | 类型 | 描述 | +| --- | --- | --- | +| labels | map[String]String | 可选，ServiceMonitor 标签；不能覆盖控制器必需的 selector 标签 | +| annotations | map[String]String | 可选，ServiceMonitor annotations | + #### VpcEgressGatewayStatus | 属性名称 | 类型 | 描述 | diff --git a/docs/vpc/vpc-egress-gateway-observability.en.md b/docs/vpc/vpc-egress-gateway-observability.en.md index d4aa8664d..db5a8980b 100644 --- a/docs/vpc/vpc-egress-gateway-observability.en.md +++ b/docs/vpc/vpc-egress-gateway-observability.en.md @@ -4,7 +4,7 @@ VPC Egress Gateway can run one native observability sidecar in every gateway Pod ## Requirements and Upgrade Notes -- Kubernetes 1.29 or later with the `SidecarContainers` feature enabled is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). The controller performs a server-side dry-run with a zero-replica Deployment and verifies that the API server preserves the restartable-init policy. If the version is too old or the feature is disabled, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. Transient capability-probe errors are retried, and an existing observer remains in its last-known-good configuration while the capability cannot be verified. +- Kubernetes 1.29 or later with the `SidecarContainers` feature enabled is required because the observer uses a [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/){: target="_blank" }. The controller performs a server-side dry-run with a zero-replica Deployment and verifies that the API server preserves the restartable-init policy. If the version is too old or the feature is disabled, the controller does not inject the observer and sets `ObservabilityConfigured=False`; the gateway data plane continues to reconcile normally. Transient capability-probe errors are retried, and an existing observer remains in its last-known-good configuration while the capability cannot be verified. - Upgrade the `vpc-egress-gateways.kubeovn.io` CRD explicitly before creating resources that use `spec.observability`. Helm does not upgrade CRDs that are already installed automatically. Apply the CRD delivered with the same Kube-OVN version, using your normal CRD upgrade procedure. - Conntrack collection requires `NET_ADMIN` in the gateway Pod network namespace. The official observer binary carries only the `CAP_NET_ADMIN` file capability, while the generated container security context admits only `NET_ADMIN` into the capability bounding set. `allowPrivilegeEscalation` is enabled so that this trusted file capability survives the non-root launcher `exec`; the observer still runs as UID and GID 65534, drops all other capabilities, and uses a read-only root filesystem. - Prometheus Operator is optional. The gateway works without the ServiceMonitor CRD. @@ -116,7 +116,7 @@ If the ServiceMonitor CRD is absent, the controller sets `ServiceMonitorReady=Fa Flow logs are JSON Lines written only to the `observability` container standard output. Diagnostics are written to standard error. Read the records with: -```shell +```bash kubectl logs -n default -c observability ``` diff --git a/docs/vpc/vpc-egress-gateway-observability.md b/docs/vpc/vpc-egress-gateway-observability.md new file mode 100644 index 000000000..7a0b98309 --- /dev/null +++ b/docs/vpc/vpc-egress-gateway-observability.md @@ -0,0 +1,159 @@ +# VPC Egress Gateway 可观测性 + +VPC Egress Gateway 可以在每个网关 Pod 中运行一个原生可观测性 sidecar。该 sidecar 可以暴露内部和外部网卡计数器、低基数 conntrack NAT 指标，并将版本化 NAT 流日志写入容器标准输出。这三项功能默认均禁用，不影响网关数据平面就绪状态。 + +## 使用要求和升级说明 + +- observer 使用 [restartable init container](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/){: target="_blank" }，因此需要 Kubernetes 1.29 或更高版本，并启用 `SidecarContainers` 特性。控制器会对一个零副本 Deployment 执行服务端 dry-run，并验证 API Server 是否保留 restartable init 策略。如果版本过低或未启用该特性，控制器不会注入 observer，并设置 `ObservabilityConfigured=False`；网关数据平面仍会正常协调。控制器会重试短暂的能力探测错误；无法验证能力时，已存在的 observer 保持最后一个已知良好配置。 +- 创建使用 `spec.observability` 的资源前，需要显式升级 `vpc-egress-gateways.kubeovn.io` CRD。Helm 不会自动升级已安装的 CRD。请使用常规 CRD 升级流程，应用与 Kube-OVN 版本匹配的 CRD。 +- conntrack 采集需要网关 Pod 网络命名空间中的 `NET_ADMIN`。官方 observer binary 仅带有 `CAP_NET_ADMIN` file capability，生成的容器 security context 仅允许 `NET_ADMIN` 进入 capability bounding set。`allowPrivilegeEscalation` 已启用，使该可信 file capability 在非 root launcher 执行 `exec` 后仍然生效；observer 仍以 UID 和 GID 65534 运行，丢弃其他所有 capability，并使用只读根文件系统。 +- Prometheus Operator 是可选依赖。缺少 ServiceMonitor CRD 时，网关仍可正常工作。 + +## 启用可观测性 + +以下示例启用两个指标采集器和 JSON 流日志： + +```yaml +apiVersion: kubeovn.io/v1 +kind: VpcEgressGateway +metadata: + name: gateway1 + namespace: default +spec: + vpc: ovn-cluster + replicas: 2 + externalSubnet: macvlan1 + policies: + - snat: true + subnets: + - ovn-default + observability: + resources: + requests: + cpu: 20m + memory: 64Mi + limits: + cpu: 200m + memory: 256Mi + interfaceMetrics: + enabled: true + conntrack: + metrics: + enabled: true + log: + enabled: true + events: [start, end] + rateLimit: + recordsPerSecond: 100 + burst: 1000 + filters: + include: [] + exclude: [] + serviceMonitor: + labels: + monitoring: platform + annotations: {} +``` + +`resources` 为空或省略时，observer 的 request 为 `20m` CPU 和 `64Mi` 内存，limit 为 `200m` CPU 和 `256Mi` 内存。 +interface-only 模式会持续测试，以确保稳态常驻内存不超过 `20MiB`；更大的默认 request 为 conntrack cache 和日志队列增长保留了空间。 + +控制器将运行时配置存储在每个网关的 ConfigMap 中。采集器开关、流日志事件、filter 和 rate limit 可在不替换网关 Pod 的情况下热加载。以下更改会更新 Deployment 并替换 Pod： + +- 首次启用可观测性。 +- 禁用所有可观测性功能。 +- 更改 `observability.resources`。 +- 更改网关工作负载镜像。 + +如果能力探测、ConfigMap 或 Service 更新短暂失败，控制器会报告可观测性 condition，并保留现有 observer 容器、volume 和辅助资源。数据平面协调会继续，不会因为可观测性错误替换使用最后一个已知良好配置的网关 Pod。 + +如果显式选择的旧工作负载镜像不包含 `/kube-ovn/vpc-egress-gateway-observer`，launcher 会改为执行 `sleep infinity`，因此缺少 binary 不会阻塞数据平面。包含 observer 的自定义镜像必须保留其 `CAP_NET_ADMIN` file capability；否则网卡指标仍可用，但 conntrack 指标和流日志会报告采集器错误。 + +## 指标 + +observer 在 TCP 端口 `10666` 上提供 `/metrics` 和 `/healthz`。observer 使用私有 Prometheus registry，不暴露 `go_*`、`process_*` 或 `promhttp_*` 指标。每个指标都包含 `namespace`、`name`、`pod` 和 `node` 身份标签。 + +对于每个网关，liveness probe 会在容器内执行 observer binary，并通过 loopback 检查 `/healthz`。这使 observer 健康状态不受节点到 Pod 的可达性影响，也使旧工作负载镜像在缺少 observer binary 并使用 `sleep infinity` 回退时能够通过探测。 + +### 网卡指标 + +每次 scrape 都会从 `/proc/net/dev` 读取网卡指标。observer 在首次成功 scrape 时通过 Pod `network-status` annotation 解析主网卡和外部 Multus 网卡，并在进程生命周期内缓存这些名称。 + +| 指标 | 附加标签 | 描述 | +| :--- | :--- | :--- | +| `kube_ovn_vpc_egress_gateway_interface_rx_bytes_total` | `interface`、`type` | 接收字节数。 | +| `kube_ovn_vpc_egress_gateway_interface_tx_bytes_total` | `interface`、`type` | 发送字节数。 | +| `kube_ovn_vpc_egress_gateway_interface_rx_packets_total` | `interface`、`type` | 接收报文数。 | +| `kube_ovn_vpc_egress_gateway_interface_tx_packets_total` | `interface`、`type` | 发送报文数。 | +| `kube_ovn_vpc_egress_gateway_interface_drops_total` | `interface`、`type`、`direction` | 丢弃报文数。 | +| `kube_ovn_vpc_egress_gateway_interface_errors_total` | `interface`、`type`、`direction` | 报文错误数。 | + +`type` 标签为 `internal` 或 `external`，`direction` 为 `rx` 或 `tx`。 + +### Conntrack 指标 + +Conntrack 指标不使用 IP 地址或端口作为标签，仅对流量进行汇总： + +| 指标 | 附加标签 | 描述 | +| :--- | :--- | :--- | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_flows_active` | `address_family`、`protocol`、`nat_type` | 当前 NAT 流数，从 conntrack table dump 初始化并通过事件维护。 | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_flows_started_total` | `address_family`、`protocol`、`nat_type` | observer 订阅后观测到的 NAT 流启动总数。 | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_flows_ended_total` | `address_family`、`protocol`、`nat_type` | 观测到的 NAT 流结束事件总数。 | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_packets_total` | `address_family`、`protocol`、`nat_type`、`direction` | conntrack accounting 可用时观测到的报文数。 | +| `kube_ovn_vpc_egress_gateway_conntrack_nat_bytes_total` | `address_family`、`protocol`、`nat_type`、`direction` | conntrack accounting 可用时观测到的字节数。 | + +`address_family` 为 `ipv4` 或 `ipv6`。已知 `protocol` 值为 `tcp`、`udp`、`sctp`、`icmp` 和 `icmpv6`；其他协议号均汇总为 `other`。`nat_type` 为 `snat`、`dnat` 或 `snat_dnat`，`direction` 为 `original` 或 `reply`。 + +运行指标使用 `kube_ovn_vpc_egress_gateway_observability_*` 前缀。它们报告采集器可用性、配置重新加载结果、conntrack 事件和错误、cache 条目／容量／驱逐、已输出和已丢弃日志记录，以及 conntrack accounting 可用性。一个采集器失败不会导致 `/metrics` 失败，也不会隐藏另一个采集器的数据。 + +## ServiceMonitor 发现 + +启用任一指标采集器时，控制器会为每个网关创建一个设置了 `publishNotReadyAddresses: true` 的 headless Service，其数字 `targetPort` 为 `10666`。控制器还会在网关所在命名空间创建 ServiceMonitor。`observability.serviceMonitor` 中的用户 label 和 annotation 会合并到 ServiceMonitor 元数据，但不能覆盖控制器必需的 selector label。 + +如果 ServiceMonitor CRD 不存在，控制器会设置 `ServiceMonitorReady=False` 并定期重试。CRD 可用后，控制器会自动创建 ServiceMonitor。该 condition 与现有网关 `Ready` condition 相互独立。 + +## JSON 流日志 + +流日志以 JSON Lines 形式仅写入 `observability` 容器标准输出，诊断信息写入标准错误。使用以下命令读取记录： + +```bash +kubectl logs -n default -c observability +``` + +`v1` schema 输出 `start` 和 `end` 生命周期记录，不记录原始 conntrack update 事件。记录示例如下： + +```json +{"schemaVersion":"v1","timestamp":"2026-08-05T03:40:00Z","event":"end","conntrackID":2471802592,"zone":0,"namespace":"default","name":"gateway1","pod":"gateway1-7c9b8f8db7-r9x8m","node":"worker-1","addressFamily":"ipv4","protocol":"tcp","protocolNumber":6,"natType":["snat"],"original":{"sourceIP":"10.16.0.25","sourcePort":42136,"destinationIP":"203.0.113.10","destinationPort":443},"translated":{"sourceIP":"172.17.0.11","sourcePort":42136,"destinationIP":"203.0.113.10","destinationPort":443},"counters":{"originalPackets":8,"originalBytes":624,"replyPackets":7,"replyBytes":591}} +``` + +对于汇总为 `other` 的协议，`protocolNumber` 保留原始 IP 协议号。`natType` 是稳定排序的数组，可同时包含 `dnat` 和 `snat`。conntrack accounting 不可用时省略 `counters` 对象；observer 从不更改 `nf_conntrack_acct`。 + +observer 在 dump 现有 conntrack table 前订阅 conntrack 事件。现有 NAT 流会初始化 active-flow 指标和 cache，但不会生成历史 `start` 日志。cache 限制为 65,536 个条目。netlink 事件 channel 和异步日志队列均限制为 4,096 个条目。容器日志 consumer 阻塞或日志队列已满时，系统会丢弃日志记录，且不会阻塞 conntrack 采集。 + +默认情况下，系统记录所有 NAT 流和两种生命周期事件，每个 Pod 限制为每秒 100 条记录，最大突发记录数为 1,000。这些记录和指标是 best-effort 运行数据，不适用于计费精度的计量。 + +## 流日志 Filter + +每条 include 或 exclude 规则可以匹配地址族、协议、NAT 类型以及原始或转换后 tuple 的字段。同一字段内的值按 OR 组合，同一规则内已配置的字段按 AND 组合，同一列表内的规则按 OR 组合。exclude 规则优先于 include 规则。空 include 列表匹配所有 NAT 流。 + +端口使用包含边界的结构化范围。以下配置会记录从 `10.16.0.0/16` 到目标端口 80 至 443 的 TCP SNAT 流，但排除转换后目标地址位于 `192.0.2.0/24` 的流： + +```yaml +spec: + observability: + conntrack: + log: + enabled: true + filters: + include: + - protocols: [tcp] + natTypes: [snat] + original: + sourceCIDRs: [10.16.0.0/16] + destinationPorts: + - start: 80 + end: 443 + exclude: + - translated: + destinationCIDRs: [192.0.2.0/24] +``` diff --git a/docs/vpc/vpc-egress-gateway.md b/docs/vpc/vpc-egress-gateway.md index f031dcbd4..77347e1bf 100644 --- a/docs/vpc/vpc-egress-gateway.md +++ b/docs/vpc/vpc-egress-gateway.md @@ -20,6 +20,8 @@ VPC Egress Gateway 用于控制 VPC（包括默认 VPC）内 Pod 使用一组固 VPC Egress Gateway 还支持通过 BGP 和 EVPN/VXLAN 与外部网络建立动态路由，详情请参考 [Egress Gateway BGP/EVPN 支持](./vpc-egress-gateway-bgp-evpn.md)。 +如需使用网卡指标、conntrack NAT 指标和 JSON 流日志，请参考 [VPC Egress Gateway 可观测性](./vpc-egress-gateway-observability.md)。 + 同时 VPC Egress Gateway 具有如下限制： - 使用 Macvlan 实现底层网络打通，需要底层网络[支持 Underlay](../start/underlay.md#_2) diff --git a/mkdocs.yml b/mkdocs.yml index 16c4102df..4fb45b677 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -41,6 +41,7 @@ nav: - VPC 网络: - VPC 入门: "vpc/vpc.md" - Egress Gateway: "vpc/vpc-egress-gateway.md" + - Egress Gateway 可观测性: "vpc/vpc-egress-gateway-observability.md" - Egress Gateway BGP/EVPN: "vpc/vpc-egress-gateway-bgp-evpn.md" - Egress QoS 配置: "vpc/vpc-qos.md" - 内部负载均衡: "vpc/vpc-internal-lb.md" @@ -192,6 +193,7 @@ plugins: DHCP: DHCP VPC 入门: VPC Introduction Egress Gateway: Egress Gateway + Egress Gateway 可观测性: Egress Gateway Observability Egress Gateway BGP/EVPN: Egress Gateway BGP/EVPN Egress QoS 配置: Egress QoS Configuration VPC 使用: Config VPC