tls_irule fails if pool has no active members

[ryanmilne-digicert]

Setup Details

CIS Version : 2.18.0
Build: f5networks/k8s-bigip-ctlr:2.18.0
BIGIP Version: Big IP 17.1.1
AS3 Version: 3.5
Agent Mode: AS3/CCCL
Orchestration: K8S
Orchestration Version: 1.27.15
Pool Mode: Cluster
Additional Setup details: Rancher RKE2 w/ Canal

Description

I created a virtual server and pool for an application which runs in a remote cluster. The virtual server and pool are created and work correctly unless none of the pods in the pool are active. In this case the connecting client receives a Connection reset by peer. Looking at the logs from the BigIP I can see that the IRule associated with the virtual server is failing.

01220001:3: TCL error: /f5-bigip-ctlr/Shared/main_vs_443_tls_irule <CLIENTSSL_DATA> - Illegal argument. Can't execute in the current context. (line 103)     invoked from within "HTTP::respond 503"    (iRule proc "/f5-bigip-ctlr/Shared/main_vs_443_tls_irule::select_ab_pool") (line 39)     invoked from within "call select_ab_pool $servername_lower $dflt_pool $domainpath"

It's clear failing when trying to execute HTTP::respond 503, but I am unsure why.

Steps To Reproduce

1. Create a virtual server and pool with one or more members.
2. Disable all members in the pool
3. Try connecting to the service using curl.

Expected Result

I expect to receive a 503 status code when there are no active members of a pool.

Actual Result

Instead the tcp connection is reset and the following is logged by the BigIP.

01220001:3: TCL error: /f5-bigip-ctlr/Shared/main_vs_443_tls_irule <CLIENTSSL_DATA> - Illegal argument. Can't execute in the current context. (line 103)     invoked from within "HTTP::respond 503"    (iRule proc "/f5-bigip-ctlr/Shared/main_vs_443_tls_irule::select_ab_pool") (line 39)     invoked from within "call select_ab_pool $servername_lower $dflt_pool $domainpath"

[vklohiya]

@ryanmilne-digicert , Please share the VS definition for further debugging.

[ryanmilne-digicert]

This is virtual server manifest:

apiVersion: "cis.f5.com/v1"
kind: VirtualServer
metadata:
  name: test-virtual-server
  namespace: f5-system
  labels:
    f5cr: "true"
spec:
  host: test.example.com
  hostAliases:
  - test.example.net
  hostGroup: "main"
  hostGroupVirtualServerName: "main_vs"
  virtualServerAddress: "10.60.25.41"
  tlsProfileName: stage-ssl
  httpTraffic: allow
  pools:
  - path: /
    serviceNamespace: test
    service: test-stable
    servicePort: 80
    monitor:
      type: http
      interval: 5
      recv: "OK"
      send: "GET /test/health/liveness HTTP/1.1\r\nHost: test.example.com\r\n"
      timeout: 10
    extendedServiceReferences:
    - clusterName: test-cluster2
      namespace: test
      servicePort: 80
      service: test-stable

[ryanmilne-digicert]

After more investigation, I think I can explain more what is going wrong.

Since I do not have a defaultPool defined and when the other pool does not have any active members then the iRule execution reaches the instruction to return a 503 status code.

proc select_ab_pool {path default_pool domainpath} {
    set last_slash [string length $path]
    set ab_class "/f5-bigip-ctlr/Shared/main_vs_443_ab_deployment_dg"
    while {$last_slash >= 0} {
        if {[class match $path equals $ab_class]} then {
            break
        }
        set last_slash [string last "/" $path $last_slash]
        incr last_slash -1
        set path [string range $path 0 $last_slash]
    }
    if {$last_slash >= 0} {
        set ab_rule [class match -value $path equals $ab_class]
        if {$ab_rule != ""} then {
            set weight_selection [expr {rand()}]
            set service_rules [split $ab_rule ";"]
            set active_pool ""
            foreach service_rule $service_rules {
                set fields [split $service_rule ","]
                set pool_name [lindex $fields 0]
                if { [active_members $pool_name] >= 1 } {
                    set active_pool $pool_name
                }
                set weight [expr {double([lindex $fields 1])}]
                if {$weight_selection <= $weight} then {
                    #check if active pool members are available
                    if { [active_members $pool_name] >= 1 } {
                        return $pool_name
                    } else {
                        # select the any of pool with active members
                        if {$active_pool!= ""} then {
                            return $active_pool
                        }
                    }
                }
            }
        }
        # If we had a match, but all weights were 0 then
        # retrun a 503 (Service Unavailable)
        HTTP::respond 503
    }
    return $default_pool
}

HTTP::xxx commands have to be used with WHEN REQUEST in an iRule and not with WHEN CLIENT_xxx or WHEN_CLIENTSSL_xxx.

[trinaths]

Created [CONTCNTR-4951] for internal tracking.

[vidyasagar-m]

@ryanmilne-digicert Could you please verify the above issue with this build quay.io/mvsr222/k8s-bigip-ctlr:issue3570 and provide us the feedback.

[ryanmilne-digicert]

@vidyasagar-m I will test this and report back.