Flaky: pipelines installation readiness wait expires when the KinD API server degrades

[danish9039]

Problem

Workflows that call tests/pipelines_install.sh fail intermittently. The
visible symptom is the readiness wait in tests/pipelines_install.sh expiring
after six hundred seconds while reporting twelve pods as not ready:

timed out waiting for the condition on pods/ml-pipeline-6848b5d4d7-pcd2v
timed out waiting for the condition on pods/ml-pipeline-persistenceagent-646f88694b-gh5sc
timed out waiting for the condition on pods/ml-pipeline-scheduledworkflow-f4994d98-sqbcq
timed out waiting for the condition on pods/ml-pipeline-ui-6f6789c496-sn7b4
timed out waiting for the condition on pods/ml-pipeline-viewer-crd-74d9fbf5cd-xw9bj
timed out waiting for the condition on pods/ml-pipeline-visualizationserver-787f4cb9f7-jt969
timed out waiting for the condition on pods/mysql-5b6cf8556-kx2rr
timed out waiting for the condition on pods/seaweedfs-5f6996f6f-kfprv
timed out waiting for the condition on pods/workflow-controller-cd549fb55-gwt7f
timed out waiting for the condition on pods/local-path-provisioner-855c7b7774-j5b72
timed out waiting for the condition on pods/oauth2-proxy-55765f6898-95fm4
timed out waiting for the condition on pods/oauth2-proxy-55765f6898-9kwrj
##[error]Process completed with exit code 1.

That message is misleading. kubectl wait --all --all-namespaces reports every
pod it was still watching, not the pods that are actually broken. Diagnostics
collected at the moment of failure show that only one or two pods were ever
unhealthy.

The timeout is not the cause

The wait budget is six hundred seconds. A healthy run completes the same wait in
about thirty five seconds, measured from a sampled green run: the pipelines
installation step ran from 07:56:37 to 07:58:03, and sixty of those eighty six
seconds are the hardcoded sleep 60 on the line above. The margin is roughly
seventeen times. Raising the timeout would not fix this and would only make
the failure slower to surface.

Root cause

Calls into cluster critical services do not complete while roughly a dozen pods
and their Istio sidecars start at once. Two different services have been
observed failing this way, so this is not specific to one component.

The Kubernetes API server. workflow-controller cannot complete a TLS
handshake against the API server service address:

level=ERROR msg="Failed to register watch for controller config map"
error="Get \"https://10.96.0.1:443/api/v1/namespaces/kubeflow/configmaps/workflow-controller-configmap\": net/http: TLS handshake timeout"

ml-pipeline-api-server fails its startup probe for the same reason and is
terminated by the kubelet:

Warning  Unhealthy  Startup probe failed: HTTP probe failed with statuscode: 500
Normal   Killing    Container ml-pipeline-api-server failed startup probe, will be restarted

Last State:   Terminated
  Reason:     Error
  Exit Code:  137
Ready:          False
Restart Count:  6

Its startup budget is failureThreshold: 12 multiplied by periodSeconds: 5,
that is sixty seconds, defined in
applications/pipeline/upstream/base/pipeline/ml-pipeline-apiserver-deployment.yaml.
Every termination restarts initialisation from the beginning, so the loop
sustains itself. All three observed occurrences reached exactly six restarts.

The Istio injection webhook. In a separate occurrence the dashboard pod was
never created at all, because admission timed out:

Error creating: Internal error occurred: failed calling webhook
"namespace.sidecar-injector.istio.io": failed to call webhook:
Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

Correction to an earlier version of this issue

This issue previously stated the cause as API server degradation, citing
kube-apiserver ... Readiness probe failed: HTTP probe failed with statuscode: 500. That event appears in two of the four captured failures and is absent
from the others, so it is a symptom that sometimes accompanies the problem
rather than the cause. Whether the underlying trigger is API server slowness or
an Istio sidecar network path that is not ready when the application starts is
still undetermined.

A misleading symptom

kubectl wait --for=condition=Ready pods --all --all-namespaces reports every
pod it was still watching, not the pods that are broken. It names about twelve
pods while diagnostics show only one to four are genuinely unhealthy, which
sends investigations toward innocent components such as
local-path-provisioner and oauth2-proxy.

Evidence from four captured failures

The first two were reproduced deliberately by re-triggering the affected
workflows. The third and fourth were captured automatically on unrelated pull
requests once the diagnostics were merged, which shows this affects whatever
work is in flight.

Run	Pull request	Workflow and failing step	Signature
30247776370	3560	pipeline_run_from_notebook, Install KF Pipelines	ml-pipeline 1/2 ready, exit 137, 6 restarts
30259363189	3560	pipeline_run_from_notebook, Install KF Pipelines	same, plus workflow-controller crash loop, 6 restarts
30433986686	3524	workspaces_pipeline_run_test, Install KF Pipelines	same, plus ml-pipeline-scheduledworkflow, ml-pipeline-persistenceagent, metadata-grpc
30437973166	3566	pipeline_run_from_notebook, Install Dashboard	dashboard and profiles pods never created, istiod webhook timed out

The ml-pipeline container reached exactly six restarts in every occurrence
where it was involved. Persistent volume claims were Bound and no node
reported a pressure condition in every run where those were captured.

What the evidence rules out

• Memory exhaustion. Available memory never fell below 10458 MiB of 15 GiB
  in a sampled run, no node reported MemoryPressure, DiskPressure or
  PIDPressure, and the terminated container reports Reason: Error rather
  than OOMKilled.
• Processor starvation as the direct trigger. Load average at the point of
  failure was 0.60 over one minute and 1.29 over fifteen minutes on four
  processors.
• A storage provisioner cascade. Both persistent volume claims were Bound
  in both failures, and claims bind in about eleven seconds on a healthy run.
• An insufficient wait budget. See the measurement above.

Proposed work

• Reduce control-plane pressure during installation, for example by
  staggering component installation rather than applying a whole stack at
  once.
• Narrow the readiness wait in tests/pipelines_install.sh so that it
  reports the genuinely unhealthy pod. The current
  kubectl wait --for=condition=Ready pods --all --all-namespaces blocks on
  every pod in the cluster, so one unhealthy pod fails the job while naming
  eleven healthy ones, which sends every investigation in the wrong
  direction.
• Review the inconsistent wait budgets across the installation scripts.
  tests/dashboard_install.sh allows sixty seconds for a rollout while
  tests/pipelines_install.sh allows six hundred seconds for readiness. The
  sixty second budget is tighter than any measured healthy duration.
• Evaluate whether the ml-pipeline-api-server startup budget of sixty
  seconds is sufficient for a cold cluster. That file is synchronized from
  kubeflow/pipelines under applications/pipeline/upstream, so it must
  not be edited directly. It requires either an overlay in this repository
  or a change accepted upstream.
• Consider whether components should tolerate a temporarily unavailable
  control plane rather than crash-looping, both workflow-controller
  against the API server and admission against istiod.
• Determine whether the trigger is control plane slowness or an Istio
  sidecar network path that is not ready when the application starts, since
  the fix differs.

Related

• Diagnostics that produced this evidence:
  ci: collect cluster diagnostics for failing component tests #3560
• Job level timeouts added after this investigation:
  ci: set a job level timeout on the KinD based workflows #3566

[danish9039]

Updated the description with two further occurrences and a correction.

The diagnostics merged in #3560 captured two more failures automatically, on pull requests 3524 and 3566, neither of which was under investigation. That confirms this affects unrelated work in flight rather than any one change.

The correction matters for anyone picking this up. The description previously named API server degradation as the cause, citing a kube-apiserver readiness probe returning 500. That event is present in two of the four captured failures and absent from the others, so it is a symptom that sometimes accompanies the problem rather than the cause. The fourth failure also implicates a different service entirely: the dashboard and profiles pods were never created because the Istio injection webhook timed out with context deadline exceeded, not because of anything on the API server.

What holds across all four is that calls with a ten second budget into cluster critical services do not complete while roughly a dozen pods and their sidecars start at once. Whether the trigger is control plane slowness or a sidecar network path that is not ready when the application starts is still open, and the fix differs between the two.

[juliusvonkohout]

It might be that the networking pods had too low limits. Please align with @jeffspahr on that, he fixed it for KFP.

[jeffspahr]

This looks closely related to the Kind networking failure we investigated in kubeflow/pipelines, especially the kindnet CPU quota.

The configuration matches: this repository creates Kind v0.32.0 / Kubernetes v1.36.1 clusters and currently leaves kindnet's default 100m CPU limit in place. KFP uses the same Kind generation, but now removes that limit while retaining the 100m scheduling request:

https://github.com/kubeflow/pipelines/blob/master/.github/actions/create-cluster/action.yml#L64-L83

The failure signature is also similar:

• Kubernetes API ClusterIP TLS handshakes time out under bursty startup.
• Multiple unrelated in-cluster Services become unreachable at the same time.
• Pods and endpoints otherwise appear healthy.
• There is no node pressure or memory-exhaustion evidence.
• Low overall runner load does not rule this out: a container capped at 100m can be heavily throttled while the host still has idle CPU.

In the KFP investigation, workload-matched capped runs showed kindnet throttled during 97–99.7% of CPU periods, with NFQUEUE depths of 384–521 packets and simultaneous Service-VIP/direct-Endpoint outages. The comparable uncapped run passed, with zero kindnet throttling and a maximum NFQUEUE depth of 2:

kubeflow/pipelines#13748 (comment)

After removing the quota, repeated full matrices remained green as concurrency was restored. The completed KFP investigation is tracked in kubeflow/pipelines#13748 and the change landed in kubeflow/pipelines#13739.

I would treat this as the leading hypothesis, but not confirmed yet. The current captures show the kindnet pods as Running, which does not reveal cgroup throttling, and they do not include kindnet cpu.stat, NFQUEUE state, or paired Service/Endpoint probes.

A targeted experiment would be to apply the same patch immediately after Kind creation:

kubectl -n kube-system patch daemonset/kindnet --type=strategic \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"kindnet-cni","resources":{"requests":{"cpu":"100m"},"limits":{"cpu":null}}}]}}}}'

kubectl -n kube-system rollout status daemonset/kindnet --timeout=180s

Then repeat the affected workflows at the same revision. Ideally, retain a capped control run and capture per-kindnet cgroup cpu.stat, /proc/net/netfilter/nfnetlink_queue, recent kindnet logs, and same-source Service-VIP/direct-Endpoint probes. That would distinguish this quota-driven NFQUEUE failure from Istio or API-server overload.

Staggering installation may reduce the triggering burst, but removing the quota is the more targeted first experiment. Increasing readiness timeouts would not address this mechanism.

[jeffspahr]

^ That was a really really challenging one to chase down. Hopefully this is helpful.

Thanks for tagging me @juliusvonkohout .