[Feature] Add nodes affinity / Toleration to KFP/Ray nodes

[roytman]

Search before asking

• I searched the issues and found no similar issues.

Component

KFP workflows, Library/kfp

Feature

Some Kubernetes deployments require that KFP and Ray Pods deploy on a certain set of nodes.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[blublinsky]

This is supported by the API server through compute template definition https://github.com/ray-project/kuberay/blob/master/proto/config.proto#L130. Pod toleration is here https://github.com/ray-project/kuberay/blob/master/proto/config.proto#L104C9-L104C17. Documentation on toleration usage is here https://medium.com/@muppedaanvesh/kubernetes-taints-tolerations-b0e0ed076cad

[roytman]

I know, but in order to use it we have to update our KFP library.
On the other hand, KFP supports affinity for its Pods, but again we have to update the pipelines.

[blublinsky]

I was making sure API server supports it

[roytman]

Done with #627, #643 and #648