Ray on Spark that is running on YARN and Kubernetes

[ericbugin1]

Description

Is the ray.util.spark import setup_ray_cluster function going to be available for Spark clusters that are running on YARN and/or K8?

This would be very useful, for it is all that I currently have available at my organization.

Looking to expand my use of Ray to utilizing full clusters.

Use case

No response

[jjyao]

@WeichenXu123 could you take this?

[WeichenXu123]

We only test ray.util.spark on spark standalone cluster , we haven't tested on spark on Yarn / K8S environment. It might not work. You can test if you want to try.

But I suggest if you have K8s, use KuberRay instead. Simpler architecture, and less overhead.

and Yarn seems to be something out of date, so we do not plan to invest it.