Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ray on Spark that is running on YARN and Kubernetes #47716

Open
ericbugin1 opened this issue Sep 17, 2024 · 2 comments
Open

Ray on Spark that is running on YARN and Kubernetes #47716

ericbugin1 opened this issue Sep 17, 2024 · 2 comments
Labels
core Issues that should be addressed in Ray Core core-clusters For launching and managing Ray clusters/jobs/kubernetes enhancement Request for new feature and/or capability P2 Important issue, but not time-critical

Comments

@ericbugin1
Copy link

Description

Is the ray.util.spark import setup_ray_cluster function going to be available for Spark clusters that are running on YARN and/or K8?

This would be very useful, for it is all that I currently have available at my organization.

Looking to expand my use of Ray to utilizing full clusters.

Use case

No response
@ericbugin1 ericbugin1 added enhancement Request for new feature and/or capability triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Sep 17, 2024
@anyscalesam anyscalesam added the core Issues that should be addressed in Ray Core label Sep 24, 2024
@jjyao
Copy link
Collaborator

jjyao commented Oct 7, 2024

@WeichenXu123 could you take this?

@jjyao jjyao added core-clusters For launching and managing Ray clusters/jobs/kubernetes P2 Important issue, but not time-critical and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Oct 7, 2024
@WeichenXu123
Copy link
Contributor

WeichenXu123 commented Oct 8, 2024 •
edited
Loading

We only test ray.util.spark on spark standalone cluster , we haven't tested on spark on Yarn / K8S environment. It might not work. You can test if you want to try.

But I suggest if you have K8s, use KuberRay instead. Simpler architecture, and less overhead.

and Yarn seems to be something out of date, so we do not plan to invest it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core Issues that should be addressed in Ray Core core-clusters For launching and managing Ray clusters/jobs/kubernetes enhancement Request for new feature and/or capability P2 Important issue, but not time-critical
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jjyao @WeichenXu123 @anyscalesam @ericbugin1