Skip to content

Commit 8487e95

Browse files
gibchikafaSirOibaf
gibchikafa
and
SirOibaf
authored
[HWORKS-1670] Ray documentation (#422)
* [HWORKS-1670] Ray documentation Co-authored-by: Fabio Buso <[email protected]>
1 parent 7ed7540 commit 8487e95

18 files changed

+357
-2
lines changed

‎docs/assets/images/guides/jobs/create_new_job.png

136 KB
Loading

‎docs/assets/images/guides/jobs/jobs_overview.png

22.2 KB
Loading

‎docs/assets/images/guides/jobs/ray_application_logs.png

145 KB
Loading

‎docs/assets/images/guides/jobs/ray_dashboard.png

152 KB
Loading

‎docs/assets/images/guides/jobs/ray_execution.png

261 KB
Loading

‎docs/assets/images/guides/jobs/ray_runtime_env_and_additional_files.png

153 KB
Loading

‎docs/assets/images/guides/jobs/resource_configuration_ray.png

135 KB
Loading

‎docs/assets/images/guides/jobs/start_ray_job.png

337 KB
Loading

‎docs/assets/images/guides/jobs/upload_ray_script.png

250 KB
Loading

‎docs/assets/images/guides/jupyter/environment_list.png

227 KB
Loading

‎docs/assets/images/guides/jupyter/jupyter_overview.png

312 KB
Loading

‎docs/assets/images/guides/jupyter/ray_jupyter_notebook_session.png

232 KB
Loading

‎docs/assets/images/guides/jupyter/ray_kernel.png

239 KB
Loading

‎docs/assets/images/guides/jupyter/ray_resource_and_compute.png

116 KB
Loading

‎docs/user_guides/projects/jobs/ray_job.md

+210
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
---
2+
description: Documentation on how to configure and execute a Ray job on Hopsworks.
3+
---
4+
5+
# How To Run A Ray Job
6+
7+
## Introduction
8+
9+
All members of a project in Hopsworks can launch the following types of applications through a project's Jobs service:
10+
11+
- Python (*Hopsworks Enterprise only*)
12+
- Apache Spark
13+
- Ray
14+
15+
Launching a job of any type is very similar process, what mostly differs between job types is
16+
the various configuration parameters each job type comes with. Hopsworks support scheduling to run jobs on a regular basis,
17+
e.g backfilling a Feature Group by running your feature engineering pipeline nightly. Scheduling can be done both through the UI and the python API,
18+
checkout [our Scheduling guide](schedule_job.md).
19+
20+
!!!warning "Enable Ray"
21+
22+
Support for Ray needs to be explicitly enabled by adding the following option in the `values.yaml` file for the deployment:
23+
24+
```yaml
25+
global:
26+
ray:
27+
enabled: true
28+
```
29+
30+
## UI
31+
32+
### Step 1: Jobs overview
33+
34+
The image below shows the Jobs overview page in Hopsworks and is accessed by clicking `Jobs` in the sidebar.
35+
36+
<p align="center">
37+
<figure>
38+
<img src="../../../../assets/images/guides/jobs/jobs_overview.png" alt="Jobs overview">
39+
<figcaption>Jobs overview</figcaption>
40+
</figure>
41+
</p>
42+
43+
### Step 2: Create new job dialog
44+
45+
Click `New Job` and the following dialog will appear.
46+
47+
<p align="center">
48+
<figure>
49+
<img src="../../../../assets/images/guides/jobs/create_new_job.png" alt="Create new job dialog">
50+
<figcaption>Create new job dialog</figcaption>
51+
</figure>
52+
</p>
53+
54+
### Step 3: Set the job type
55+
56+
By default, the dialog will create a Spark job. Make sure `RAY` is chosen.
57+
58+
### Step 4: Set the script
59+
60+
Next step is to select the program to run. You can either select `From project`, if the file was previously uploaded to Hopsworks, or `Upload new file` which lets you select a file from your local filesystem as demonstrated below. After that set the name for the job. By default, the job name is the same as the file name, but you can customize it here.
61+
62+
<p align="center">
63+
<figure>
64+
<img src="../../../../assets/images/guides/jobs/upload_ray_script.png" alt="Configure program">
65+
<figcaption>Configure program</figcaption>
66+
</figure>
67+
</p>
68+
69+
### Step 5 (optional): Advanced configuration
70+
71+
Resource allocation for the Driver and Workers can be configured.
72+
73+
!!! notice "Using the resources in the Ray script"
74+
The resource configurations describe the cluster that will be provisioned when launching the Ray job. User can still
75+
provide extra configurations in the job script using `ScalingConfig`, i.e. `ScalingConfig(num_workers=4, trainer_resources={"CPU": 1}, use_gpu=True)`.
76+
77+
* `Driver memory`: Memory in MBs to allocate for Driver
78+
79+
* `Driver virtual cores`: Number of cores to allocate for the Driver
80+
81+
* `Worker memory`: Memory in MBs to allocate for each worker
82+
83+
* `Worker cores`: Number of cores to allocate for each worker
84+
85+
* `Min workers`: Minimum number of workers to start with
86+
87+
* `Max workers`: Maximum number of workers to scale up to
88+
89+
90+
<p align="center">
91+
<figure>
92+
<img src="../../../../assets/images/guides/jobs/resource_configuration_ray.png" alt="Resource configuration
93+
for Ray Job">
94+
<figcaption>Resource configuration for the Ray Job</figcaption>
95+
</figure>
96+
</p>
97+
98+
Runtime environment and Additional files required for the Ray job can also be provided.
99+
100+
* `Runtime Environment (Optional)`: A [runtime environment](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments) describes the dependencies required for the Ray job including files, packages, environment variables, and more. This is useful when you need to install specific packages and set environment variables for this particular Ray job. It should be provided as a YAML file. You can select the file from the project or upload a new one.
101+
102+
* `Additional files`: List of other files required for the Ray job. These files will be placed in `/srv/hops/ray/job`.
103+
104+
<p align="center">
105+
<figure>
106+
<img src="../../../../assets/images/guides/jobs/ray_runtime_env_and_additional_files.png" alt="Runtime
107+
environment and additional files">
108+
<figcaption>Runtime configuration and additional files for Ray job</figcaption>
109+
</figure>
110+
</p>
111+
112+
### Step 6: Execute the job
113+
114+
Now click the `Run` button to start the execution of the job, and then click on `Executions` to see the list of all executions.
115+
116+
<p align="center">
117+
<figure>
118+
<img src="../../../../assets/images/guides/jobs/start_ray_job.png" alt="Start job execution">
119+
<figcaption>Start job execution</figcaption>
120+
</figure>
121+
</p>
122+
123+
## Ray Dashboard
124+
When the Ray job is running, you can access the Ray dashboard to monitor the job. The Ray dashboard is accessible from the
125+
`Executions` page. Please note that the Ray dashboard is only available when the job execution is running. In the Ray Dashboard,
126+
you can monitor the resources used by the job, the number of workers, logs, and the tasks that are running.
127+
<p align="center">
128+
<figure>
129+
<img src="../../../../assets/images/guides/jobs/ray_dashboard.png" alt="Access Ray Dashboard">
130+
<figcaption>Access Ray Dashboard</figcaption>
131+
</figure>
132+
</p>
133+
134+
### Step 9: Application logs
135+
Once the execution is finished, you can click on `Logs` to see the full logs for execution.
136+
137+
<p align="center">
138+
<figure>
139+
<img src="../../../../assets/images/guides/jobs/ray_application_logs.png" alt="Access Ray logs">
140+
<figcaption>Access Ray job execution logs</figcaption>
141+
</figure>
142+
</p>
143+
144+
## Code
145+
146+
### Step 1: Upload the Ray script
147+
148+
This snippet assumes the Ray program is in the current working directory and named `ray_job.py`. If the file is already in the project, you can skip this step.
149+
150+
It will upload the jar to the `Resources` dataset in your project.
151+
152+
```python
153+
154+
import hopsworks
155+
156+
project = hopsworks.login()
157+
158+
dataset_api = project.get_dataset_api()
159+
160+
uploaded_file_path = dataset_api.upload("ray_job.py", "Resources")
161+
162+
```
163+
164+
165+
### Step 2: Create Ray job
166+
167+
In this snippet we get the `JobsApi` object to get the default job configuration for a `RAY` job, set the python script to run and create the `Job` object.
168+
169+
```python
170+
171+
jobs_api = project.get_jobs_api()
172+
173+
ray_config = jobs_api.get_configuration("RAY")
174+
175+
ray_config['appPath'] = uploaded_file_path
176+
ray_config['environmentName'] = "ray-training-pipeline"
177+
ray_config['driverCores'] = 2
178+
ray_config['driverMemory'] = 2048
179+
ray_config['workerCores'] = 2
180+
ray_config['workerMemory'] = 4096
181+
ray_config['minWorkers'] = 1
182+
ray_config['maxWorkers'] = 4
183+
184+
job = jobs_api.create_job("ray_job", ray_config)
185+
186+
```
187+
188+
### Step 3: Execute the job
189+
190+
In this snippet we execute the job synchronously, that is wait until it reaches a terminal state, and then download and print the logs.
191+
192+
```python
193+
194+
execution = job.run(await_termination=True)
195+
196+
out, err = execution.download_logs()
197+
198+
f_out = open(out, "r")
199+
print(f_out.read())
200+
201+
f_err = open(err, "r")
202+
print(f_err.read())
203+
204+
```
205+
206+
### API Reference
207+
208+
[Jobs](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/jobs/)
209+
210+
[Executions](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/executions/)

‎docs/user_guides/projects/jupyter/ray_notebook.md

+142
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
---
2+
description: Documentation on how to run Ray applications on Jupyter on Hopsworks.
3+
---
4+
# How To Run A Ray Notebook
5+
6+
### Introduction
7+
8+
Jupyter is provided as a service in Hopsworks, providing the same user experience and features as if run on your laptop.
9+
10+
* Supports JupyterLab and the classic Jupyter front-end
11+
* Configured with Python3, PySpark and Ray kernels
12+
13+
!!!warning "Enable Ray"
14+
15+
Support for Ray needs to be explicitly enabled by adding the following option in the `values.yaml` file for the deployment:
16+
17+
```yaml
18+
global:
19+
ray:
20+
enabled: true
21+
```
22+
23+
## Step 1: Jupyter dashboard
24+
25+
The image below shows the Jupyter service page in Hopsworks and is accessed by clicking `Jupyter` in the sidebar.
26+
27+
<p align="center">
28+
<figure>
29+
<img src="../../../../assets/images/guides/jupyter/jupyter_overview.png" alt="Jupyter dashboard in Hopsworks">
30+
<figcaption>Jupyter dashboard in Hopsworks</figcaption>
31+
</figure>
32+
</p>
33+
34+
From this page, you can configure various options and settings to start Jupyter with as described in the sections below.
35+
36+
## Step 2 (Optional): Configure Ray
37+
38+
Next step is to configure the Ray cluster configuration that will be created when you start Ray session later in
39+
Jupyter. Click `edit configuration` to get to the configuration page and select `Ray`.
40+
41+
### Resource and compute
42+
43+
Resource allocation for the Driver and Workers can be configured.
44+
45+
!!! notice "Using the resources in the Ray script"
46+
The resource configurations describe the cluster that will be provisioned when launching the Ray job. User can still
47+
provide extra configurations in the job script using `ScalingConfig`, i.e. `ScalingConfig(num_workers=4, trainer_resources={"CPU": 1}, use_gpu=True)`.
48+
49+
* `Driver memory`: Memory in MBs to allocate for Driver
50+
51+
* `Driver virtual cores`: Number of cores to allocate for the Driver
52+
53+
* `Worker memory`: Memory in MBs to allocate for each worker
54+
55+
* `Worker cores`: Number of cores to allocate for each worker
56+
57+
* `Min workers`: Minimum number of workers to start with
58+
59+
* `Max workers`: Maximum number of workers to scale up to
60+
61+
<p align="center">
62+
<figure>
63+
<img src="../../../../assets/images/guides/jupyter/ray_resource_and_compute.png" alt="Resource configuration for
64+
the Ray kernels">
65+
<figcaption>Resource configuration for the Ray kernels</figcaption>
66+
</figure>
67+
</p>
68+
69+
Runtime environment and Additional files required for the Ray job can also be provided.
70+
71+
* `Runtime Environment (Optional)`: A [runtime environment](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments) describes the dependencies required for the Ray job including files, packages, environment variables, and more. This is useful when you need to install specific packages and set environment variables for this particular Ray job. It should be provided as a YAML file. You can select the file from the project or upload a new one.
72+
73+
74+
* `Additional files`: List of other files required for the Ray job. These files will be placed in `/srv/hops/ray/job`.
75+
76+
<p align="center">
77+
<figure>
78+
<img src="../../../../assets/images/guides/jobs/ray_runtime_env_and_additional_files.png" alt="Runtime
79+
environment and additional files">
80+
<figcaption>Runtime configuration and additional files for Ray jupyter session</figcaption>
81+
</figure>
82+
</p>
83+
84+
85+
Click `Save` to save the new configuration.
86+
87+
## Step 3 (Optional): Configure max runtime and root path
88+
89+
Before starting the server there are two additional configurations that can be set next to the `Run Jupyter` button.
90+
91+
The runtime of the Jupyter instance can be configured, this is useful to ensure that idle instances will not be hanging around and keep allocating resources. If a limited runtime is not desirable, this can be disabled by setting `no limit`.
92+
93+
<p align="center">
94+
<figure>
95+
<img src="../../../../assets/images/guides/jupyter/configure_shutdown.png" alt="Configure maximum runtime">
96+
<figcaption>Configure maximum runtime</figcaption>
97+
</figure>
98+
</p>
99+
100+
The root path from which to start the Jupyter instance can be configured. By default it starts by setting the `/Jupyter` folder as the root.
101+
102+
<p align="center">
103+
<figure>
104+
<img src="../../../../assets/images/guides/jupyter/start_from_folder.png" alt="Configure root folder">
105+
<figcaption>Configure root folder</figcaption>
106+
</figure>
107+
</p>
108+
109+
## Step 4: Select the environment
110+
Hopsworks provides a variety of environments to run Jupyter notebooks. Select the environment you want to use by clicking on the dropdown menu.
111+
In order to be able to run a Ray notebook, you need to select the environment that has the Ray kernel installed.
112+
Environment with Ray kernel have a `Ray Enabled` label next to them.
113+
114+
## Step 5: Start Jupyter
115+
116+
Start the Jupyter instance by clicking the `Run Jupyter` button.
117+
118+
## Running Ray code in Jupyter
119+
Once the Jupyter instance is started, you can create a new notebook by clicking on the `New` button and selecting
120+
`Ray` kernel. You can now write and run Ray code in the notebook. When you first run a cell with Ray code, a Ray session will be started and you can monitor the resources used by the job in the Ray dashboard.
121+
122+
<p align="center">
123+
<figure>
124+
<img src="../../../../assets/images/guides/jupyter/ray_kernel.png" alt="Ray Kernel">
125+
<figcaption>Ray Kernel</figcaption>
126+
</figure>
127+
</p>
128+
129+
## Step 6: Access Ray Dashboard
130+
131+
When you start a Ray session in Jupyter, a new application will appear in the Jupyter page.
132+
The notebook name from which the session was started is displayed. You can access the Ray UI by clicking on the `Ray Dashboard` and a new
133+
tab will be opened. The Ray dashboard is only available while the Ray kernel is running.
134+
You can kill the Ray session to free up resources by shutting down the kernel in Jupyter.
135+
In the Ray Dashboard, you can monitor the resources used by code you are running, the number of workers, logs, and the tasks that are running.
136+
137+
<p align="center">
138+
<figure>
139+
<img src="../../../../assets/images/guides/jupyter/ray_jupyter_notebook_session.png" alt="Access Ray Dashboard">
140+
<figcaption>Access Ray Dashboard for Jupyter Ray session</figcaption>
141+
</figure>
142+
</p>

‎docs/user_guides/projects/jupyter/spark_notebook.md

+3-2
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
Jupyter is provided as a service in Hopsworks, providing the same user experience and features as if run on your laptop.
66

77
* Supports JupyterLab and the classic Jupyter front-end
8-
* Configured with Python and PySpark kernels
8+
* Configured with Python3, PySpark and Ray kernels
99

1010

1111
## Step 1: Jupyter dashboard
@@ -34,7 +34,8 @@ You can easily refer to the green ticks as to what kernels are available in whic
3434

3535
## Step 3 (Optional): Configure spark properties
3636

37-
Next step is to configure the Spark properties to be used in Jupyter, Click `edit configuration` to get to the configuration page and select `Spark`.
37+
Next step is to configure the Ray properties to be used in Jupyter, Click `edit configuration` to get to the
38+
configuration page and select `Ray`.
3839

3940
### Resource and compute
4041

‎mkdocs.yml

+2
Original file line numberDiff line numberDiff line change
@@ -147,11 +147,13 @@ nav:
147147
- Jupyter:
148148
- Run Python Notebook: user_guides/projects/jupyter/python_notebook.md
149149
- Run PySpark Notebook: user_guides/projects/jupyter/spark_notebook.md
150+
- Run Ray Notebook: user_guides/projects/jupyter/ray_notebook.md
150151
- Remote Filesystem Driver: user_guides/projects/jupyter/remote_filesystem_driver.md
151152
- Jobs:
152153
- Run PySpark Job: user_guides/projects/jobs/pyspark_job.md
153154
- Run Spark Job: user_guides/projects/jobs/spark_job.md
154155
- Run Python Job: user_guides/projects/jobs/python_job.md
156+
- Run Ray Job: user_guides/projects/jobs/ray_job.md
155157
- Run Jupyter Notebook Job: user_guides/projects/jobs/notebook_job.md
156158
- Scheduling: user_guides/projects/jobs/schedule_job.md
157159
- Kubernetes Scheduling: user_guides/projects/scheduling/kube_scheduler.md

0 commit comments

Comments
 (0)