Skip to content

[HWORKS-2175][APPEND] Kueue - queues, cohorts and topologies #496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions docs/user_guides/mlops/serving/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,13 @@ You will be redirected to a full-page deployment creation form where you can see

Once you are done with the changes, click on `Create new deployment` at the bottom of the page to create the deployment for your model.

### Step 4: Deployment creation
### Step 4: (Kueue enabled) Select a Queue

If the cluster is installed with Kueue enabled, you will need to select a queue in which the deployment should run. This can be done from `Advance configuration -> Scheduler section`.

![Default queue for job](../../../assets/images/guides/project/scheduler/job_queue.png)

### Step 5: Deployment creation

Wait for the deployment creation process to finish.

Expand All @@ -98,7 +104,7 @@ Wait for the deployment creation process to finish.
</figure>
</p>

### Step 5: Deployment overview
### Step 6: Deployment overview

Once the deployment is created, you will be redirected to the list of all your existing deployments in the project. You can use the filters on the top of the page to easily locate your new deployment.

Expand Down
10 changes: 8 additions & 2 deletions docs/user_guides/projects/jobs/notebook_job.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,13 @@ You can always modify the arguments in the job settings.
</figure>
</p>

### Step 7: Execute the job
### Step 7: (Kueue enabled) Select a Queue

If the cluster is installed with Kueue enabled, you will need to select a queue in which the job should run. This can be done from `Advance configuration -> Scheduler section`.

![Default queue for job](../../../assets/images/guides/project/scheduler/job_queue.png)

### Step 8: Execute the job

Now click the `Run` button to start the execution of the job. You will be redirected to the `Executions` page where you can see the list of all executions.

Expand All @@ -103,7 +109,7 @@ Now click the `Run` button to start the execution of the job. You will be redire
</figure>
</p>

### Step 8: Visualize output notebook
### Step 9: Visualize output notebook
Once the execution is finished, click `Logs` and then `notebook out` to see the logs for the execution.

<p align="center">
Expand Down
8 changes: 6 additions & 2 deletions docs/user_guides/projects/jobs/pyspark_job.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,11 @@ Line-separates [properties](https://spark.apache.org/docs/3.1.1/configuration.ht
</figure>
</p>

### Step 7: Execute the job
### Step 7: (Kueue enabled) Select a Queue

Currently we do not have Kueue support for Spark. You do not need to select a queue to run the job in.

### Step 8: Execute the job

Now click the `Run` button to start the execution of the job. You will be redirected to the `Executions` page where you can see the list of all executions.

Expand All @@ -139,7 +143,7 @@ Now click the `Run` button to start the execution of the job. You will be redire
</figure>
</p>

### Step 8: Application logs
### Step 9: Application logs

To monitor logs while the execution is running, click `Spark UI` to open the Spark UI in a separate tab.

Expand Down
8 changes: 7 additions & 1 deletion docs/user_guides/projects/jobs/python_job.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,13 @@ It is possible to also set following configuration settings for a `PYTHON` job.
</figure>
</p>

### Step 7: Execute the job
### Step 7: (Kueue enabled) Select a Queue

If the cluster is installed with Kueue enabled, you will need to select a queue in which the job should run. This can be done from `Advance configuration -> Scheduler section`.

![Default queue for job](../../../assets/images/guides/project/scheduler/job_queue.png)

### Step 8: Execute the job

Now click the `Run` button to start the execution of the job. You will be redirected to the `Executions` page where you can see the list of all executions.

Expand Down
10 changes: 8 additions & 2 deletions docs/user_guides/projects/jobs/ray_job.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,13 @@ environment and additional files">
</figure>
</p>

### Step 6: Execute the job
### Step 6: (Kueue enabled) Select a Queue

If the cluster is installed with Kueue enabled, you will need to select a queue in which the job should run. This can be done from `Advance configuration -> Scheduler section`.

![Default queue for job](../../../assets/images/guides/project/scheduler/job_queue.png)

### Step 7: Execute the job

Now click the `Run` button to start the execution of the job, and then click on `Executions` to see the list of all executions.

Expand All @@ -120,7 +126,7 @@ Now click the `Run` button to start the execution of the job, and then click on
</figure>
</p>

## Ray Dashboard
### Step 8: Ray Dashboard
When the Ray job is running, you can access the Ray dashboard to monitor the job. The Ray dashboard is accessible from the
`Executions` page. Please note that the Ray dashboard is only available when the job execution is running. In the Ray Dashboard,
you can monitor the resources used by the job, the number of workers, logs, and the tasks that are running.
Expand Down
8 changes: 6 additions & 2 deletions docs/user_guides/projects/jobs/spark_job.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,11 @@ Line-separates [properties](https://spark.apache.org/docs/3.1.1/configuration.ht
</figure>
</p>

### Step 8: Execute the job
### Step 8: (Kueue enabled) Select a Queue

Currently we do not have Kueue support for Spark. You do not need to select a queue to run the job in.

### Step 9: Execute the job

Now click the `Run` button to start the execution of the job, and then click on `Executions` to see the list of all executions.

Expand All @@ -143,7 +147,7 @@ Now click the `Run` button to start the execution of the job, and then click on
</figure>
</p>

### Step 9: Application logs
### Step 10: Application logs

To monitor logs while the execution is running, click `Spark UI` to open the Spark UI in a separate tab.

Expand Down
8 changes: 7 additions & 1 deletion docs/user_guides/projects/jupyter/python_notebook.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,14 @@ The root path from which to start the Jupyter instance can be configured. By def
</figure>
</p>

## Step 4: (Kueue enabled) Select a Queue

## Step 4: Start Jupyter
If the cluster is installed with Kueue enabled, you will need to select a queue in which the notebook should run. This can be done from `Advance configuration -> Scheduler section`.

![Default queue for job](../../../assets/images/guides/project/scheduler/job_queue.png)


## Step 5: Start Jupyter

Start the Jupyter instance by clicking the `Run Jupyter` button.

Expand Down
10 changes: 8 additions & 2 deletions docs/user_guides/projects/jupyter/ray_notebook.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,13 @@ Hopsworks provides a variety of environments to run Jupyter notebooks. Select th
In order to be able to run a Ray notebook, you need to select the environment that has the Ray kernel installed.
Environment with Ray kernel have a `Ray Enabled` label next to them.

## Step 5: Start Jupyter
## Step 5: (Kueue enabled) Select a Queue

If the cluster is installed with Kueue enabled, you will need to select a queue in which the notebook should run. This can be done from `Advance configuration -> Scheduler section`.

![Default queue for job](../../../assets/images/guides/project/scheduler/job_queue.png)

## Step 6: Start Jupyter

Start the Jupyter instance by clicking the `Run Jupyter` button.

Expand All @@ -126,7 +132,7 @@ Once the Jupyter instance is started, you can create a new notebook by clicking
</figure>
</p>

## Step 6: Access Ray Dashboard
## Step 7: Access Ray Dashboard

When you start a Ray session in Jupyter, a new application will appear in the Jupyter page.
The notebook name from which the session was started is displayed. You can access the Ray UI by clicking on the `Ray Dashboard` and a new
Expand Down
4 changes: 4 additions & 0 deletions docs/user_guides/projects/jupyter/spark_notebook.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,10 @@ The root path from which to start the Jupyter instance can be configured. By def
</p>


## Step 5: (Kueue enabled) Select a Queue

Currently we do not have Kueue support for Spark. You do not need to select a queue to run the notebook in.

## Step 5: Start Jupyter

Start the Jupyter instance by clicking the `Run Jupyter` button.
Expand Down