Skip to content

Commit 732c86c

Browse files
DilumAluthgeDilumAluthge
authored
Docs: Move SGE docs from README to separate file (#248)
The SGE functionality in this package is not currently being maintained, so I don't think we should present the SGE docs up-front in the README. Instead, let's put the SGE docs in a separate Markdown file. --- Depends on: - [x] #247
1 parent d3b5c22 commit 732c86c

File tree

2 files changed

+74
-64
lines changed

2 files changed

+74
-64
lines changed

‎README.md

+4-64
Original file line numberDiff line numberDiff line change
@@ -39,70 +39,6 @@ You can also write your own custom cluster manager; see the instructions in the
3939

4040
For Slurm, please see the [SlurmClusterManager.jl](https://github.com/JuliaParallel/SlurmClusterManager.jl) package.
4141

42-
### SGE - a simple interactive example
43-
44-
```julia
45-
julia> using ClusterManagers
46-
47-
julia> ClusterManagers.addprocs_sge(5; qsub_flags=`-q queue_name`)
48-
job id is 961, waiting for job to start .
49-
5-element Array{Any,1}:
50-
2
51-
3
52-
4
53-
5
54-
6
55-
56-
julia> @parallel for i=1:5
57-
run(`hostname`)
58-
end
59-
60-
julia> From worker 2: compute-6
61-
From worker 4: compute-6
62-
From worker 5: compute-6
63-
From worker 6: compute-6
64-
From worker 3: compute-6
65-
```
66-
67-
Some clusters require the user to specify a list of required resources.
68-
For example, it may be necessary to specify how much memory will be needed by the job - see this [issue](https://github.com/JuliaLang/julia/issues/10390).
69-
The keyword `qsub_flags` can be used to specify these and other options.
70-
Additionally the keyword `wd` can be used to specify the working directory (which defaults to `ENV["HOME"]`).
71-
72-
```julia
73-
julia> using Distributed, ClusterManagers
74-
75-
julia> addprocs_sge(5; qsub_flags=`-q queue_name -l h_vmem=4G,tmem=4G`, wd=mktempdir())
76-
Job 5672349 in queue.
77-
Running.
78-
5-element Array{Int64,1}:
79-
2
80-
3
81-
4
82-
5
83-
6
84-
85-
julia> pmap(x->run(`hostname`),workers());
86-
87-
julia> From worker 26: lum-7-2.local
88-
From worker 23: pace-6-10.local
89-
From worker 22: chong-207-10.local
90-
From worker 24: pace-6-11.local
91-
From worker 25: cheech-207-16.local
92-
93-
julia> rmprocs(workers())
94-
Task (done)
95-
```
96-
97-
### SGE via qrsh
98-
99-
`SGEManager` uses SGE's `qsub` command to launch workers, which communicate the
100-
TCP/IP host:port info back to the master via the filesystem. On filesystems
101-
that are tuned to make heavy use of caching to increase throughput, launching
102-
Julia workers can frequently timeout waiting for the standard output files to appear.
103-
In this case, it's better to use the `QRSHManager`, which uses SGE's `qrsh`
104-
command to bypass the filesystem and captures STDOUT directly.
105-
10642
### Using `LocalAffinityManager` (for pinning local workers to specific cores)
10743

10844
- Linux only feature.
@@ -149,3 +85,7 @@ ElasticManager:
14985
By default, the printed command uses the absolute path to the current Julia executable and activates the same project as the current session. You can change either of these defaults by passing `printing_kwargs=(absolute_exename=false, same_project=false))` to the first form of the `ElasticManager` constructor.
15086

15187
Once workers are connected, you can print the `em` object again to see them added to the list of active workers.
88+
89+
### Sun Grid Engine (SGE)
90+
91+
See [`docs/sge.md`](docs/sge.md)

‎docs/sge.md

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Sun Grid Engine (SGE)
2+
3+
> [!WARNING]
4+
> The SGE functionality is not currently being maintained.
5+
>
6+
> We are currently seeking a new maintainer for the SGE functionality. If you are an active user of SGE and are interested in being a maintainer, please open a GitHub issue - say that you are interested in being a maintainer for the SGE functionality.
7+
8+
## SGE via `qsub`: Use `ClusterManagers.addprocs_sge` (or `ClusterManagers.SGEManager`)
9+
10+
```julia
11+
julia> using ClusterManagers
12+
13+
julia> ClusterManagers.addprocs_sge(5; qsub_flags=`-q queue_name`)
14+
job id is 961, waiting for job to start .
15+
5-element Array{Any,1}:
16+
2
17+
3
18+
4
19+
5
20+
6
21+
22+
julia> @parallel for i=1:5
23+
run(`hostname`)
24+
end
25+
26+
julia> From worker 2: compute-6
27+
From worker 4: compute-6
28+
From worker 5: compute-6
29+
From worker 6: compute-6
30+
From worker 3: compute-6
31+
```
32+
33+
Some clusters require the user to specify a list of required resources.
34+
For example, it may be necessary to specify how much memory will be needed by the job - see this [issue](https://github.com/JuliaLang/julia/issues/10390).
35+
The keyword `qsub_flags` can be used to specify these and other options.
36+
Additionally the keyword `wd` can be used to specify the working directory (which defaults to `ENV["HOME"]`).
37+
38+
```julia
39+
julia> using Distributed, ClusterManagers
40+
41+
julia> addprocs_sge(5; qsub_flags=`-q queue_name -l h_vmem=4G,tmem=4G`, wd=mktempdir())
42+
Job 5672349 in queue.
43+
Running.
44+
5-element Array{Int64,1}:
45+
2
46+
3
47+
4
48+
5
49+
6
50+
51+
julia> pmap(x->run(`hostname`),workers());
52+
53+
julia> From worker 26: lum-7-2.local
54+
From worker 23: pace-6-10.local
55+
From worker 22: chong-207-10.local
56+
From worker 24: pace-6-11.local
57+
From worker 25: cheech-207-16.local
58+
59+
julia> rmprocs(workers())
60+
Task (done)
61+
```
62+
63+
## SGE via `qrsh`: Use `ClusterManagers.addprocs_qrsh` (or `ClusterManagers.QRSHManager`)
64+
65+
`SGEManager` uses SGE's `qsub` command to launch workers, which communicate the
66+
TCP/IP host:port info back to the master via the filesystem. On filesystems
67+
that are tuned to make heavy use of caching to increase throughput, launching
68+
Julia workers can frequently timeout waiting for the standard output files to appear.
69+
In this case, it's better to use the `QRSHManager`, which uses SGE's `qrsh`
70+
command to bypass the filesystem and captures STDOUT directly.

0 commit comments

Comments
 (0)