Skip to content

Issues starting GPU workers #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rcthomas opened this issue Sep 4, 2021 · 3 comments
Closed

Issues starting GPU workers #78

rcthomas opened this issue Sep 4, 2021 · 3 comments

Comments

@rcthomas
Copy link

rcthomas commented Sep 4, 2021 •
edited
Loading

I've been able to manually launch a Dask GPU cluster using Slurm successfully. The setup for that looks like:

<get allocation>
dask-scheduler --scheduler-file scheduler.json &
srun -G <number-of-gpus-in-allocation> dask-cuda-worker --scheduler-file scheduler.json

This works OK for say, 2 nodes with 4 GPUs each but I find that when I request more workers, say, 80 GPUs, some of them just never come up, or they take an inordinately long amount of time to come up. I had this problem with manual start in a CPU-only context with Dask Distributed, and the fix for that kind of on-demand scale was dask-mpi. Combined with containers, this is a very good solution we've found for reliable Dask startup on HPC. I'm hoping for the same kind of solution with GPUs.

I've tried out dask-mpi support for dask_cuda.CUDAWorker but I can't seem to find a working invocation. I tried just adapting the above, or using something similar to what I've done before:

srun -G 40 -u python -u $(which dask-mpi) --scheduler-file scheduler.json \
    --nthreads 1 --worker-class dask_cuda.CUDAWorker --dashboard-address 0

The first thing I noticed is that I have to specify --nthreads 1 because otherwise you get an error which might be easy to fix:

TypeError: '<' not supported between instances of 'NoneType' and 'int'
    return self.main(*args, **kwargs)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 147, in main
    asyncio.get_event_loop().run_until_complete(run_worker())
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 144, in run_worker
    async with WorkerType(**opts) as worker:
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_cuda/cuda_worker.py", line 95, in __init__
    if nthreads < 1:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

Once you have --nthreads 1 in place though, you hit this:

Traceback (most recent call last):
  File "/pscratch/sd/r/rthomas/dask/env/bin/dask-mpi", line 8, in <module>
    sys.exit(go())
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 152, in go
    main()
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 147, in main
    asyncio.get_event_loop().run_until_complete(run_worker())
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_mpi/cli.py", line 144, in run_worker
    async with WorkerType(**opts) as worker:
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_cuda/cuda_worker.py", line 216, in __init__
    self.nannies = [
  File "/pscratch/sd/r/rthomas/dask/env/lib/python3.8/site-packages/dask_cuda/cuda_worker.py", line 217, in <listcomp>
    Nanny(
TypeError: __init__() got multiple values for argument 'scheduler_ip'

as if a keyword arg is trying to overwrite a positional one. I started wondering how people were trying this out themselves and thought maybe they weren't going with a scheduler file for the use cases tested up till now, but I had trouble properly specifying options and arguments for dask_cuda.CUDAWorker via --worker-options (they seem to not matter) or specifying SCHEDULER_ADDRESS on the command line.

My conda env, based off RAPIDS stable 
# packages in environment at /pscratch/sd/r/rthomas/dask/env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             4.5                       1_gnu    defaults
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
aiohttp                   3.7.4.post0      py38h497a2fe_0    conda-forge
anyio                     3.3.0            py38h578d9bd_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py38h497a2fe_2    conda-forge
arrow-cpp                 4.0.1           py38h9a0cccc_7_cuda    conda-forge
arrow-cpp-proc            3.0.0                      cuda    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
aws-c-cal                 0.5.11               h95a6274_0    conda-forge
aws-c-common              0.6.2                h7f98852_0    conda-forge
aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
aws-c-io                  0.10.5               hfb6a706_0    conda-forge
aws-checksums             0.1.11               ha31a3da_7    conda-forge
aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bleach                    4.1.0              pyhd8ed1ab_0    conda-forge
blosc                     1.21.0               h9c3ff4c_0    conda-forge
bokeh                     2.3.3            py38h578d9bd_0    conda-forge
boost                     1.74.0           py38hc10631b_3    conda-forge
boost-cpp                 1.74.0               h312852a_4    conda-forge
brotli                    1.0.9                h7f98852_5    conda-forge
brotli-bin                1.0.9                h7f98852_5    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2021.5.30            ha878542_0    conda-forge
cachetools                4.2.2              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
certifi                   2021.5.30        py38h578d9bd_0    conda-forge
cffi                      1.14.6           py38ha65f79e_0    conda-forge
cfitsio                   3.470                hb418390_7    conda-forge
chardet                   4.0.0            py38h578d9bd_1    conda-forge
charls                    2.2.0                h9c3ff4c_0    conda-forge
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     7.1.2              pyh9f0ad1d_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_0    conda-forge
cloudpickle               1.6.0                      py_0    conda-forge
colorcet                  2.0.6              pyhd8ed1ab_0    conda-forge
conda                     4.10.3           py38h578d9bd_0    conda-forge
conda-package-handling    1.7.3            py38h497a2fe_0    conda-forge
cryptography              3.4.7            py38ha5dfef3_0    conda-forge
cucim                     21.08.01        cuda_11.0_py38_ga89f250_0    rapidsai
cudatoolkit               11.0.221             h6bb024c_0    nvidia
cudf                      21.08.02        cuda_11.0_py38_gf6d31fa95d_0    rapidsai
cudf_kafka                21.08.02        py38_gf6d31fa95d_0    rapidsai
cugraph                   21.08.03        cuda11.0_py38_g9e9f1570_0    rapidsai
cuml                      21.08.01        cuda11.0_py38_g5c0e99300_0    rapidsai
cupy                      9.0.0            py38hc350bd8_0    conda-forge
curl                      7.78.0               hea6ffbf_0    conda-forge
cusignal                  21.08.00        py37_g33f663e_0    rapidsai
cuspatial                 21.08.01        py38_g7c0151b_0    rapidsai
custreamz                 21.08.02        py38_gf6d31fa95d_0    rapidsai
cuxfilter                 21.08.00        py38_g274c584_0    rapidsai
cycler                    0.10.0                     py_2    conda-forge
cyrus-sasl                2.1.27               h230043b_2    conda-forge
cytoolz                   0.11.0           py38h497a2fe_3    conda-forge
dask                      2021.7.1           pyhd8ed1ab_0    conda-forge
dask-core                 2021.7.1           pyhd8ed1ab_0    conda-forge
dask-cuda                 21.08.00                 py38_0    rapidsai
dask-cudf                 21.08.02        py38_gf6d31fa95d_0    rapidsai
dask-mpi                  2.21.0+49.gccacb62          pypi_0    pypi
datashader                0.11.1             pyh9f0ad1d_0    conda-forge
datashape                 0.5.4                      py_1    conda-forge
debugpy                   1.4.1            py38h709712a_0    conda-forge
decorator                 4.4.2                      py_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distributed               2021.7.1         py38h578d9bd_0    conda-forge
dlpack                    0.5                  h9c3ff4c_0    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
expat                     2.4.1                h9c3ff4c_0    conda-forge
faiss-proc                1.0.0                      cuda    rapidsai
fastavro                  1.4.4            py38h497a2fe_0    conda-forge
fastrlock                 0.6              py38h709712a_1    conda-forge
fiona                     1.8.20           py38hdb5a769_0    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
freexl                    1.0.6                h7f98852_0    conda-forge
fsspec                    2021.8.1           pyhd8ed1ab_0    conda-forge
gdal                      3.2.2            py38h507a4fd_7    conda-forge
geopandas                 0.9.0              pyhd8ed1ab_1    conda-forge
geopandas-base            0.9.0              pyhd8ed1ab_1    conda-forge
geos                      3.9.1                h9c3ff4c_2    conda-forge
geotiff                   1.6.0                h4f31c25_6    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glog                      0.5.0                h48cff8f_0    conda-forge
grpc-cpp                  1.39.0               hf1f433d_2    conda-forge
hdf4                      4.2.15               h10796ff_3    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
idna                      3.2                pyhd3eb1b0_0    defaults
imagecodecs               2021.7.30        py38hb5ce8f7_0    conda-forge
imageio                   2.9.0                      py_0    conda-forge
importlib-metadata        4.8.1            py38h578d9bd_0    conda-forge
ipykernel                 6.3.1            py38he5a9106_0    conda-forge
ipython                   7.27.0           py38he5a9106_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.6.4              pyhd8ed1ab_0    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jedi                      0.18.0           py38h578d9bd_2    conda-forge
jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge
joblib                    1.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
json-c                    0.15                 h98cffda_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter-server-proxy      3.1.0              pyhd8ed1ab_0    conda-forge
jupyter_client            7.0.2              pyhd8ed1ab_0    conda-forge
jupyter_core              4.7.1            py38h578d9bd_0    conda-forge
jupyter_server            1.10.2             pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_widgets        1.0.1              pyhd8ed1ab_0    conda-forge
jxrlib                    1.1                  h7f98852_2    conda-forge
kealib                    1.4.14               hcc255d8_2    conda-forge
kiwisolver                1.3.1            py38h1fd1430_1    conda-forge
krb5                      1.19.2               hcc1bbae_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               h7274673_9    defaults
lerc                      2.2.1                h9c3ff4c_0    conda-forge
libaec                    1.0.5                h9c3ff4c_0    conda-forge
libblas                   3.9.0           11_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h7f98852_5    conda-forge
libbrotlidec              1.0.9                h7f98852_5    conda-forge
libbrotlienc              1.0.9                h7f98852_5    conda-forge
libcblas                  3.9.0           11_linux64_openblas    conda-forge
libcucim                  21.08.01        cuda11.0_ga89f250_0    rapidsai
libcudf                   21.08.02        cuda11.0_gf6d31fa95d_0    rapidsai
libcudf_kafka             21.08.02          gf6d31fa95d_0    rapidsai
libcugraph                21.08.03        cuda11.0_g9e9f1570_0    rapidsai
libcuml                   21.08.01        cuda11.0_g5c0e99300_0    rapidsai
libcumlprims              21.08.00        cuda11.0_g9c188ef_0    nvidia
libcurl                   7.78.0               h2574ce0_0    conda-forge
libcuspatial              21.08.01        cuda11.0_g7c0151b_0    rapidsai
libdap4                   3.20.6               hd7c4107_2    conda-forge
libdeflate                1.8                  h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libfaiss                  1.7.0           cuda110h8045045_8_cuda    conda-forge
libffi                    3.3                  he6710b0_2    defaults
libgcc-ng                 9.3.0               h5101ec6_17    defaults
libgcrypt                 1.9.3                h7f98852_1    conda-forge
libgdal                   3.2.2                h8f005ca_7    conda-forge
libgfortran-ng            11.1.0               h69a702a_8    conda-forge
libgfortran5              11.1.0               h6c583b3_8    conda-forge
libglib                   2.68.3               h3e27bee_0    conda-forge
libgomp                   9.3.0               h5101ec6_17    defaults
libgpg-error              1.42                 h9c3ff4c_0    conda-forge
libgsasl                  1.8.0                         2    conda-forge
libhwloc                  2.3.0                h5e5b7d1_1    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libkml                    1.3.0             h238a007_1014    conda-forge
liblapack                 3.9.0           11_linux64_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libnetcdf                 4.8.0           nompi_hcd642e3_103    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libntlm                   1.4               h7f98852_1002    conda-forge
libopenblas               0.3.17          pthreads_h8fe5266_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     13.3                 hd57d9b9_0    conda-forge
libprotobuf               3.16.0               h780b84a_0    conda-forge
librdkafka                1.6.1                hc49e61c_1    conda-forge
librmm                    21.08.01        cuda11.0_g66cf439_0    rapidsai
librttopo                 1.1.0                h1185371_6    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
libspatialite             5.0.1                h8694cbe_5    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-ng              9.3.0               hd4cf53a_17    defaults
libthrift                 0.14.2               he6d91bd_1    conda-forge
libtiff                   4.3.0                hf544144_0    conda-forge
libutf8proc               2.6.1                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libuv                     1.42.0               h7f98852_0    conda-forge
libwebp                   1.2.0                h3452ae3_0    conda-forge
libwebp-base              1.2.0                h7f98852_2    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxgboost                1.4.2dev.rapidsai21.08      cuda11.0_0    rapidsai
libxml2                   2.9.12               h72842e0_0    conda-forge
libzip                    1.8.0                h4de3113_0    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
llvmlite                  0.36.0           py38h4630a5e_0    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
mapclassify               2.4.3              pyhd8ed1ab_0    conda-forge
markdown                  3.3.4              pyhd8ed1ab_0    conda-forge
markupsafe                2.0.1            py38h497a2fe_0    conda-forge
matplotlib-base           3.4.2            py38hcc49a3a_0    conda-forge
matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
mistune                   0.8.4           py38h497a2fe_1004    conda-forge
mpi4py                    3.1.1                    pypi_0    pypi
msgpack-python            1.0.2            py38h1fd1430_1    conda-forge
multidict                 5.1.0            py38h497a2fe_1    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
munch                     2.5.0                      py_0    conda-forge
nbclient                  0.5.4              pyhd8ed1ab_0    conda-forge
nbconvert                 6.1.0            py38h578d9bd_0    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
nccl                      2.10.3.1             h96e36e3_0    conda-forge
ncurses                   6.2                  he6710b0_1    defaults
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
networkx                  2.6.2              pyhd8ed1ab_0    conda-forge
nodejs                    14.17.4              h92b4a50_0    conda-forge
notebook                  6.4.3              pyha770c72_0    conda-forge
numba                     0.53.1           py38h8b71fd7_1    conda-forge
numpy                     1.21.1           py38h9894fe3_0    conda-forge
nvtx                      0.2.3            py38h497a2fe_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
orc                       1.6.9                h58a87f1_0    conda-forge
packaging                 21.0               pyhd8ed1ab_0    conda-forge
pandas                    1.2.5            py38h1abd341_0    conda-forge
pandoc                    2.14.2               h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
panel                     0.12.1             pyhd8ed1ab_0    conda-forge
param                     1.11.1             pyh6c4a22f_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.3.1            py38h8e6f84c_0    conda-forge
pip                       21.2.4             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pooch                     1.5.1              pyhd8ed1ab_0    conda-forge
poppler                   21.03.0              h93df280_0    conda-forge
poppler-data              0.4.10                        0    conda-forge
postgresql                13.3                 h2510834_0    conda-forge
proj                      8.0.1                h277dcde_0    conda-forge
prometheus_client         0.11.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.20             pyha770c72_0    conda-forge
protobuf                  3.16.0           py38h709712a_0    conda-forge
psutil                    5.8.0            py38h497a2fe_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py-xgboost                1.4.2dev.rapidsai21.08  cuda11.0py38_0    rapidsai
pyarrow                   4.0.1           py38hdd2221d_7_cuda    conda-forge
pycosat                   0.6.3           py38h497a2fe_1006    conda-forge
pycparser                 2.20                       py_2    defaults
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pydeck                    0.5.0              pyh9f0ad1d_0    conda-forge
pygments                  2.10.0             pyhd8ed1ab_0    conda-forge
pynvml                    11.0.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 20.0.1             pyhd3eb1b0_1    defaults
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    3.1.0            py38h03a1999_3    conda-forge
pyrsistent                0.17.3           py38h497a2fe_2    conda-forge
pysocks                   1.7.1            py38h578d9bd_3    conda-forge
python                    3.8.10          h49503c6_1_cpython    conda-forge
python-confluent-kafka    1.6.0            py38h497a2fe_1    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyviz_comms               2.1.0              pyhd8ed1ab_0    conda-forge
pywavelets                1.1.1            py38h5c078b8_3    conda-forge
pyyaml                    5.4.1            py38h497a2fe_0    conda-forge
pyzmq                     22.1.0           py38h2035c66_0    conda-forge
rapids                    21.08.00        cuda11.0_py38_ga776927_74    rapidsai
rapids-xgboost            21.08.00        cuda11.0_py38_ga776927_74    rapidsai
re2                       2021.08.01           h9c3ff4c_0    conda-forge
readline                  8.1                  h27cfd23_0    defaults
requests                  2.26.0             pyhd3eb1b0_0    defaults
requests-unixsocket       0.2.0                      py_0    conda-forge
rmm                       21.08.01        cuda_11.0_py38_g66cf439_0    rapidsai
rtree                     0.9.7            py38h02d302b_2    conda-forge
ruamel_yaml               0.15.80         py38h497a2fe_1004    conda-forge
s2n                       1.0.10               h9b69904_0    conda-forge
scikit-image              0.18.1           py38h51da96c_0    conda-forge
scikit-learn              0.24.2           py38hdc147b9_0    conda-forge
scipy                     1.7.0            py38h7b17777_1    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                49.6.0           py38h578d9bd_3    conda-forge
shapely                   1.7.1            py38haeee4fe_5    conda-forge
simpervisor               0.4                pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyhd3eb1b0_0    defaults
snappy                    1.1.8                he1b5a44_3    conda-forge
sniffio                   1.2.0            py38h578d9bd_1    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
spdlog                    1.8.5                h4bd325d_0    conda-forge
sqlite                    3.36.0               hc218d9a_0    defaults
streamz                   0.6.2              pyh44b312d_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.11.1           py38h578d9bd_0    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             2.2.0              pyh8a188c0_0    conda-forge
tifffile                  2021.8.30          pyhd8ed1ab_0    conda-forge
tiledb                    2.3.2                he87e0bf_0    conda-forge
tk                        8.6.10               hbc83047_0    defaults
toolz                     0.11.1                     py_0    conda-forge
tornado                   6.1              py38h497a2fe_1    conda-forge
tqdm                      4.62.1             pyhd3eb1b0_1    defaults
traitlets                 5.1.0              pyhd8ed1ab_0    conda-forge
treelite                  2.0.0            py38hc9ad5e7_0    conda-forge
treelite-runtime          2.0.0                    pypi_0    pypi
typing-extensions         3.10.0.0             hd8ed1ab_0    conda-forge
typing_extensions         3.10.0.0           pyha770c72_0    conda-forge
tzcode                    2021a                h7f98852_2    conda-forge
tzdata                    2021a                h5d7bf9c_0    defaults
ucx                       1.9.0+gcd9efd3       cuda11.0_0    rapidsai
ucx-proc                  1.0.0                       gpu    rapidsai
ucx-py                    0.21.0          py38_gcd9efd3_0    rapidsai
urllib3                   1.26.6             pyhd3eb1b0_1    defaults
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          0.57.0           py38h578d9bd_4    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
widgetsnbextension        3.5.1            py38h578d9bd_4    conda-forge
xarray                    0.19.0             pyhd8ed1ab_1    conda-forge
xerces-c                  3.2.3                h9d8b166_2    conda-forge
xgboost                   1.4.2dev.rapidsai21.08  cuda11.0py38_0    rapidsai
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h7b6447c_0    defaults
yaml                      0.2.5                h7b6447c_0    defaults
yarl                      1.6.3            py38h497a2fe_2    conda-forge
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zfp                       0.5.5                h9c3ff4c_5    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.5.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11               h7b6447c_3    defaults
zstd                      1.5.0                ha95c52a_0    conda-forge

For completeness, I did this to set up my Dask cluster env and Jupyter kernel env:

  • Install miniconda
  • Install RAPIDS
  • Build mpi4py to link against pre-existing MPI (don't think this is part of the problem)
  • pip install --no-cache-dir --force git+https://github.com/dask/dask-mpi (to keep from overwriting my mpi4py)
@jacobtomlinson
Copy link
Member

Sorry you're having trouble here @rcthomas.

Could you try the fix in #79 and see if that resolves it for you?

@rcthomas
Copy link
Author

rcthomas commented Sep 7, 2021

Thanks @jacobtomlinson that does fix the options collision. Now I'm onto other problems, thanks!

@jacobtomlinson
Copy link
Member

Great. Please feel free to open more issues as you come across them!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rcthomas @jacobtomlinson