Coiled notebooks don't work with the ghcr.io/dask/dask-notebook:dev-py3.12 image

[jacobtomlinson]

When trying to use the ghcr.io/dask/dask-notebook:dev-py3.12 container with Coiled Notebooks it fails to start up. Am I doing something wrong? Do we need to fix something in the Dask container images?

$ coiled notebook start --container ghcr.io/dask/dask-notebook:dev-py3.12
╭─────────────────────── Notebook notebook-d888299a... ────────────────────────╮
│                                                                              │
│ Jupyter: ..                                                                  │
│ Details: https://cloud.coiled.io/clusters/625353?account=rapidsai            │
│                                                                              │
│ Error  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━      │
│                                                                              │
│ Environment:                                                                 │
│ dask-notebook_dev-py3_12-x86_64-7a4d7702-f0b4-5f64-af50-b5df71385ee4         │
│ Region:   us-east-1                Uptime:                               45s │
│ VM Type:  m6i.xlarge               Approx cloud cost:               $0.19/hr │
│                                    Approx cloud total:                 $0.00 │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/bin/coiled", line 8, in <module>
    sys.exit(cli())
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/coiled/cli/notebook/notebook.py", line 244, in start_notebook
    _start_notebook(
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/coiled/cli/notebook/notebook.py", line 490, in _start_notebook
    raise e
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/coiled/cli/notebook/notebook.py", line 344, in _start_notebook
    cluster = coiled.Cluster(
  File "/Users/jtomlinson/miniconda3/lib/python3.10/site-packages/coiled/v2/cluster.py", line 908, in __init__
    raise e.with_traceback(None)  # noqa: B904
coiled.errors.ClusterCreationError: Cluster status is error (reason: Scheduler Stopped -> Software environment exited with error code 1. json.decoder.JSONDecodeError: Expecting value: line 1 column 8 (char 7)) (cluster_id: 625353)

[ntabris]

I'm not familiar with that particular container. My guess is that it has an entrypoint that's getting in the way of us running python -m distributed.cli.dask_spec. If you're familiar with this container, do you happen to know if there's anything special we'd need to do to make that work—i.e., set an ENV var like on the rapids containers, or ignore the entrypoint?

In more detail, here's more detail about what we want to work:

https://docs.coiled.io/user_guide/software/docker.html#verification

coiled notebook start doesn't have a flag to ignore the entrypoint, but we could add that. For now you have to explicitly create the software environment to control this option (just a more verbose API, still using the existing docker image).

[jacobtomlinson]

Yeah I am familiar so we can definitely find a way to fix this up. Thanks for the verification docs, those are very helpful.

It looks like the quotes are being stripped from the JSON string during the entrypoint of the upstream image.

$ docker run --rm ghcr.io/dask/dask-notebook:dev-py3.12 python -m distributed.cli.dask_spec --spec '{"cls":"dask.distributed.Scheduler", "opts":{}}'
+ '[' '' ']'
+ '[' '' == true ']'
+ CONDA_BIN=/opt/conda/bin/conda
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
no environment.yml
+ '[' '' ']'
+ '[' '' ']'
+ exec start.sh python -m distributed.cli.dask_spec --spec '{"cls":"dask.distributed.Scheduler",' '"opts":{}}'
Entered start.sh with args: python -m distributed.cli.dask_spec --spec {"cls":"dask.distributed.Scheduler", "opts":{}}
Running hooks in: /usr/local/bin/start-notebook.d as uid: 1000 gid: 100
Done running hooks in: /usr/local/bin/start-notebook.d
Running hooks in: /usr/local/bin/before-notebook.d as uid: 1000 gid: 100
Sourcing shell script: /usr/local/bin/before-notebook.d/10activate-conda-env.sh
Done running hooks in: /usr/local/bin/before-notebook.d
Executing the command: python -m distributed.cli.dask_spec --spec {"cls":"dask.distributed.Scheduler", "opts":{}}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/lib/python3.12/site-packages/distributed/cli/dask_spec.py", line 67, in <module>
    main()
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/distributed/cli/dask_spec.py", line 33, in main
    _spec.update(json.loads(spec))
                 ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 37 (char 36)