You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Act and Edit are no-ops on inactive maps (#155)
* resolves#145
* Improvements to the map.stderr/stdout API (#149)
* Add htmap-exec Docker image and change default image to it (#153)
* move test infrastructure into tests dir
* add htmap-exec image
* updates docs
* Transferring Arbitrary Output Files (#151)
Copy file name to clipboardexpand all lines: docs/source/recipes/docker-image-cookbook.rst
+66-19
Original file line number
Diff line number
Diff line change
@@ -14,15 +14,20 @@ are installed on the computers your code actually runs on.
14
14
15
15
To use Docker, you write a **Dockerfile** which tells Docker how to generate an **image**,
16
16
which is a blueprint to construct a **container**.
17
-
The Dockerfile is a list of instructions, such as shell commands or instructions for Docker to copy files from the build environment into the image.
17
+
The Dockerfile is a list of instructions, such as shell commands or instructions
18
+
for Docker to copy files from the build environment into the image.
18
19
You then tell Docker to "build" the image from the Dockerfile.
19
20
20
-
For use with HTMap, you then upload this image to `Docker Hub <https://hub.docker.com>`_, where it can then be downloaded to execute nodes in an HTCondor pool.
21
-
When your HTMap component lands on an execute node, HTCondor will download your image from Docker Hub and run your code inside it using HTMap.
21
+
For use with HTMap, you then upload this image to `Docker Hub <https://hub.docker.com>`_,
22
+
where it can then be downloaded to execute nodes in an HTCondor pool.
23
+
When your HTMap component lands on an execute node, HTCondor will download your
24
+
image from Docker Hub and run your code inside it using HTMap.
22
25
23
-
The following sections describe, roughly in order of increasing complexity, different ways to build Docker images for use with HTMap.
26
+
The following sections describe, roughly in order of increasing complexity,
27
+
different ways to build Docker images for use with HTMap.
24
28
Each level of complexity is introduced to solve a more advanced dependency management problem.
25
-
We recommend reading them in order until reach one that works for your dependencies (each section assumes knowledge of the previous sections).
29
+
We recommend reading them in order until reach one that works for your dependencies
30
+
(each section assumes knowledge of the previous sections).
26
31
27
32
More detailed information on how Dockerfiles work can be found
28
33
`in the Docker documentation itself <https://docs.docker.com/engine/reference/builder/>`_
@@ -37,10 +42,11 @@ This page only covers the bare minimum to get started with HTMap and Docker.
37
42
Can I use HTMap's default image?
38
43
--------------------------------
39
44
40
-
HTMap's default Docker image is `continuumio/anaconda3:latest <https://hub.docker.com/r/continuumio/anaconda3/>`_.
45
+
HTMap's default Docker image is `htcondor/htmap-exec <https://hub.docker.com/r/htcondor/htmap-exec/>`_,
46
+
which is itself based on`continuumio/anaconda3 <https://hub.docker.com/r/continuumio/anaconda3/>`_.
41
47
It is based on Python 3 and has many useful packages pre-installed, such as ``numpy``, ``scipy``, and ``pandas``.
42
-
If your software only depends on packages included in the `Anaconda distribution <https://docs.anaconda.com/anaconda/packages/pkg-docs/>`_ by default,
43
-
you can use HTMap's default and won't need to create your own image.
48
+
If your software only depends on packages included in the `Anaconda distribution <https://docs.anaconda.com/anaconda/packages/pkg-docs/>`_,
49
+
you can use HTMap's default image and won't need to create your own.
44
50
45
51
46
52
I depend on Python packages that aren't in the Anaconda distribution
@@ -52,13 +58,14 @@ I depend on Python packages that aren't in the Anaconda distribution
52
58
and `make an account on Docker Hub <https://hub.docker.com/>`_.
53
59
54
60
55
-
Let's pretend that there's a package called ``foobar`` that your Python code depends on, but isn't part of the Anaconda distribution.
61
+
Let's pretend that there's a package called ``foobar`` that your Python function depends on,
62
+
but isn't part of the Anaconda distribution.
56
63
You will need to write your own Dockerfile to include this package in your Docker image.
57
64
58
65
Docker images are built in **layers**.
59
66
You always start a Dockerfile by stating which existing Docker image you'd like to use as your base layer.
60
67
A good choice is the same Anaconda image that HTMap uses as the default,
61
-
which comes with both the ``conda`` package manager and the standard ``pip``.
68
+
which comes with both the ``conda`` package manager and the standard ``pip``.
62
69
Create a file named ``Dockerfile`` and write this into it:
63
70
64
71
.. code-block:: docker
@@ -67,18 +74,41 @@ Create a file named ``Dockerfile`` and write this into it:
67
74
68
75
FROM continuumio/anaconda3:latest
69
76
70
-
Lines that begin with a ``#`` are comments in a Dockerfile.
77
+
RUN pip install --no-cache-dir htmap
78
+
79
+
ARG USER=htmap
80
+
RUN groupadd ${USER} \
81
+
&& useradd -m -g ${USER} ${USER}
82
+
USER ${USER}
71
83
72
84
Each line in the Dockerfile starts with a short, capitalized word which tells Docker what kind of build instruction it is.
73
-
``FROM`` means "start with this base image".
74
-
Now we need to tell Docker to run a shell command during the build to install ``foobar``.
85
+
86
+
* ``FROM`` means "start with this base image".
87
+
* ``RUN`` means "execute these shell commands in the container".
88
+
* ``ARG`` means "set build argument" - it acts like an environment variable that's only set during the image build.
89
+
90
+
Lines that begin with a ``#`` are comments in a Dockerfile.
91
+
The above lines say that we want to inherit from the image ``continuumio/anaconda3:latest`` and build on top of it.
92
+
To be compatible with HTMap, we install ``htmap`` via ``pip``.
93
+
We also set up a non-root user to do the execution, which is important for security.
94
+
Naming that user ``htmap`` is arbitrary and has nothing to do with the ``htmap`` package itself.
95
+
96
+
Now we need to tell Docker to run a shell command during the build to install ``foobar``
97
+
by adding one more line to the bottom of the Dockerfile.
75
98
76
99
.. code-block:: docker
77
100
78
101
# Dockerfile
79
102
80
103
FROM continuumio/anaconda3:latest
81
104
105
+
RUN pip install --no-cache-dir htmap
106
+
107
+
ARG USER=htmap
108
+
RUN groupadd ${USER} \
109
+
&& useradd -m -g ${USER} ${USER}
110
+
USER ${USER}
111
+
82
112
# if foobar can be install via conda, use these lines
83
113
RUN conda install -y foobar \
84
114
&& conda clean -y --all
@@ -101,6 +131,13 @@ If you need install many packages, we recommend writing a ``requirements.txt`` f
101
131
102
132
FROM continuumio/anaconda3:latest
103
133
134
+
RUN pip install --no-cache-dir htmap
135
+
136
+
ARG USER=htmap
137
+
RUN groupadd ${USER} \
138
+
&& useradd -m -g ${USER} ${USER}
139
+
USER ${USER}
140
+
104
141
COPY requirements.txt requirements.txt
105
142
RUN pip install --no-cache-dir -r requirements.txt
106
143
@@ -153,10 +190,13 @@ Instead of using the full Anaconda distribution, use a base Docker image that on
153
190
154
191
FROM continuumio/miniconda3:latest
155
192
156
-
RUN conda install -y cloudpickle \
157
-
&& conda clean -y -all
193
+
RUN pip install --no-cache-dir htmap
194
+
195
+
ARG USER=htmap
196
+
RUN groupadd ${USER} \
197
+
&& useradd -m -g ${USER} ${USER}
198
+
USER ${USER}
158
199
159
-
Note that we need to install ``cloudpickle``, which HTMap depends on execute-side, ourselves.
160
200
From here, install your particular dependencies as above.
161
201
162
202
If you prefer to not use ``conda``, an even-barer-bones image could be produced from
@@ -167,8 +207,14 @@ If you prefer to not use ``conda``, an even-barer-bones image could be produced
167
207
168
208
FROM python:latest
169
209
170
-
RUN pip install --no-cache-dir cloudpickle
210
+
RUN pip install --no-cache-dir htmap
211
+
212
+
ARG USER=htmap
213
+
RUN groupadd ${USER} \
214
+
&& useradd -m -g ${USER} ${USER}
215
+
USER ${USER}
171
216
217
+
We use ``python:latest`` as our base image, so we don't have ``conda`` anymore.
172
218
173
219
I want to use a Python package that's not on PyPI or Anaconda
@@ -225,8 +271,9 @@ We recommend adding ``miniconda`` to the image by adding these lines to your Doc
225
271
&& conda install python=${PYTHON_VERSION} \
226
272
&& conda clean -y -all
227
273
228
-
After this, you can install any other Python packages you need as in the preceeding sections.
274
+
After this, you can install HTMap and any other Python packages you need as in the preceeding sections.
229
275
230
276
Note that in this example we based the image on Ubuntu's base image and installed ``wget``,
231
277
which we used to download the ``miniconda`` installer.
232
-
Depending on your base image, you may need to use a different package manager (for example, ``yum``) or different command-line file download tool (for example, ``curl``).
278
+
Depending on your base image, you may need to use a different package manager
279
+
(for example, ``yum``) or different command-line file download tool (for example, ``curl``).
0 commit comments