Skip to content

Commit dd1afdb

Browse files
satrasatra
authored
Merge pull request #13 from yarikoptic/enh-docker
ENH: prepare docker image using neurodebian snapshots
2 parents de7daac + a18ddb7 commit dd1afdb

File tree

4 files changed

+176
-3
lines changed

4 files changed

+176
-3
lines changed

‎README.md‎

Lines changed: 47 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,10 @@
55
#### Information queried from NITRC-IR and stored in a google spreadsheet
66
https://docs.google.com/spreadsheets/d/11an55u9t2TAf0EV2pHN0vOd8Ww2Gie-tHp9xGULh_dA
77

8-
#### Setup your own environment
8+
#### Within your current environment
9+
10+
##### Setup
11+
912
1. Make sure FSL is available in your environment and accessible from the command line.
1013

1114
2. If you already have a `conda` environment, please follow the detailed steps below.
@@ -17,7 +20,7 @@ curl -Ok https://raw.githubusercontent.com/ReproNim/simple_workflow/e4063fa95cb4
1720
source Simple_Prep.sh
1821
```
1922

20-
#### Run the demo
23+
##### Run the demo
2124

2225
```bash
2326
python run_demo_workflow.py --key 11an55u9t2TAf0EV2pHN0vOd8Ww2Gie-tHp9xGULh_dA
@@ -29,7 +32,7 @@ To run on one subject you can do:
2932
python run_demo_workflow.py --key 11an55u9t2TAf0EV2pHN0vOd8Ww2Gie-tHp9xGULh_dA -n 1
3033
```
3134

32-
### Detailed steps for setting up environment
35+
#### Detailed steps for setting up environment
3336

3437
Install miniconda if you do not have it.
3538

@@ -64,3 +67,44 @@ conda env create -f environment.yml
6467
source activate bh_demo
6568
pip install https://github.com/satra/prov/archive/enh/rdf-1.x.zip
6669
```
70+
71+
### Within Docker
72+
73+
Using containerization solutions, such as docker, allows to create
74+
multiple complete computation environments while varying versions of any
75+
analysis pipeline components or inputs. You could use [Simple_Prep_docker](Simple_Prep_docker)
76+
script to generate environments based on previous [Debian](http://www.debian.org) or [Ubuntu](http://ubuntu.com) releases
77+
for which [NeuroDebian](http://neuro.debian.net) builds of FSL [were available in the past](http://snapshot-neuro.debian.net:5002/package/fsl).
78+
79+
N.B. ATM NeuroDebian snapshots repository is not widely open yet, so if
80+
you would like to browse it, please "knock" first by running
81+
`curl -s http://neuro.debian.net/_files/knock-snapshots` command in your shell.
82+
83+
#### Generate an environment
84+
85+
For an example we will generate an environment based on Debian jessie
86+
release with FSL 5.0.8-3 as it was available in March of 2015:
87+
88+
```bash
89+
./Simple_Prep_docker jessie 20150306T060524Z
90+
```
91+
92+
which will generate a local docker image `repronim:simple_prep_USER_jessie_20150306T060524Z`
93+
(`USER` will correspond to your user name), with all necessary for computation
94+
components installed.
95+
96+
#### Run the demo
97+
98+
You can normally run a demo with only **one additional step** necessary -- setting up
99+
environment variables (to point to FSL binaries and enable conda environment):
100+
101+
```bash
102+
docker run -it --rm repronim:simple_prep_USER_jessie_20150306T060524Z /bin/bash -c '
103+
. setup_environment;
104+
cd simple_workflow-master
105+
&& python run_demo_workflow.py --key 11an55u9t2TAf0EV2pHN0vOd8Ww2Gie-tHp9xGULh_dA
106+
&& python check_output.py'
107+
```
108+
109+
which would generate a new temporary container, perform analysis, run
110+
the check, and quit.

‎Simple_Prep_docker‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/bin/bash
2+
#emacs: -*- mode: shell-script; c-basic-offset: 4; tab-width: 4; indent-tabs-mode: t -*-
3+
#ex: set sts=4 ts=4 sw=4 noet:
4+
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
5+
#
6+
# See COPYING file distributed along with the XXX package for the
7+
# copyright and license terms.
8+
#
9+
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
10+
#
11+
# Helper to generate a Docker instance given release and snapshots date specification
12+
# and run Simple_Prep.sh analysis
13+
#
14+
set -e
15+
export PS4=+
16+
#set -x
17+
set -u
18+
19+
DL_DIST=${1:-jessie}
20+
DL_DATE=${2:-}
21+
22+
topdir=$(readlink -f `dirname $0`)
23+
cd `dirname $0`
24+
dockerfile=$topdir/Simple_Prep_docker-Dockerfile
25+
# echo "D: $DL_APT"
26+
# Some substitutes aren't used but left for possibly later use
27+
sed -e "s,DL_DIST,$DL_DIST,g" \
28+
-e "s,DL_USER,$USER,g" \
29+
-e "s,DL_UID,`id -u`,g" \
30+
-e "s,DL_GID,`id -g`,g" \
31+
-e "s,DL_GIT_USER_EMAIL,`git config --get user.email`,g" \
32+
-e "s,DL_GIT_USER_NAME,`git config --get user.name`,g" \
33+
$dockerfile.in >| $dockerfile
34+
35+
if [ ! -z "$DL_DATE" ]; then
36+
{
37+
# replace with snapshots
38+
echo "sed -i -e 's,http://neuro.debian.net/debian/* ,http://snapshot-neuro.debian.net:5002/archive/neurodebian/$DL_DATE/ ,g' etc/apt/sources.list.d/neurodebian.sources.list;"
39+
# for now we need to knock to expose snapshots repository
40+
echo "curl -s http://neuro.debian.net/_files/knock-snapshots;"
41+
# TODO: figure out and add snapshot for DL_DIST itself if available and force possible downgrade somehow
42+
echo "eatmydata apt-get update; eatmydata apt-get dist-upgrade -y;"
43+
} | while read aptline; do
44+
sed -i -e "s|\(\(.*\)DL_APT\(.*\)\)|\2$aptline\3\n\1|g" $dockerfile
45+
:
46+
done
47+
fi
48+
sed -e '/DL_APT/d' -i $dockerfile
49+
50+
tag_=simple_prep_${USER}_${DL_DIST}_${DL_DATE}
51+
tag=repronim:${tag_}
52+
echo "I: tag $tag"
53+
if docker images | grep -q repronim.*${tag_}; then
54+
echo "I: tag $tag already exists -- skipping rebuilding"
55+
else
56+
docker build -t $tag -f $dockerfile .
57+
#docker build --no-cache=True -t $tag -f $dockerfile . #&& rm Dockerfile
58+
:
59+
fi
60+

‎Simple_Prep_docker-Dockerfile.in‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
FROM neurodebian:DL_DIST
2+
MAINTAINER Yaroslav Halchenko <[email protected]>
3+
4+
USER root
5+
6+
# Speed up installation using our apt cacher - modify conf/etc/apt/apt.conf.d/99apt-cacher if you have any
7+
RUN mkdir -p /etc/apt/apt.conf.d/
8+
COPY conf/etc/apt/apt.conf.d/99apt-cacher /etc/apt/apt.conf.d/99apt-cacher
9+
RUN chmod a+r /etc/apt/apt.conf.d/99apt-cacher
10+
11+
# Make deb-src avail
12+
# RUN sed -i -e 's,^deb\(.*\),deb\1\ndeb-src\1,g' /etc/apt/sources.list.d/neurodebian.sources.list /etc/apt/sources.list
13+
14+
# Make contrib and non-free avail for FSL
15+
RUN sed -i -e 's, main$, main contrib non-free,g' /etc/apt/sources.list.d/neurodebian.sources.list
16+
17+
# Assure popcon doesn't kick in
18+
RUN bash -c "echo 'debconf debconf/frontend select noninteractive' | debconf-set-selections -"
19+
20+
RUN apt-get update
21+
# Use bash for extended syntax
22+
RUN apt-get install -y -q eatmydata
23+
# Some rudimentary tools if we need to do anything within docker and curl and unzip needed for setting up conda
24+
RUN bash -c "eatmydata apt-get install -y -q vim less man-db curl unzip bzip2"
25+
# Run additional lines, primarily to setup/enable snapshots repository etc
26+
RUN bash -c "DL_APT"
27+
# Install fsl-complete
28+
#RUN bash -c "eatmydata apt-get install -y -q fsl-complete"
29+
# We might be just fine with the core here
30+
RUN bash -c "eatmydata apt-get install -y -q fsl-core fsl-first-data"
31+
32+
RUN apt-get clean
33+
34+
# Setting up conda environment given simple_workflow specifications
35+
WORKDIR /opt/repronim/simple_workflow
36+
RUN curl -Ok https://raw.githubusercontent.com/ReproNim/simple_workflow/e4063fa95cb494da496565ec27c4ffe8a4901c45/Simple_Prep.sh
37+
# conda installations take way too long -- might benefit from setting up
38+
# to use proxy
39+
# http://stackoverflow.com/a/31120854
40+
RUN bash Simple_Prep.sh
41+
42+
#
43+
# There seems to be no easy consistent way to load our customizations to env
44+
# variables for both interactive and non-interactive shells. So let's create
45+
# a file which would have all the necessary tuneups which would be passed
46+
# explicitly into bash
47+
#
48+
RUN bash -c 'echo -e "echo IN setup_environments\n. /etc/fsl/fsl.sh\nexport PATH=/root/miniconda2/bin:\$PATH\nsource activate bh_demo\n" > setup_environment'
49+
50+
# Make fsl available in the containers by pointing ENV variable to it
51+
# which both bash and dash (ubuntu) should warrant
52+
# ENV ENV /etc/fsl/fsl.sh
53+
# not effective :-/ for now let's try to place in bashrc
54+
# RUN bash -c "echo '. /etc/fsl/fsl.sh' >> /etc/bash.bashrc"
55+
56+
# Tune the environment variables
57+
#ENV PATH /root/miniconda2/bin:$PATH
58+
# RUN bash -c "echo 'source activate bh_demo' >> /etc/bash.bashrc"
59+
60+
# Tune bash behavior so it loads our environment setup even in non-interactive mode
61+
ENV ENV /opt/repronim/simple_workflow/setup_environment
62+
RUN bash -c 'echo -e ". /opt/repronim/simple_workflow/setup_environment" >> /etc/profile'
63+
64+
# Let's setup user matching user
65+
## RUN groupadd --gid DL_GID -r DL_USER && useradd -m --uid DL_UID -g DL_USER DL_USER
66+
67+
## CMD ["/bin/bash"]

‎conf/etc/apt/apt.conf.d/99apt-cacher‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#Acquire::http { Proxy "http://172.17.42.1:3142"; };
2+
#Acquire::http { Proxy "http://smaug.datalad.org:3142"; };

0 commit comments

Comments
 (0)