Skip to content

Commit f8c048c

Browse files
wk8wk8
committedSep 13, 2018
Adding a Dockerfile and making it easy to use it for dev
Credit goes to dperny (#2687) **- What I did** Adds a Dockerfile for the swarmkit project, to easily get off the ground. Modifies the Makefile to make intelligent use of Docker. Also made small clean up changes to the Makefile. **- How I did it** Modifies the Makefile to have two paths: containerized.mk, which builds the docker image and forwards any make targets to a container, and direct.mk, which encompasses the old Makefile's workflow. By default, nothing will run inside a container. Set the environment variable `DOCKER_SWARMKIT_USE_CONTAINER` to use dockerized making. Also leverages docker-sync for synchronizing code to the container if the `DOCKER_SWARMKIT_USE_DOCKER_SYNC` env variable is set; comes in handy on Macs, for example. **- How to test it** Set `DOCKER_SWARMKIT_USE_CONTAINER` and verify that your favorite make targets all work! Signed-off-by: Jean Rouge <[email protected]>
1 parent d7d23d7 commit f8c048c

7 files changed

+271
-133
lines changed
 

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,6 @@ bin/swarmkitstate
3131

3232
# ignore code coverage output
3333
*coverage.txt
34+
35+
# dev sync, if used
36+
/.docker-sync/

‎BUILDING.md

+12
Original file line numberDiff line numberDiff line change
@@ -113,3 +113,15 @@ NB: As of version 3.0.0-7 the Debian `protobuf-compiler` package lacks
113113
a dependency on `libprotobuf-dev` which contains some standard proto
114114
definitions, be sure to install both packages. This is [Debian bug
115115
#842158](https://bugs.debian.org/842158).
116+
117+
### Build in a container instead of your local environment
118+
119+
You can also choose to use a container to build SwarmKit and run tests. Simply
120+
set the `DOCKER_SWARMKIT_USE_CONTAINER` environment variable to any value,
121+
export it, then run `make` targets as you would have done within your local
122+
environment.
123+
124+
Additionally, if your OS is not Linux, you might want to set and export the
125+
`DOCKER_SWARMKIT_USE_DOCKER_SYNC` environment variable, which will make use of
126+
[docker-sync](https://github.com/EugenMayer/docker-sync) to sync the code to
127+
the container, instead of native mounted volumes.

‎Dockerfile

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# NOTE(dperny): for some reason, alpine was giving me trouble
2+
FROM golang:1.10.3-stretch
3+
4+
RUN apt-get update && apt-get install -y make git unzip
5+
6+
# should stay consistent with the version we use in Circle builds
7+
ARG PROTOC_VERSION=3.5.0
8+
# make a directory to do these operations in
9+
RUN export PROTOC_TMP_DIR=protoc && mkdir -p $PROTOC_TMP_DIR && cd $PROTOC_TMP_DIR \
10+
# download the pre-built protoc binary
11+
&& curl --silent --show-error --location --output protoc.zip \
12+
https://github.com/google/protobuf/releases/download/v$PROTOC_VERSION/protoc-$PROTOC_VERSION-linux-x86_64.zip \
13+
# move the binary to /bin. move the well-known types ot /usr/local/include
14+
&& unzip protoc.zip && mv bin/protoc /bin/protoc && mv include/* /usr/local/include \
15+
# remove all of the installation files
16+
&& cd .. && rm -rf $PROTOC_TMP_DIR
17+
18+
WORKDIR /go/src/github.com/docker/swarmkit/
19+
20+
# install the dependencies from `make setup`
21+
# we only copy `direct.mk` to avoid busting the cache too easily
22+
COPY direct.mk .
23+
RUN make --file=direct.mk setup
24+
25+
# now we can copy the rest
26+
COPY . .
27+
28+
# default to just `make`. If you want to change the default command, change the
29+
# default make command, not this command.
30+
CMD ["make"]

‎Makefile

+12-133
Original file line numberDiff line numberDiff line change
@@ -24,136 +24,15 @@ VNDR=$(shell which vndr || echo '')
2424

2525
GO_LDFLAGS=-ldflags "-X `go list ./version`.Version=$(VERSION)"
2626

27-
.PHONY: clean all AUTHORS fmt vet lint build binaries test integration setup generate protos checkprotos coverage ci check help install uninstall dep-validate
28-
.DEFAULT: default
29-
30-
all: check binaries test integration ## run fmt, vet, lint, build the binaries and run the tests
31-
32-
check: fmt vet lint ineffassign misspell ## run fmt, vet, lint, ineffassign, misspell
33-
34-
ci: check binaries checkprotos coverage coverage-integration ## to be used by the CI
35-
36-
AUTHORS: .mailmap .git/HEAD
37-
git log --format='%aN <%aE>' | sort -fu > $@
38-
39-
# This only needs to be generated by hand when cutting full releases.
40-
version/version.go:
41-
./version/version.sh > $@
42-
43-
setup: ## install dependencies
44-
@echo "🐳 $@"
45-
# TODO(stevvooe): Install these from the vendor directory
46-
@go get -u github.com/golang/lint/golint
47-
#@go get -u github.com/kisielk/errcheck
48-
@go get -u github.com/gordonklaus/ineffassign
49-
@go get -u github.com/client9/misspell/cmd/misspell
50-
@go get -u github.com/lk4d4/vndr
51-
@go get -u github.com/stevvooe/protobuild
52-
53-
generate: protos
54-
@echo "🐳 $@"
55-
@PATH=${ROOTDIR}/bin:${PATH} go generate -x ${PACKAGES}
56-
57-
protos: bin/protoc-gen-gogoswarm ## generate protobuf
58-
@echo "🐳 $@"
59-
@PATH=${ROOTDIR}/bin:${PATH} protobuild ${PACKAGES}
60-
61-
checkprotos: generate ## check if protobufs needs to be generated again
62-
@echo "🐳 $@"
63-
@test -z "$$(git status --short | grep ".pb.go" | tee /dev/stderr)" || \
64-
((git diff | cat) && \
65-
(echo "👹 please run 'make generate' when making changes to proto files" && false))
66-
67-
# Depends on binaries because vet will silently fail if it can't load compiled
68-
# imports
69-
vet: binaries ## run go vet
70-
@echo "🐳 $@"
71-
@test -z "$$(go vet ${PACKAGES} 2>&1 | grep -v 'constant [0-9]* not a string in call to Errorf' | egrep -v '(timestamp_test.go|duration_test.go|exit status 1)' | tee /dev/stderr)"
72-
73-
misspell:
74-
@echo "🐳 $@"
75-
@test -z "$$(find . -type f | grep -v vendor/ | grep -v bin/ | grep -v .git/ | grep -v MAINTAINERS | xargs misspell | tee /dev/stderr)"
76-
77-
fmt: ## run go fmt
78-
@echo "🐳 $@"
79-
@test -z "$$(gofmt -s -l . | grep -v vendor/ | grep -v ".pb.go$$" | tee /dev/stderr)" || \
80-
(echo "👹 please format Go code with 'gofmt -s -w'" && false)
81-
@test -z "$$(find . -path ./vendor -prune -o ! -name timestamp.proto ! -name duration.proto -name '*.proto' -type f -exec grep -Hn -e "^ " {} \; | tee /dev/stderr)" || \
82-
(echo "👹 please indent proto files with tabs only" && false)
83-
@test -z "$$(find . -path ./vendor -prune -o -name '*.proto' -type f -exec grep -Hn "Meta meta = " {} \; | grep -v '(gogoproto.nullable) = false' | tee /dev/stderr)" || \
84-
(echo "👹 meta fields in proto files must have option (gogoproto.nullable) = false" && false)
85-
86-
lint: ## run go lint
87-
@echo "🐳 $@"
88-
@test -z "$$(golint ./... | grep -v vendor/ | grep -v ".pb.go:" | tee /dev/stderr)"
89-
90-
ineffassign: ## run ineffassign
91-
@echo "🐳 $@"
92-
@test -z "$$(ineffassign . | grep -v vendor/ | grep -v ".pb.go:" | tee /dev/stderr)"
93-
94-
#errcheck: ## run go errcheck
95-
# @echo "🐳 $@"
96-
# @test -z "$$(errcheck ./... | grep -v vendor/ | grep -v ".pb.go:" | tee /dev/stderr)"
97-
98-
build: ## build the go packages
99-
@echo "🐳 $@"
100-
@go build -i -tags "${DOCKER_BUILDTAGS}" -v ${GO_LDFLAGS} ${GO_GCFLAGS} ${PACKAGES}
101-
102-
test: ## run tests, except integration tests
103-
@echo "🐳 $@"
104-
@go test -parallel 8 ${RACE} -tags "${DOCKER_BUILDTAGS}" $(filter-out ${INTEGRATION_PACKAGE},${PACKAGES})
105-
106-
integration: ## run integration tests
107-
@echo "🐳 $@"
108-
@go test -parallel 8 ${RACE} -tags "${DOCKER_BUILDTAGS}" ${INTEGRATION_PACKAGE}
109-
110-
FORCE:
111-
112-
# Build a binary from a cmd.
113-
bin/%: cmd/% FORCE
114-
@test $$(go list) = "${PROJECT_ROOT}" || \
115-
(echo "👹 Please correctly set up your Go build environment. This project must be located at <GOPATH>/src/${PROJECT_ROOT}" && false)
116-
@echo "🐳 $@"
117-
@go build -i -tags "${DOCKER_BUILDTAGS}" -o $@ ${GO_LDFLAGS} ${GO_GCFLAGS} ./$<
118-
119-
binaries: $(BINARIES) ## build binaries
120-
@echo "🐳 $@"
121-
122-
clean: ## clean up binaries
123-
@echo "🐳 $@"
124-
@rm -f $(BINARIES)
125-
126-
install: $(BINARIES) ## install binaries
127-
@echo "🐳 $@"
128-
@mkdir -p $(DESTDIR)/bin
129-
@install $(BINARIES) $(DESTDIR)/bin
130-
131-
uninstall:
132-
@echo "🐳 $@"
133-
@rm -f $(addprefix $(DESTDIR)/bin/,$(notdir $(BINARIES)))
134-
135-
coverage: ## generate coverprofiles from the unit tests
136-
@echo "🐳 $@"
137-
@( for pkg in $(filter-out ${INTEGRATION_PACKAGE},${PACKAGES}); do \
138-
go test -i ${RACE} -tags "${DOCKER_BUILDTAGS}" -test.short -coverprofile="../../../$$pkg/coverage.txt" -covermode=atomic $$pkg || exit; \
139-
go test ${RACE} -tags "${DOCKER_BUILDTAGS}" -test.short -coverprofile="../../../$$pkg/coverage.txt" -covermode=atomic $$pkg || exit; \
140-
done )
141-
142-
coverage-integration: ## generate coverprofiles from the integration tests
143-
@echo "🐳 $@"
144-
go test ${RACE} -tags "${DOCKER_BUILDTAGS}" -test.short -coverprofile="../../../${INTEGRATION_PACKAGE}/coverage.txt" -covermode=atomic ${INTEGRATION_PACKAGE}
145-
146-
help: ## this help
147-
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) | sort
148-
149-
dep-validate:
150-
@echo "+ $@"
151-
$(if $(VNDR), , \
152-
$(error Please install vndr: go get github.com/lk4d4/vndr))
153-
@rm -Rf .vendor.bak
154-
@mv vendor .vendor.bak
155-
@$(VNDR)
156-
@test -z "$$(diff -r vendor .vendor.bak 2>&1 | tee /dev/stderr)" || \
157-
(echo >&2 "+ inconsistent dependencies! what you have in vendor.conf does not match with what you have in vendor" && false)
158-
@rm -Rf vendor
159-
@mv .vendor.bak vendor
27+
SHELL := /bin/bash
28+
29+
# stop here. do we want to run everything inside of a container, or do we want
30+
# to run it directly on the host? if the user has set ANY non-empty value for
31+
# the variable DOCKER_SWARMKIT_USE_CONTAINER, then we do all of the making
32+
# inside of a container. We will default to using no container, to avoid
33+
# breaking anyone's workflow
34+
ifdef DOCKER_SWARMKIT_USE_CONTAINER
35+
include containerized.mk
36+
else
37+
include direct.mk
38+
endif

‎containerized.mk

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
IMAGE_NAME=docker/swarmkit
2+
GOPATH=/go
3+
DOCKER_IMAGE_DIR=${GOPATH}/src/${PROJECT_ROOT}
4+
5+
# don't bother writing every single make target. just pass the call through to
6+
# docker and make
7+
# we prefer `%:` to `.DEFAULT` as the latter doesn't run phony deps
8+
# (see https://www.gnu.org/software/make/manual/html_node/Special-Targets.html)
9+
%::
10+
@ echo "Running target $@ inside a container"
11+
@ DOCKER_SWARMKIT_DOCKER_RUN_CMD="make $*" $(MAKE) run
12+
13+
shell:
14+
@ DOCKER_SWARMKIT_DOCKER_RUN_CMD='bash' DOCKER_SWARMKIT_DOCKER_RUN_FLAGS='-i' $(MAKE) run
15+
16+
.PHONY: image
17+
image:
18+
docker build -t ${IMAGE_NAME} .
19+
20+
# internal target, only builds the image if it doesn't exist
21+
.PHONY: ensure_image_exists
22+
ensure_image_exists:
23+
@ if [ ! $$(docker images -q ${IMAGE_NAME}) ]; then $(MAKE) image; fi
24+
25+
# internal target, starts the sync if needed
26+
# uses https://github.com/EugenMayer/docker-sync/blob/47363ee31b71810a60b05822b9c4bd2176951ce8/tasks/sync/sync.thor#L193-L196
27+
# which is not great, but that's all they expose so far to do this...
28+
# checks if the daemon pid in the .docker-sync directory maps to a running
29+
# process owned by the current user, and otherwise assumes the sync is not
30+
# running, and starts it
31+
.PHONY: ensure_sync_started
32+
ensure_sync_started:
33+
@ kill -0 $$(cat .docker-sync/daemon.pid) 2&> /dev/null || docker-sync start
34+
35+
# internal target, actually runs a command inside a container
36+
# we don't use the `-i` flag for `docker run` by default as that makes it a pain
37+
# to kill running containers (can't kill with ctrl-c)
38+
.PHONY: run
39+
run: ensure_image_exists
40+
@ [ "$$DOCKER_SWARMKIT_DOCKER_RUN_CMD" ] || exit 1
41+
@ DOCKER_RUN_COMMAND="docker run -t -v swarmkit-cache:${GOPATH}" \
42+
&& if [ "$$DOCKER_SWARMKIT_USE_DOCKER_SYNC" ]; then \
43+
$(MAKE) ensure_sync_started && DOCKER_RUN_COMMAND="$$DOCKER_RUN_COMMAND -v swarmkit-sync:${DOCKER_IMAGE_DIR}"; \
44+
else \
45+
DOCKER_RUN_COMMAND="$$DOCKER_RUN_COMMAND -v ${ROOTDIR}:${DOCKER_IMAGE_DIR}"; \
46+
fi \
47+
&& DOCKER_RUN_COMMAND="$$DOCKER_RUN_COMMAND $$DOCKER_SWARMKIT_DOCKER_RUN_FLAGS ${IMAGE_NAME} $$DOCKER_SWARMKIT_DOCKER_RUN_CMD" \
48+
&& echo $$DOCKER_RUN_COMMAND \
49+
&& $$DOCKER_RUN_COMMAND

‎direct.mk

+156
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
.DEFAULT_GOAL = all
2+
.PHONY: all
3+
all: check binaries test integration-tests ## run fmt, vet, lint, build the binaries and run the tests
4+
5+
.PHONY: check
6+
check: fmt vet lint ineffassign misspell
7+
8+
.PHONY: ci
9+
ci: check binaries checkprotos coverage coverage-integration ## to be used by the CI
10+
11+
.PHONY: AUTHORS
12+
AUTHORS: .mailmap .git/HEAD
13+
git log --format='%aN <%aE>' | sort -fu > $@
14+
15+
# This only needs to be generated by hand when cutting full releases.
16+
version/version.go:
17+
./version/version.sh > $@
18+
19+
.PHONY: setup
20+
setup: ## install dependencies
21+
@echo "🐳 $@"
22+
# TODO(stevvooe): Install these from the vendor directory
23+
@go get -u github.com/golang/lint/golint
24+
#@go get -u github.com/kisielk/errcheck
25+
@go get -u github.com/gordonklaus/ineffassign
26+
@go get -u github.com/client9/misspell/cmd/misspell
27+
@go get -u github.com/lk4d4/vndr
28+
@go get -u github.com/stevvooe/protobuild
29+
30+
.PHONY: generate
31+
generate: protos
32+
@echo "🐳 $@"
33+
@PATH=${ROOTDIR}/bin:${PATH} go generate -x ${PACKAGES}
34+
35+
.PHONY: protos
36+
protos: bin/protoc-gen-gogoswarm ## generate protobuf
37+
@echo "🐳 $@"
38+
@PATH=${ROOTDIR}/bin:${PATH} protobuild ${PACKAGES}
39+
40+
.PHONY: checkprotos
41+
checkprotos: generate ## check if protobufs needs to be generated again
42+
@echo "🐳 $@"
43+
@test -z "$$(git status --short | grep ".pb.go" | tee /dev/stderr)" || \
44+
((git diff | cat) && \
45+
(echo "👹 please run 'make generate' when making changes to proto files" && false))
46+
47+
# Depends on binaries because vet will silently fail if it can't load compiled
48+
# imports
49+
.PHONY: vet
50+
vet: binaries ## run go vet
51+
@echo "🐳 $@"
52+
@test -z "$$(go vet ${PACKAGES} 2>&1 | grep -v 'constant [0-9]* not a string in call to Errorf' | egrep -v '(timestamp_test.go|duration_test.go|exit status 1)' | tee /dev/stderr)"
53+
54+
.PHONY: misspell
55+
misspell:
56+
@echo "🐳 $@"
57+
@test -z "$$(find . -type f | grep -v vendor/ | grep -v bin/ | grep -v .git/ | grep -v MAINTAINERS | xargs misspell | tee /dev/stderr)"
58+
59+
.PHONY: fmt
60+
fmt: ## run go fmt
61+
@echo "🐳 $@"
62+
@test -z "$$(gofmt -s -l . | grep -v vendor/ | grep -v ".pb.go$$" | tee /dev/stderr)" || \
63+
(echo "👹 please format Go code with 'gofmt -s -w'" && false)
64+
@test -z "$$(find . -path ./vendor -prune -o ! -name timestamp.proto ! -name duration.proto -name '*.proto' -type f -exec grep -Hn -e "^ " {} \; | tee /dev/stderr)" || \
65+
(echo "👹 please indent proto files with tabs only" && false)
66+
@test -z "$$(find . -path ./vendor -prune -o -name '*.proto' -type f -exec grep -Hn "Meta meta = " {} \; | grep -v '(gogoproto.nullable) = false' | tee /dev/stderr)" || \
67+
(echo "👹 meta fields in proto files must have option (gogoproto.nullable) = false" && false)
68+
69+
.PHONY: lint
70+
lint: ## run go lint
71+
@echo "🐳 $@"
72+
@test -z "$$(golint ./... | grep -v vendor/ | grep -v ".pb.go:" | tee /dev/stderr)"
73+
74+
.PHONY: ineffassign
75+
ineffassign: ## run ineffassign
76+
@echo "🐳 $@"
77+
@test -z "$$(ineffassign . | grep -v vendor/ | grep -v ".pb.go:" | tee /dev/stderr)"
78+
79+
#errcheck: ## run go errcheck
80+
# @echo "🐳 $@"
81+
# @test -z "$$(errcheck ./... | grep -v vendor/ | grep -v ".pb.go:" | tee /dev/stderr)"
82+
83+
.PHONY: build
84+
build: ## build the go packages
85+
@echo "🐳 $@"
86+
@go build -i -tags "${DOCKER_BUILDTAGS}" -v ${GO_LDFLAGS} ${GO_GCFLAGS} ${PACKAGES}
87+
88+
.PHONY: test
89+
test: ## run tests, except integration tests
90+
@echo "🐳 $@"
91+
@go test -parallel 8 ${RACE} -tags "${DOCKER_BUILDTAGS}" $(filter-out ${INTEGRATION_PACKAGE},${PACKAGES})
92+
93+
.PHONY: integration-tests
94+
integration-tests: ## run integration tests
95+
@echo "🐳 $@"
96+
@go test -parallel 8 ${RACE} -tags "${DOCKER_BUILDTAGS}" ${INTEGRATION_PACKAGE}
97+
98+
# Build a binary from a cmd.
99+
bin/%: cmd/% .FORCE
100+
@test $$(go list) = "${PROJECT_ROOT}" || \
101+
(echo "👹 Please correctly set up your Go build environment. This project must be located at <GOPATH>/src/${PROJECT_ROOT}" && false)
102+
@echo "🐳 $@"
103+
@go build -i -tags "${DOCKER_BUILDTAGS}" -o $@ ${GO_LDFLAGS} ${GO_GCFLAGS} ./$<
104+
105+
.PHONY: .FORCE
106+
.FORCE:
107+
108+
.PHONY: binaries
109+
binaries: $(BINARIES) ## build binaries
110+
@echo "🐳 $@"
111+
112+
.PHONY: clean
113+
clean: ## clean up binaries
114+
@echo "🐳 $@"
115+
@rm -f $(BINARIES)
116+
117+
.PHONY: install
118+
install: $(BINARIES) ## install binaries
119+
@echo "🐳 $@"
120+
@mkdir -p $(DESTDIR)/bin
121+
@install $(BINARIES) $(DESTDIR)/bin
122+
123+
.PHONY: uninstall
124+
uninstall:
125+
@echo "🐳 $@"
126+
@rm -f $(addprefix $(DESTDIR)/bin/,$(notdir $(BINARIES)))
127+
128+
.PHONY: coverage
129+
coverage: ## generate coverprofiles from the unit tests
130+
@echo "🐳 $@"
131+
@( for pkg in $(filter-out ${INTEGRATION_PACKAGE},${PACKAGES}); do \
132+
go test -i ${RACE} -tags "${DOCKER_BUILDTAGS}" -test.short -coverprofile="../../../$$pkg/coverage.txt" -covermode=atomic $$pkg || exit; \
133+
go test ${RACE} -tags "${DOCKER_BUILDTAGS}" -test.short -coverprofile="../../../$$pkg/coverage.txt" -covermode=atomic $$pkg || exit; \
134+
done )
135+
136+
.PHONY: coverage-integration
137+
coverage-integration: ## generate coverprofiles from the integration tests
138+
@echo "🐳 $@"
139+
go test ${RACE} -tags "${DOCKER_BUILDTAGS}" -test.short -coverprofile="../../../${INTEGRATION_PACKAGE}/coverage.txt" -covermode=atomic ${INTEGRATION_PACKAGE}
140+
141+
.PHONY: help
142+
help: ## this help
143+
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) | sort
144+
145+
.PHONY: dep-validate
146+
dep-validate:
147+
@echo "+ $@"
148+
$(if $(VNDR), , \
149+
$(error Please install vndr: go get github.com/lk4d4/vndr))
150+
@rm -Rf .vendor.bak
151+
@mv vendor .vendor.bak
152+
@$(VNDR)
153+
@test -z "$$(diff -r vendor .vendor.bak 2>&1 | tee /dev/stderr)" || \
154+
(echo >&2 "+ inconsistent dependencies! what you have in vendor.conf does not match with what you have in vendor" && false)
155+
@rm -Rf vendor
156+
@mv .vendor.bak vendor

‎docker-sync.yml

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
version: "2"
2+
3+
options:
4+
verbose: true
5+
syncs:
6+
# should stay the same as the volume name used in `containerized.mk`'s `run` target
7+
swarmkit-sync:
8+
src: '.'
9+
sync_excludes: ['_obj', '_test', 'bin']

0 commit comments

Comments
 (0)
Please sign in to comment.