docker-ckan

This repository provides a complete setup for deploying CKAN using Docker Compose. It is based on the official Docker images for CKAN and the official Docker Compose setup for CKAN. CKAN is an open-source data management system that makes data accessible by providing tools to streamline publishing, sharing, finding, and using data.

The deployment is composed of several Docker services, including CKAN itself, PostgreSQL for database management, Solr for search functionality, Redis for caching, and Nginx for handling SSL traffic. Additionally, the deployment includes the DataPusher service for handling resource uploads.

We use Make as main entry point and command line interface to build, run and destroy docker-ckan instances.

Quick Start

git clone https://github.com/TIBHannover/docker-ckan
cd docker-ckan
cp .env.example .env

Create a docker-compose.override.yml with the following content:

services:
  ckan:
    ports:
      - 5000:5000

Run:

make build up

Open your browser at https://localhost:8443. Login as ckan_admin with password test1234 (see .env).

Key Features

Dockerized CKAN Setup

CKAN and its dependencies (PostgreSQL, Solr, Redis, Nginx) are fully containerized using Docker Compose, making it easy to spin up and manage the entire stack.

Custom Initialization and Configuration

• The setup includes custom entrypoint scripts that handle initialization tasks, such as:

  • Setting up plugins, including the CKAN datapusher plugin.

  • Automatically generating and configuring security tokens (e.g., JWT, session secrets).

  • Configuring system settings through environment variables and initialization scripts.

• The initialization process uses Python-based scripts located in the ckan/setup/ folder to manage essential CKAN setup tasks.

Modular Service Architecture

• The repository is divided into distinct contexts for each service:

  • nginx/: Configures Nginx to act as a reverse proxy for CKAN, handling SSL termination.

  • ckan/: Builds CKAN from the ckan-base image, installs extensions, and applies patches. It also includes custom scripts in docker-entrypoint.d/ to handle plugin setup (like DataPusher).

  • postgresql/: Sets up a PostgreSQL instance with a main CKAN database and a DataStore database.

  • solr/: Provides a Solr instance to power CKAN’s search functionality.

  • redis/: Configures Redis for caching CKAN’s data.

Makefile for Easy Management

A Makefile is provided to streamline the management of Docker Compose commands. Some key targets include:

• make build: Build the containers. Required after changing Dockerfiles or files added to the containers.

• make up: Starts the CKAN stack in the background.

• make down: Stops and removes all containers, networks, and volumes (keeping data).

• make show-logs: Shows logs from all services in real-time.

• make destroy: Completely tears down the environment, removing all associated volumes (including persistent data).

• make bash: Bash into the running CKAN container.

Patches and Customizations

The ckan/ context includes support for applying patches to CKAN core and installed extensions. These patches are applied automatically during the build process, allowing for easy customization and updates.

Environment-Driven Configuration

The configuration of services is driven entirely by environment variables defined in the .env file (loaded via env_file in docker-compose.yml). This allows for flexible configuration of:

• Service ports, database credentials, and container networking.

• CKAN’s core functionality, e.g. site title/logo, secrets, SMTP, storage limits.

• Which of the built-in extensions are active, via CKAN__PLUGINS (see below).

• API tokens and callback URLs for services like DataPusher.

.env.example is the authoritative, self-documenting reference for every available environment variable: it is organized into commented sections (Host & Network, Database, CKAN Core, Secrets, Sysadmin, E-Mail, Storage, Solr, Redis, DataPusher, Harvest, Extensions, Backup), with a comment above each variable (or group of variables) explaining its purpose and, where relevant, the default used when it is left unset. Consult it directly rather than a separate list here, so this README cannot drift out of sync with the actual variables. .env itself is git-tracked and only contains dummy/development values, so it is safe to inspect and copy as a starting point (cp .env.example .env).

Settings of CKAN extensions are plain CKANEXT<extension><key> variables, for example CKANEXTSMWBASEURL. ckanext-envvars (plugin envvars) lowercases the name and turns __ into ., which gives the config key ckanext.smw.baseurl. There is no mapping table to maintain: a new variable works as soon as the extension reads that key. This only works for lowercase config keys, so extensions must not use upper-case characters in their keys. Extensions should declare their keys in a config_declaration.yaml (the recommended CKAN way). A variable that does not match a declared key is usually a typo and has no effect; the ckan container logs a WARNING for it at startup.

make ci checks this: ci-config-keys fails if an installed extension reads or declares a key with upper-case characters (intentional exceptions are listed in ckan/tools/config-key-exceptions.txt), and ci-extension-config verifies that every CKANEXT__* variable set in the CI environment arrives in CKAN as a declared config key with the given value.

Most of these variables are read at container startup by the scripts in ckan/docker-entrypoint.d/ (e.g. 00_configure_core.sh applies them to ckan.ini via ckan config-tool), so changing .env and running make build up again is normally enough — no image rebuild is required unless you changed a Dockerfile or a file that gets baked into the image (see Plugins: How CKAN Extensions Are Built Into the Image).

Plugins: How CKAN Extensions Are Built Into the Image

CKAN extensions are not installed at container startup or mounted at runtime — they are compiled into the ckan image at build time, pinned to an exact version. This keeps every environment built from the same image fully reproducible.

Installing an extension in the Dockerfile

ckan/Dockerfile installs each extension with the helper script ckan/tools/ckanext-install.sh:

RUN ckanext-install.sh TIBHannover/ckanext-cancel-dataset-creation 1.0.0 && \
    ckanext-install.sh TIBHannover/ckanext-close-for-guests 1.1.7 && \
    ...

ckanext-install.sh <github-org/repo> <tag>:

1. Installs the extension straight from GitHub at the given tag: pip install -e "git+https://github.com/<repo>.git@<tag>#egg=<repo-name>".

2. Installs that tag’s requirements.txt from the same ref.

To add a new extension, add another ckanext-install.sh line (alphabetically) with the GitHub org/repo and the exact release tag you want pinned. To upgrade one, bump its tag and rebuild (make build). If the extension has settings, document the CKANEXT__* variables in .env.example, so make ci verifies them; a new version that reads a key with upper-case characters fails ci-config-keys.

Activating an installed extension

Being installed in the image does not mean it is active — CKAN only loads plugins listed in CKAN__PLUGINS in .env:

CKAN__PLUGINS="cancel_dataset_creation dataset_metadata_automation dataset_reference datatables_view datastore datapusher email_notification envvars feature_image image_view multiuploader text_view tif_imageview user_manual"

The plugin name here is the extension’s CKAN plugin entry point (usually documented in the extension’s own README), not the pip/repo name. So adding an extension is a two-step process: install it in the Dockerfile, and add its plugin name to CKAN__PLUGINS, then rebuild.

Patches

Extensions (and CKAN core itself) can additionally be patched at build time. Patch files live under ckan/patches/<package-name>/ (e.g. ckan/patches/ckan/01_datasets_per_page.patch) and are applied in alphabetical order during the Dockerfile build — see ckan/patches/README.adoc.

Site customization (logo, theme, translations, licenses)

ckan/customization/ holds files that are baked into the image on top of the installed extensions:

• logo.png — copied over CKAN’s default logo.

• theme.css — custom theme stylesheet.

• i18n/<lang>/LC_MESSAGES/*.po — any .po file found here is compiled with pybabel into ckan.mo at build time, letting you override or add translation strings per language (see ckan/customization/i18n/en/LC_MESSAGES/ckan.po.example).

Licensing is configured at container startup instead of build time: if ckan/src/ckan/ckan/config/licenses.json exists in the image, 00_configure_core.sh points licenses_group_url at it automatically.

Folder Structure

• nginx/: Contains the Nginx Dockerfile and configuration files for serving CKAN over HTTPS.

• ckan/: Includes the CKAN Dockerfile, extension install tooling (tools/ckanext-install.sh), patches, site customization files (customization/), and custom entrypoint scripts (docker-entrypoint.d/).

• postgresql/: Contains the Docker setup for PostgreSQL, including database initialization.

• solr/: Contains the Solr Dockerfile and its entrypoint wrapper, which recreates the Solr core automatically when the CKAN Solr schema changes.

• setup/: Houses the initialization scripts that handle database setup, plugin configuration, Solr checks, and sysadmin creation.

Developing

In the following section, we show how changes can be made to the CKAN extension "ckanext-organisation-group" and how these are immediately visible in the running containers. The customisation of CSS classes in a template serves as an example. Before we can make the actual changes, we need to make some preparations:

1) Clone the extension code next to CKAN Docker (docker-ckan)

To be able to make changes independently of CKAN, it makes sense to check out the extension alongside CKAN Docker. This makes it easy to commit changes to the extension without having to change CKAN Docker.

A typical directory might look like this:

drwxrwxrwx  6 alex  502 4096 Feb 17 18:17 ckanext-organization-group
drwxr-xr-x  5 alex alex 4096 Jan 20 08:45 ckanext-system-stats
drwxr-xr-x  5 alex alex 4096 Jan 23 10:52 ckanext-tif-imageview
drwxr-xr-x  5 alex alex 4096 Jan 20 16:46 ckanext-user-manual
drwxr-xr-x  7 alex alex 4096 Feb 17 16:05 docker-ckan

2) Add a volume to docker-compose.override.yaml

Now we want to replace the code of the extension included in the container with the checked-out code. To do this, we use a so-called "volume" which is specified in docker-compose.override.yaml. The part before the colon (~/_dev/github.com/TIBHannover/ckanext-organization-group) in the example must be replaced by the absolute path in the local environment.

services:
  ckan:
    ports:
      - 5000:5000
    volumes:
      - ~/_dev/github.com/TIBHannover/ckanext-organization-group:/srv/app/src/ckanext-organization-group

3) Add a temporary startup script for the CKAN container

Add a startup script ckan/docker-entrypoint.d/99_dev.sh with the following content. This script ensures that the correct file permissions and ownership are set, and that the installation is complete:

#!/bin/bash

chgrp -R ckan-sys /srv/app/src
find /srv/app/src -type d -exec chmod 777 {} \;
pip install -e /srv/app/src/ckanext-organization-group
pip install -r /srv/app/src/ckanext-organization-group/requirements.txt

4) Build and run the containers

Run make build bash to build and run the containers including the volume and the startup script. All changes to the extension code are now "mirrored" directly in the container and are generally visible immediately.

Note	This volume-mount workflow is for local development only. It is a shortcut to skip rebuilding the image on every code change, not an alternative way to ship extensions — production images always get extensions baked in as described above, pinned to a fixed version.