Add container image and slim down the JDK used for Datomic

.gitignore

@@ -1 +1,4 @@
 result
+.lsp
+.clj-kondo
+data

CHANGELOG.md

@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [UNRELEASED]
+
+- Added Docker container image
+- Slimmed down the JDK used in the nix package
+
 ## v0.1.0 (2024-06-12)
 
 

README.md

@@ -4,13 +4,37 @@
 [![GitHub License](https://img.shields.io/github/license/ramblurr/datomic-pro-flake)](./LICENSE)
 [![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/ramblurr/datomic-pro-flake/ci.yml)](https://github.com/Ramblurr/datomic-pro-flake/actions/workflows/ci.yml)
 
-❄️ This flake exposes a `datomic-pro` nix package and several NixOS modules for running Datomic Pro on NixOS.
+️ This flake exposes:
 
-> 🐋 Looking for a container / docker image instead? Check out my other repo [ramblurr/containers](https://github.com/Ramblurr/containers/tree/main/apps/datomic-pro).
+* A `datomic-pro` nix package
+* ❄ Several NixOS modules for running Datomic Pro on NixOS
+* 🐋 A container image that you can use to run Datomic (no nix required!)
 
-This repo also contains a working NixOS module test that runs the flake inside a virtual machine in CI to verify that it's working.
+All of the above are tested automatically with a virtual machine!
 
-## Usage
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [datomic-pro-flake](#datomic-pro-flake)
+    - [Usage - NixOS Module](#usage---nixos-module)
+        - [`flake.nix`](#flakenix)
+        - [`/etc/datomic-pro/secrets.properties`](#etcdatomic-prosecretsproperties)
+        - [`configuration.nix`](#configurationnix)
+    - [🐋 Usage - Docker (OCI) Container Image](#🐋-usage---docker-oci-container-image)
+        - [Transactor Mode](#transactor-mode)
+            - [Env vars](#env-vars)
+        - [Console Mode](#console-mode)
+            - [Env vars](#env-vars-1)
+        - [Example Compose](#example-compose)
+            - [Datomic Pro with Local Storage](#datomic-pro-with-local-storage)
+            - [Datomic Pro with Postgres Storage and memcached](#datomic-pro-with-postgres-storage-and-memcached)
+        - [Discussion](#discussion)
+    - [License](#license)
+
+<!-- markdown-toc end -->
+
+
+## Usage - NixOS Module
 
 ### `flake.nix`
 
@@ -94,6 +118,210 @@ A basic dev-mode datomic that stores its state in `/var/lib/datomic-pro`:
 }
 ```
 
+## 🐋 Usage - Docker (OCI) Container Image
+
+This flake also provides a container image for Datomic Pro that can be driven entirely with environment variables or `_FILE` style secrets.
+
+If you don't want to build the container image yourself with nix, you can get the latest image with:
+
+``` shell
+docker/podman pull ghcr.io/ramblurr/datomic-pro:<LATEST_DATOMIC_VERSION>
+```
+
+The available tags you can find here: https://github.com/users/Ramblurr/packages/datomic-pro-flake/package/datomic-pro
+
+### Transactor Mode
+
+Runs a Datomic Transactor. This is the default mode when the container is run with no command.
+
+* The default port is `4334`.
+* A rw volume of `/config` is required.
+* Configure with env vars (see below) or add `/config/transactor.properties` to supply a config to the transactor.
+* A rw volume of `/data` is optional (for use in H2 mode).
+
+#### Env vars
+
+> [!IMPORTANT]
+> All env vars can be passed with `_FILE` to read the value from a file
+> (e.g, when using secrets). Example: `DATOMIC_STORAGE_ADMIN_PASSWORD` can be
+> passed as `DATOMIC_STORAGE_ADMIN_PASSWORD_FILE=/run/secrets/admin-password` and
+> the value from that file will be used as the admin password.
+
+* `DATOMIC_TRANSACTOR_PROPERTIES_PATH` - The path to the properties file for the transactor. Defaults to `/config/transactor.properties`
+
+The following environment vars configure the properties, refer to the Datomic documentation for more information:
+
+* `DATOMIC_ALT_HOST` - `alt-host`
+* `DATOMIC_DATA_DIR` - `data-dir` (default: /data)
+* `DATOMIC_ENCRYPT_CHANNEL` - `encrypt-channel`
+* `DATOMIC_HEARTBEAT_INTERVAL_MSEC` - `heartbeat-interval-msec`
+* `DATOMIC_HOST` - `host` (default: 0.0.0.0)
+* `DATOMIC_MEMCACHED` - `memcached`
+* `DATOMIC_MEMCACHED_AUTO_DISCOVERY` - `memcached-auto-discovery`
+* `DATOMIC_MEMCACHED_CONFIG_TIMEOUT_MSEC` - `memcached-config-timeout-msec`
+* `DATOMIC_MEMCACHED_PASSWORD` - `memcached-password`
+* `DATOMIC_MEMCACHED_USERNAME` - `memcached-username`
+* `DATOMIC_MEMORY_INDEX_MAX` - `memory-index-max` (default: 256m)
+* `DATOMIC_MEMORY_INDEX_THRESHOLD` - `memory-index-threshold` (default: 32m)
+* `DATOMIC_OBJECT_CACHE_MAX` - `object-cache-max` (default: 128m)
+* `DATOMIC_PID_FILE` - `pid-file`
+* `DATOMIC_HEALTHCHECK_CONCURRENCY` - `ping-concurrency`
+* `DATOMIC_HEALTHCHECK_HOST` - `ping-host`
+* `DATOMIC_HEALTHCHECK_PORT` - `ping-port`
+* `DATOMIC_PORT` - `port` (default: 4334)
+* `DATOMIC_PROTOCOL` - `protocol` (default: dev)
+* `DATOMIC_READ_CONCURRENCY` - `read-concurrency`
+* `DATOMIC_SQL_DRIVER_CLASS` - `sql-driver-class`
+* `DATOMIC_SQL_URL` - `sql-url`
+* `DATOMIC_STORAGE_ACCESS` - `storage-access` (default: remote)
+* `DATOMIC_STORAGE_ADMIN_PASSWORD` - `storage-admin-password`
+* `DATOMIC_STORAGE_DATOMIC_PASSWORD` - `storage-datomic-password`
+* `DATOMIC_VALCACHE_MAX_GB` - `valcache-max-gb`
+* `DATOMIC_VALCACHE_PATH` - `valcache-path`
+* `DATOMIC_WRITE_CONCURRENCY` - `write-concurrency`
+
+### Console Mode
+
+Runs the Datomic Console.
+
+* Run this mode by passing the `console` as the first and only argument to the container.
+* The default port is `8080`.
+
+#### Env vars
+
+* `DB_URI` - the database connection URI that console uses to connect to datomic
+* `DB_URI_FILE` - will read the connection URI from the file specified by this env var
+
+### Example Compose
+
+#### Datomic Pro with Local Storage
+
+Please note the tag below may not be up to date.
+
+``` yaml
+---
+services:
+  datomic-transactor:
+    image: ghcr.io/ramblurr/datomic-pro:1.0.7075
+    environment:
+      DATOMIC_STORAGE_ADMIN_PASSWORD: unsafe
+      DATOMIC_STORAGE_DATOMIC_PASSWORD: unsafe
+    volumes:
+      - ./data:/data
+    ports:
+      - 127.0.0.1:4334:4334
+    #user: 1000:1000 # if using rootful containers uncomment this
+
+  datomic-console:
+    image: ghcr.io/ramblurr/datomic-pro:1.0.7075
+    command: console
+    environment:
+      DB_URI: datomic:dev://datomic-transactor:4334/?password=unsafe
+    ports:
+      - 127.0.0.1:8081:8080
+    #user: 1000:1000 # if using rootful containers uncomment this
+```
+
+#### Datomic Pro with Postgres Storage and memcached
+
+This compose file is near-production ready. But you shouldn't manage the
+lifecycle of the postgres schema this way. How you do it depends on your
+environment.
+
+Please note the tag below may not be up to date.
+
+``` yaml
+---
+services:
+  datomic-memcached:
+    image: docker.io/memcached:latest
+    command: memcached -m 1024
+    ports:
+      - 127.0.0.1:11211:11211
+    restart: always
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          'bash -c ''echo "version" | (exec 3<>/dev/tcp/localhost/11211; cat >&3; timeout 0.1 cat <&3; exec 3<&-)''',
+        ]
+      interval: 5s
+      retries: 60
+
+  datomic-storage:
+    image: docker.io/library/postgres:latest
+    environment:
+      POSTGRES_PASSWORD: unsafe
+    command: postgres -c 'max_connections=1024'
+    volumes:
+      - ./data:/var/lib/postgresql/data
+    ports:
+      - 127.0.0.1:5432:5432
+    restart: always
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 5s
+      timeout: 3s
+      retries: 30
+
+  datomic-storage-migrator:
+    image: ghcr.io/ramblurr/datomic-pro:1.0.7075
+    environment:
+      PGUSER: postgres
+      PGPASSWORD: unsafe
+    volumes:
+      - "./postgres-migrations:/migrations"
+    entrypoint: /bin/sh
+    command: >
+      -c '(psql -h datomic-storage -lqt | cut -d \| -f 1 | grep -qw "datomic" || psql -h datomic-storage -f /opt/datomic-pro/bin/sql/postgres-db.sql) &&
+             (psql -h datomic-storage -d datomic -c "\dt" | grep -q "datomic_kvs" || psql -h datomic-storage -d datomic -f /opt/datomic-pro/bin/sql/postgres-table.sql) &&
+             (psql -h datomic-storage -d datomic -c "\du" | cut -d \| -f 1 | grep -qw "datomic" || psql -h datomic-storage -d datomic -f /opt/datomic-pro/bin/sql/postgres-user.sql)'
+    depends_on:
+      datomic-storage:
+        condition: service_healthy
+
+  datomic-transactor:
+    image: ghcr.io/ramblurr/datomic-pro:1.0.7075
+    environment:
+      DATOMIC_STORAGE_ADMIN_PASSWORD: unsafe
+      DATOMIC_STORAGE_DATOMIC_PASSWORD: unsafe
+      DATOMIC_PROTOCOL: sql
+      DATOMIC_SQL_URL: jdbc:postgresql://datomic-storage:5432/datomic?user=datomic&password=datomic
+      DATOMIC_HEALTHCHECK_HOST: 127.0.0.1
+      DATOMIC_HEALTHCHECK_PORT: 9999
+      DATOMIC_MEMCACHED: datomic-memcached:11211
+    ports:
+      - 127.0.0.1:4334:4334
+    #user: 1000:1000 # if using rootful containers uncomment this
+    restart: always
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          'if [[ $(curl -s -o /dev/null -w "%{http_code}" -X GET http://127.0.0.1:9999/health)  = "200" ]]; then echo 0; else echo 1; fi',
+        ]
+      interval: 10s
+      timeout: 3s
+      retries: 30
+    depends_on:
+      datomic-storage:
+        condition: service_healthy
+      datomic-memcached:
+        condition: service_healthy
+      datomic-storage-migrator:
+        condition: service_completed_successfully
+
+  datomic-console:
+    image: ghcr.io/ramblurr/datomic-pro:1.0.7075
+    command: console
+    environment:
+      DB_URI: datomic:sql://?jdbc:postgresql://datomic-storage:5432/datomic?user=datomic&password=datomic
+    ports:
+      - 127.0.0.1:8081:8080
+    #user: 1000:1000 # if using rootful containers uncomment this
+```
+
+
 ### Discussion
 
 Feel free to [open an issue](https://github.com/Ramblurr/datomic-pro-flake/issues/new) or reach out to me on the Clojurians slack ([@Ramblurr](https://clojurians.slack.com/team/U70QFSCG2)).

flake.nix

@@ -17,78 +17,50 @@
           inherit system;
           overlays = [ self.overlays."${system}" ];
         };
+
+        jdk-minimal = pkgs.jdk21.override {
+          headless = true;
+          enableJavaFX = false;
+          enableGnome2 = false;
+        };
       in
       {
-        overlays = final: prev: { datomic-pro = pkgs.callPackage ./pkgs/datomic-pro.nix { }; };
+        overlays = final: prev: {
+          jdk-minimal = jdk-minimal;
+          datomic-pro = pkgs.callPackage ./pkgs/datomic-pro.nix { };
+          datomic-pro-container = pkgs.callPackage ./pkgs/datomic-pro-container-image.nix { };
+          datomic-generate-properties = pkgs.callPackage ./pkgs/datomic-generate-properties.nix { };
+        };
         packages = {
           default = self.packages.${system}.datomic-pro;
           datomic-pro = pkgs.datomic-pro;
+          datomic-pro-container = pkgs.datomic-pro-container;
+          datomic-generate-properties = pkgs.datomic-generate-properties;
         };
         nixosModules = {
           datomic-pro = import ./nixos-modules/datomic-pro.nix;
           datomic-console = import ./nixos-modules/datomic-console.nix;
         };
         checks = {
-          # A VM test of the NixOS module.
-          vmTest =
-            with import (nixpkgs + "/nixos/lib/testing-python.nix") { inherit system; };
-
-            makeTest {
-              name = "datomic-pro module test";
-              nodes = {
-                client =
-                  { ... }:
-                  {
-                    nixpkgs.overlays = [ self.overlays."${system}" ];
-                    virtualisation.memorySize = 2048;
-                    imports = [
-                      self.nixosModules.${system}.datomic-pro
-                      self.nixosModules.${system}.datomic-console
-                    ];
-                    environment.etc."datomic-pro/do-not-do-this.properties" = {
-                      text = ''
-                        storage-admin-password=do-not-do-it-this-way-in-prod
-                        storage-datomic-password=do-not-do-it-this-way-in-prod
-                      '';
-                      mode = "0600";
-                    };
-                    services.datomic-pro = {
-                      enable = true;
-                      secretsFile = "/etc/datomic-pro/do-not-do-this.properties";
-                      settings = {
-                        enable = true;
-                        host = "localhost";
-                        port = 4334;
-                        memory-index-max = "256m";
-                        memory-index-threshold = "32m";
-                        object-cache-max = "128m";
-                        protocol = "dev";
-                        storage-access = "remote";
-                      };
-                    };
-
-                    environment.etc."datomic-console/do-not-do-this" = {
-                      text = "datomic:dev://localhost:4334/?password=do-not-do-it-this-way-in-prod";
-                      mode = "0600";
-                    };
-                    services.datomic-console = {
-                      enable = true;
-                      alias = "dev";
-                      port = 8080;
-                      dbUriFile = "/etc/datomic-console/do-not-do-this";
-                    };
-                  };
-              };
+          # A test of the NixOS module that runs in a VM
+          moduleTest = import ./tests/nixos-module.nix {
+            inherit
+              system
+              pkgs
+              nixpkgs
+              self
+              ;
+          };
 
-              testScript = ''
-                start_all()
-                machine.wait_for_unit("datomic-pro.service")
-                machine.wait_for_open_port(4334)
-                machine.wait_for_unit("datomic-console.service")
-                machine.wait_for_open_port(8080)
-                machine.succeed("curl --fail http://localhost:8080/browse")
-              '';
-            };
+          # A test of the container image that runs in a VM
+          containerImageTest = import ./tests/container-image.nix {
+            inherit
+              system
+              pkgs
+              nixpkgs
+              self
+              ;
+          };
         };
       }
     );

nixos-modules/datomic-console.nix

@@ -13,13 +13,13 @@ in
     services.datomic-console = {
       enable = lib.mkEnableOption "Datomic Pro Console";
       package = lib.mkPackageOption pkgs "datomic-pro" { };
-      javaPackage = lib.mkPackageOption pkgs "temurin-bin" { };
+      javaPackage = lib.mkPackageOption pkgs "jdk-minimal" { };
       port = lib.mkOption {
         type = lib.types.port;
         description = "The port the console will bind to";
       };
       alias = lib.mkOption {
-        type = lib.types.string;
+        type = lib.types.str;
         default = "dev";
         description = "A text-based name for a transactor that is associated with a transactor URI that does not include a database name";
       };