Backup strategy with > 20 containers?

[alexander-zimmermann]

Hey, sorry for the maybe dump question, but it's not clear for me and I don't see that is covered in the documentation. What's is the desired config for a "bigger" docker compose with 20 containers?

A sidecar per container or one big backup for all? I believe it is the second approach. If so, how then I can influence the ordering / timing of the backups? Is the backup in that order I provide the volume mounts?

Thanks for the help
Alex

[m90]

If you want to back up many volumes on the same Docker host you can basically go down two routes:

One docker-volume-backup container per cron schedule

As you describe, defining one sidecar per cron schedule lets you define any schedule with perfect control over timing, ordering et al. Such a setup can run any number of backup tasks in parallel if you do not make mistakes when orchestrating start/stop behavior. The downside is that it's a bit cumbersome to define all of the containers' configurations. As far as resource consumption is concerned I wouldn't be too worried as an idling container is very lightweight.

A single container with configuration provided in conf.d

As described in the docs you can also mount an arbitrary number of configurations into a single docker-volume-backup container at runtime. One cron rule will be defined per configuration file. It's possible to have the same or overlapping cron schedules in such a setup, however it comes with one caveat: inside this single container, only a single backup process can run at the same time. If you define overlapping schedules, they will race for acquiring a global lock and will then be processed one by one. No guarantees about ordering are given.

---

My rules of thumb for selecting one of the approaches would be:

• if you can spread out your backup jobs in a way that they don't overlap, prefer a single container
• if you have complex rules for starting and stopping containers and multiple configurations might affect the same containers, prefer a single container
• if you have overlapping schedules or a very high frequency of backups, prefer multiple containers

The locking mechanism is also discussed here: #80 (comment)

[m90]

You can also check out the test case for an example of how using conf.d works: https://github.com/offen/docker-volume-backup/tree/3661a4b49b75562bf0a33e4cabf24c62daea777f/test/confd

[alexander-zimmermann]

@m90 thanks for the fast reply. Two approaches are understood, but what I still don't get is how the mapping between the configuration files in conf.d and the volumes are. In other words how the backup process knows which config file is valid for which volume.

backup:
  image: offen/docker-volume-backup
  container_name: docker-volume-backup
  environment:
    ...
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - node-red_data:/backup/node-red:ro
    - mosquitto_data:/backup/mosquitto:ro
    - influx_data:/backup/influx:ro
    -  ...
    - ./docker-volume-backup/config:/etc/dockervolumebackup/conf.d

assuming we have in ./docker-volume-backup/config an app-name.env file per application / container

[m90]

In that case you can set BACKUP_SOURCES for each cron schedule. So for example if you have a volume definition like:

  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - node-red_data:/backup/node-red:ro
    - mosquitto_data:/backup/mosquitto:ro
    - influx_data:/backup/influx:ro

you then set set BACKUP_SOURCES=/backup/node-red and so on per-job.

[alexander-zimmermann]

So like this...

  # --------------------------------------------------------------------------
  # Docker Volume Backup
  # --------------------------------------------------------------------------
  backup:
    image: offen/docker-volume-backup
    container_name: docker-volume-backup
    restart: unless-stopped
    volumes:
      - ./docker-volume-backup/config:/etc/dockervolumebackup/conf.d
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - node-red_data:/backup/node-red:ro
      - mosquitto_data:/backup/mosquitto:ro
      - influx_data:/backup/influx:ro
      - grafana_data:/backup/grafana:ro
      - loki_data:/backup/loki:ro
      - syslog-ng_data:/backup/syslog-ng:ro
      - redis_data:/backup/redis:ro
      - mariadb_data:/backup/mariadb:ro
      - heimdall_data:/backup/heimdall:ro
      - portainer_data:/backup/portainer:ro
      - gollum_data:/backup/gollum:ro
    environment:
      <<: *default-environment
      AWS_ENDPOINT: storage.mydomain.com
      AWS_S3_BUCKET_NAME: docker-backup
      AWS_ACCESS_KEY_ID: $MINIO_ACCESS_KEY
      AWS_SECRET_ACCESS_KEY: $MINIO_SECRET_KEY
      NOTIFICATION_URLS: pushover://shoutrrr:$PUSHOVER_TOKEN_DOCKER_BACKUP@$PUSHOVER_USER_KEY
    labels:
      - com.centurylinklabs.watchtower.enable=true
    logging:
      <<: *default-logging
    depends_on:
      - minio

with

alexander@ubuntu:~/home-automation/docker-volume-backup/config$ cat node-red_*
BACKUP_SOURCES=/backup/node-red
BACKUP_FILENAME="daily-node-red-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="0 2 * * *"
BACKUP_PRUNING_PREFIX="daily-node-red-backup-"
BACKUP_RETENTION_DAYS="7"

BACKUP_SOURCES=/backup/node-red
BACKUP_FILENAME="monthly-node-red-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="0 6 1 * *"
BACKUP_PRUNING_PREFIX="monthly-node-red-backup-"
BACKUP_RETENTION_DAYS="365"

BACKUP_SOURCES=/backup/node-red
BACKUP_FILENAME="weekely-node-red-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="0 4 * * *"
BACKUP_PRUNING_PREFIX="weekely-node-red-backup-"
BACKUP_RETENTION_DAYS="31"

and this for each volume.

[m90]

As long as BACKUP_SOURCES matches what you want to be backed up for that particular job, it will work, yes.

[alexander-zimmermann]

Ok, cool.

I will test the whole setup tomorrow when my backup hdd arrives. Will close the ticket if everything works as expected.

@m90 thanks for the nice support 👍

Hi !
I have a question about this kind of strategy.

I have several docker-compose.yml files with differents stacks from my swarm cluster.
How can I share volumes to my single instance of offen DVB ?

Do you know what I mean ?

Thanks !

[m90]

How can I share volumes to my single instance of offen DVB ?

I think you could use explicitly named volumes which makes them available across stacks. That being said, what keeps you from using one docker-volume-backup container per stack? That'd be a bit easier to understand and maintain I would guess?

You're right. I was looking for another point of view to this architecture!
I will keep one instance per stack!

I wanted to avoid repetition across the cluster, this is why.

[alexander-zimmermann]

Solution works like a charm. Thanks for the support!