Skip to content

feat(chart): scheduled backup & defrag CronJobs - #138

Open
Jakob3xD wants to merge 5 commits into
clastix:masterfrom
Jakob3xD:feat/backup-defrag-cronjobs
Open

feat(chart): scheduled backup & defrag CronJobs#138
Jakob3xD wants to merge 5 commits into
clastix:masterfrom
Jakob3xD:feat/backup-defrag-cronjobs

Conversation

@Jakob3xD

@Jakob3xD Jakob3xD commented Jul 9, 2026

Copy link
Copy Markdown

What

Adds two opt-in chart-managed CronJobs, closing the gap between the README roadmap (which lists Scheduled snapshots and Scheduled defragmentation as delivered) and the current reality where both ship only as scripts/:

  • backup.enabled — scheduled etcdctl snapshot save into an emptyDir, then upload to S3/MinIO via rclone. Retention is configurable: none (bucket lifecycle), count (keep N newest), or age (--min-age).
  • defrag.enabled — scheduled defragmentation of all members via etcd-defrag (--cluster).

Both default to falseno behavior change for existing installs (default render produces zero new objects). Refs #137.

Design decisions

  • rclone (MIT) instead of the MinIO client mcmc is AGPL-3.0 and MinIO's client tooling is in maintenance mode; rclone is MIT, tiny, actively released, and configured entirely via env vars. The existing backup-storage-secret contract (storage-url, storage-access-key, storage-secret-key, storage-bucket-name, storage-bucket-folder) is unchanged.
  • Defrag default rule is relative-fragmentation baseddbSizeInUse / dbSize <= 0.55 || dbQuotaUsage > 0.8. This scales with DB size, unlike the absolute > 200MiB threshold in scripts/defrag.sh which over-triggers on larger databases. Fully configurable via defrag.rule.
  • Restricted-PSA compliant, non-rootrunAsNonRoot, runAsUser/Group/fsGroup: 1000, seccompProfile: RuntimeDefault, readOnlyRootFilesystem: true, capabilities.drop: [ALL], automountServiceAccountToken: false, secret defaultMode: 0440. Verified that neither etcd-defrag nor etcdctl snapshot save needs root here (the upstream runAsUser: 0 example is for hostNetwork + host-path certs, which does not apply to secret-volume certs).
  • No RBAC — the jobs only mount certs and talk to etcd over TLS; no Kubernetes API calls.
  • Endpoints from chart helpers — respects clusterDomain and replicas (no hardcoded cluster.local); pinned image tags (rclone 1.74.4, etcd-defrag v0.41.0), never :latest.
  • CronJob pods deliberately do NOT carry the etcd selector labels (app.kubernetes.io/name+instance) — otherwise the client/headless Services would route etcd client traffic to a backup/defrag pod.

Incidental fix

Moved the selfSignedCertificates/certManager mutual-exclusion guard out of _helpers.tpl into a rendered templates/validations.yaml. Helm does not execute top-level content of underscore-prefixed partials, so that guard was dead code — setting both flags true produced a broken render instead of the intended clear error. It now fires. The new backup-config validation lives in the same file.

Testing

Verified with helm lint + a helm template matrix: default (0 objects), each retention mode, cert-manager vs self-signed cert branching, custom clusterDomain, and replicas=5 endpoint scaling. helm-docs README regenerated (no CI drift).

Not yet verified on a live cluster (no cluster access from the authoring environment) — reviewers/maintainers may want to confirm:

  • a real backup run uploads a valid snapshot and etcdutl snapshot status passes;
  • readOnlyRootFilesystem: true doesn't break rclone/etcdctl (a /tmp emptyDir is already provisioned for rclone as a precaution);
  • retention.mode=count|age prunes correctly (requires bucket delete permission).

The scripts/schedule.sh and scripts/defrag.sh helpers are retained for ad-hoc use; docs/backup.md and docs/defragmentation.md now point at the chart values as the recommended path.


🤖 Implemented with Claude Code.

Jakob3xD and others added 5 commits July 9, 2026 12:34
Add opt-in backup/defrag value blocks and a rendered validations.yaml enforcing them. Also move the self-signed/cert-manager mutual-exclusion check out of _helpers.tpl (dead: Helm never executes underscore-partial top-level) into validations.yaml so it actually fires.

Co-authored-by: Claude <noreply@anthropic.com>
Opt-in scheduled etcd snapshot: etcdctl snapshot into an emptyDir, then rclone upload to S3/MinIO (retention: none/count/age). Adds shared _helpers.tpl partials (cert volumes with cert-manager branching, non-root security contexts, endpointsCSV, pinned images). Backup pods use distinguishing labels so etcd Services never route to them.

Co-authored-by: Claude <noreply@anthropic.com>
Opt-in scheduled etcd defragmentation via ahrtr/etcd-defrag against all members (--cluster), non-root and restricted-PSA compliant. Default rule is relative-fragmentation based (dbSizeInUse/dbSize <= 0.55 || dbQuotaUsage > 0.8), which scales with DB size unlike the absolute threshold in scripts/defrag.sh.

Co-authored-by: Claude <noreply@anthropic.com>
Regenerate README via helm-docs for the new backup/defrag values, add artifacthub.io/changes entries, and point docs/backup.md + docs/defragmentation.md at the chart-managed CronJobs (scripts remain for ad-hoc use).

Co-authored-by: Claude <noreply@anthropic.com>
The upload shell already treats the bucket folder as optional (${VAR:+/${VAR}}); requiring the secret key contradicted that and caused CreateContainerConfigError when the key was omitted. Final-review finding.

Co-authored-by: Claude <noreply@anthropic.com>
@prometherion
prometherion requested a review from bsctl July 10, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant