Redis backups

[roivaz]

This PR adds the redis backups to the saas-operator as a new api with a new controller. An example of the custom resource with the status:

apiVersion: saas.3scale.net/v1alpha1
kind: ShardedRedisBackup
metadata:
  finalizers:
    - saas.3scale.net
  name: backup
  namespace: default
spec:
  dbFile: /data/dump.rdb
  historyLimit: 2
  pause: false
  pollInterval: 10s
  s3Options:
    bucket: my-bucket
    credentialsSecretRef:
      name: aws-credentials
    path: backups
    region: us-east-1
    serviceEndpoint: http://minio.default.svc.cluster.local:9000
  schedule: '* * * * *'
  sentinelRef: sentinel
  sshOptions:
    port: 2222
    privateKeySecretRef:
      name: redis-backup-ssh-private-key
    sudo: true
    user: docker
  timeout: 5m0s
status:
  backups:
    - message: backup scheduled
      scheduledFor: "2023-09-20T08:33:00Z"
      shard: shard02
      state: Pending
    - message: backup scheduled
      scheduledFor: "2023-09-20T08:33:00Z"
      shard: shard01
      state: Pending
    - backupFile: s3://my-bucket/backups/redis-backup_shard02_2023-09-20T08:32:01Z.rdb.gz
      backupSize: 234
      finishedAt: "2023-09-20T08:32:12Z"
      message: backup complete
      scheduledFor: "2023-09-20T08:32:00Z"
      serverAlias: redis-shard-shard02-1
      serverID: 10.244.0.11:6379
      shard: shard02
      startedAt: "2023-09-20T08:32:01Z"
      state: Completed
    - backupFile: s3://my-bucket/backups/redis-backup_shard01_2023-09-20T08:32:01Z.rdb.gz
      backupSize: 232
      finishedAt: "2023-09-20T08:32:12Z"
      message: backup complete
      scheduledFor: "2023-09-20T08:32:00Z"
      serverAlias: redis-shard-shard01-1
      serverID: 10.244.0.8:6379
      shard: shard01
      startedAt: "2023-09-20T08:32:01Z"
      state: Completed

Features:

• Backups are triggered at the same time for all redis shards. Even though this does not ensure perfect consistency among shards, it's way better than the current scenario.
• Backups are fully executed from the operator. It only requires the redis instance to have ssh, gzip and awscli tools installed. An ssh user and private key must be provided for access to the instance
• The s3 service endpoint can be injected in the custom resource to point to a minio instance or any other s3 compatible api.
• The following metrics are published for backups: size, failure_count, success_count, duration.
• Backup status is stored in the custom resource status. Errors are reported in the logs, in the status and in the metrics, greatly improving backup observability.
• Automated e2e test that ensures backups do work.
• BGSave command is directly executed instead of being scheduled (as we were doing up until now) so this removes the need to manually configure the RO slave in each shard.
• The controller automatically looks for the RO slave in the shard by inspecting the Sentinel status.
• Backups can be paused using the pause flag in the custom resource.

Limitations:

• The backups are each run by its own goroutine. The problem with this approach is that anything that kills or terminates the saas-operator pod instantly kills all running backups. I decided not to wait for running backups to finish before the controller shutting down as this could mean waiting for a long time. I don't think this limitation has a huge impact as the saas-operator is very stable in production and if a backup is terminated ahead of time, the next one will just trigger soon enough. The way to overcome this limitation would be to run each backup in a Job, but this greatly complicates things, specially reporting the status of the backup back to the controller.

/kind feature
/priority important-soon
/assign

[roivaz]

@slopezz @raelga final commits to fix some bugs are in place and tested in staging env:

• fixed memory issue (not closing ssh connections properly)
• fixed cleanup of redis backup files in redis instances
• added some validation to the required Secrets that the backup uses
• improved the AWS client configuration (extracted functionality to util package)
• fixed a minor problem with the schedule of backups when the schedule is changed

[3scale-robot]

LGTM label has been added.

Git tree hash: 69390d9c0adb6672aa8847e235d4caddd6c565e6

[slopezz]

/approve

[3scale-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slopezz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [slopezz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[roivaz]

Related to https://github.com/3scale/platform/issues/868