Skip to content

Support copying namespace-scoped secrets for backup PVC provisioning#9920

Draft
shubham-pampattiwar wants to merge 7 commits into
velero-io:mainfrom
shubham-pampattiwar:backup-pvc-secret-copy
Draft

Support copying namespace-scoped secrets for backup PVC provisioning#9920
shubham-pampattiwar wants to merge 7 commits into
velero-io:mainfrom
shubham-pampattiwar:backup-pvc-secret-copy

Conversation

@shubham-pampattiwar

@shubham-pampattiwar shubham-pampattiwar commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add support for copying namespace-scoped secrets and configmaps from the source PVC namespace to the Velero namespace during datamover backup PVC creation. This enables backup of CSI volumes that use encrypted StorageClasses requiring namespace-scoped secrets and configmaps (e.g., ODF/ceph-csi with Vault KMS).

New secretNames and configMapNames fields in backupPVCConfig allow users to specify which resources should be copied:

{
  "backupPVC": {
    "ocs-storagecluster-ceph-rbd-encrypted": {
      "secretNames": ["ceph-csi-kms-token"],
      "configMapNames": ["ceph-csi-kms-config"]
    }
  }
}

The copy happens in the DataUpload controller after acceptDataUpload(), so only the accepting node handles it:

  1. Copy: Secrets and configmaps are copied from the source namespace to the Velero namespace with a tracking label (velero.io/backup-pvc-secret)
  2. Collision handling: If a resource with the same name but different data already exists (from another namespace's DataUpload), the DataUpload fails with a clear error message
  3. Same-namespace: If the existing resource has identical data, it's a no-op (concurrent backups from the same namespace work in parallel)
  4. Cleanup: Secrets and configmaps are deleted in CleanUp() using label-based lookup after the DataUpload completes

Does your change fix a particular issue?

Fixes #9879

Please indicate you've done the following:

@netlify

netlify Bot commented Jun 15, 2026

Copy link
Copy Markdown

👷 Deploy request for velero pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit ef81f09

shubham-pampattiwar added a commit to shubham-pampattiwar/velero that referenced this pull request Jun 15, 2026
Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
Add a SecretNames field to the BackupPVC type to allow users to specify
secrets that need to be copied from the source PVC namespace to the
Velero namespace before creating the backup PVC. This is needed for CSI
drivers that require namespace-scoped secrets for volume provisioning,
such as encrypted volumes with KMS.

Fixes velero-io#9879

Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
Add CopySecret, DeleteSecretIfAny, and DeleteSecretsWithLabel utilities
for copying namespace-scoped secrets to the Velero namespace during
datamover backup PVC creation.

CopySecret handles three cases:
- Secret does not exist in target: copies it with a tracking label
- Secret exists with same data: no-op (same source namespace)
- Secret exists with different data: returns ErrSecretCollision so the
  caller can requeue

Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
Copy configured secrets from the source namespace to the Velero
namespace in the New phase of the DataUpload reconcile loop, before
calling Expose(). This is done in the controller rather than the
exposer because Expose() errors are non-retryable (marked as permanent
failure), while the controller can requeue on collision.

On secret collision (same name, different data from another
DataUpload), the controller requeues with a 5s delay, matching the
existing pattern used for VGDP constraint checking.

Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
Add label-based secret cleanup in CleanUp() to delete any secrets
that were copied to the Velero namespace for backup PVC provisioning.
Uses the velero.io/backup-pvc-secret label to find secrets associated
with the DataUpload being cleaned up.

Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
@codecov

codecov Bot commented Jun 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 39.00000% with 61 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/util/kube/secrets.go 40.22% 48 Missing and 4 partials ⚠️
pkg/controller/data_upload_controller.go 10.00% 8 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

- Fix import ordering in test file (gofmt)
- Add nolint:gosec for BackupPVCSecretLabel constant (not a credential)
- Use assert.Error instead of assert.True(err != nil) (testifylint)

Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
// Copy secrets required for backup PVC provisioning (e.g., encrypted volumes with KMS).
// This must happen before Expose() since Expose() errors are non-retryable.
// On collision (same secret name, different data from another DataUpload), requeue.
if du.Spec.CSISnapshot != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will bring a contest among all the cluster nodes, because the DU is not taken by any nodes yet.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lyndon-Li Good catch. I'll move the secret copy after acceptDataUpload() so only the accepting node handles it. Does that work for you?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once a DU is accepted, there is no way to return it back and retry.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lyndon-Li Since the DU can't be retried after accept, for the collision case (same secret name, different data from concurrent cross-tenant backups) the DataUpload would fail with a clear error message and the user can retry.

The collision is a narrow edge case -- it requires concurrent encrypted backups from different namespaces using the same secret name with different values. For the common cases (single tenant, same-namespace concurrent, sequential backups), there's no collision.

If we want automatic retry for the collision case, we'd need to handle it in the Accepted phase. Would you prefer we keep it simple with a fail + retry for now, or should we add the Accepted phase handling?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if you have a different approach in mind for handling this, I'm open to suggestions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The collision is a narrow edge case -- it requires concurrent encrypted backups from different namespaces using the same secret name with different values. For the common cases (single tenant, same-namespace concurrent, sequential backups), there's no collision.

For sure, I prefer a simpler approach AS LONG AS it could meet the majority cases.
However, for this judgement, I am not fully convinced --- multiple tenant is a common case, concurrent backup of multiple volumes among multiple nodes are very common. The only thing I am not sure is whether it is a common case that different tenants/namespaces would use different secrets.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lyndon-Li This could be a documented exception. Secrets for encrypted volumes should be unique cluster-wide to avoid possible failures with concurrent backups.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lyndon-Li @sseago Agreed with Scott's suggestion. We can document that for concurrent multi-tenant encrypted backups, secret names should be unique per namespace. On the ceph-csi side, this is already supported via the tenantTokenName configuration in the KMS config.

So the approach would be:

  • Secret copy after accept, fail on collision with a clear error message
  • Document that users with concurrent cross-tenant encrypted backups should configure unique secret names per namespace

Does that work for you?

- Add ConfigMapNames field to BackupPVC config for copying tenant
  configmaps (e.g., ceph-csi-kms-config with Vault connection overrides)
- Add CopyConfigMap, DeleteConfigMapIfAny, DeleteConfigMapsWithLabel
  utilities mirroring the secret copy functions
- Move secret/configmap copy after acceptDataUpload() so only the
  accepting node handles it, avoiding multi-node contest
- Clean up copied configmaps in CleanUp() alongside secrets

Signed-off-by: Shubham Pampattiwar <spampatt@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Datamover backup fails for CSI volumes that require namespace-scoped secrets (e.g., encrypted volumes)

3 participants