Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: backup and restore specification #1610

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions docs/proposals/backup-restore.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Backup and Restore Commands
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a problem statement that is included within this proposal that illustrates why this feature is needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

The backup and restore commands will add the capability to backup a list of artifacts from a registry and restore them to another registry.
The artifacts that ill be supported are any OCI compatible artifact such as container images, helm charts and other artifacts.
This feature will have many use cases, but the use case that will benefit the most is air gapped registries.
These commands will allow copying artifacts to and from air gapped registries over the sneakernet or other limited communication channel.

This document describes how the backup and restore commands will work and how the feature will be rolled out.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are we backing up and restoring?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added something in a new paragraph above this.

Copy link
Member

@FeynmanZhou FeynmanZhou Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, other reviewers may have the similar fundamental questions like @sabre1041 asked above.
I guess we may need an enhancement proposal doc template to structure the content and make sure necessary background/scenarios of each proposal are included.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proposal is to create the feature as a minimum viable product and follow that up with non breaking features to the commands.

## Minimum Viable Product

The backup command will initially just support writing to a directory and the restore command will only support reading from a directory.
As well as the flags described here, the commands will support the normal set of flags to support TLS and authentication.

### oras backup

The backup command will initially only support reading a list of atifacts from the command line and writing to a directory:

```bash
oras backup --output ./mirror registry.k8s.io/kube-apiserver-arm64:v1.31.0 registry.k8s.io/kube-controller-manager-arm64:v1.31.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will here be an option to suppress output?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially, there will be no output to stdout.

```

The generated directory structure is `<specified-directory>/<repository>`.
The command above puts the OCI layout for the Kubernetes API server in `mirror/kube-api-server-arm64`.
The directory structure with intermediate blobs removed:

```bash
$ find mirror
mirror
mirror/kube-apiserver-arm64
mirror/kube-apiserver-arm64
mirror/kube-apiserver-arm64/ingest
mirror/kube-apiserver-arm64/oci-layout
mirror/kube-apiserver-arm64/blobs
mirror/kube-apiserver-arm64/blobs/sha256
mirror/kube-apiserver-arm64/blobs/sha256/3f4e2c5863480125882d92060440a5250766bce764fee10acdbac18c872e4dc7
...
mirror/kube-apiserver-arm64/blobs/sha256/4f80fb2b9442dbecd41e68b598533dcaaf58f9d45cce2e03a715499aa9f6b676
mirror/kube-apiserver-arm64/index.json
mirror/kube-controller-manager-arm64
mirror/kube-controller-manager-arm64
mirror/kube-controller-manager-arm64/ingest
mirror/kube-controller-manager-arm64/oci-layout
mirror/kube-controller-manager-arm64/blobs
mirror/kube-controller-manager-arm64/blobs/sha256
mirror/kube-controller-manager-arm64/blobs/sha256/3f4e2c5863480125882d92060440a5250766bce764fee10acdbac18c872e4dc7
...
mirror/kube-controller-manager-arm64/blobs/sha256/4f80fb2b9442dbecd41e68b598533dcaaf58f9d45cce2e03a715499aa9f6b676
mirror/kube-controller-manager-arm64/index.json
$
```

The directory `mirror` must exist for the initial feature.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it support the creation of a directory if it does not exist?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking not for the initial release because later we want to support writing to a tgz file and this will help differentiate file output vs directory output. Some option could be added later.

Each image will be stored in a subdirectory which matches the repository name.

### oras restore

Initially, the restore command will support reading a directory and writing the content to a remote registry:

```bash
oras restore --input ./mirror localhost:15000/my-mirror
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will there be flags to support authentication and TLS for the remote registry?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, all the normal registry flags. I added some text about that.

```

The above backup example would result in:
```bash
localhost:15000/my-mirror/kube-apiserver-arm64:v1.31.0
localhost:15000/my-mirror/kube-controller-manager-arm64:v1.31.0
```

A namespace in the registry will be optional.
The registry in the above example could be specified as `localhost:15000`.


Any directory in the input that does not contain an `index.json` shall be silently ignored.

## Compressed Tar File Support

After the command supports directory write and read, support will be added for write and read to a compressed tar file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this trying to state that future enhancements after the initial release will add support for compressed archives?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

The directory structure in the tar file would be the same as in the directory output.

```bash
oras backup --output ./mirror.tgz registry.k8s.io/kube-apiserver-arm64:v1.31.0 registry.k8s.io/kube-controller-manager-arm64:v1.31.0
```

There will be no validation on file name.
The file name does not need to end in `tgz`, but the format will be compressed tar file.
If the file exists, it will be overwritten.

```bash
oras restore --input ./mirror.tgz localhost:15000/my-mirror
```
If the specified source is a file, the format is assumed to be a compressed tar file.

## Backup file input from a file

The backup command will support an `--input filename` argument which will be a file containing the remote resources to retrieve.
The format of the contents of the file is a list of images names separated by newlines.
If the `--input` argument is specified, no images may be specified on the command line.

## Backup file input from standard input

The backup command will support the `--input -` argument to read a list of images from standard input.
The format of the input is a list of images names separated by newlines.

## Restore file input from standard input

The restore command will support the `--input -` argument to read a compressed tar input from standard input.

## Optimize blobs

A further enhancement is to create blobs that are duplicated between images as hard links in the directory output and compressed tar files.
Loading