Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: backup and restore specification #1610

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions docs/proposals/backup-restore.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Backup and Restore Commands
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a problem statement that is included within this proposal that illustrates why this feature is needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.


This document describes how the backup and restore commands will work and how the feature will be rolled out.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are we backing up and restoring?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added something in a new paragraph above this.

The proposal is to create the feature as a minimum viable product and follow that up with non breaking features to the commands.
The initial implementation will be the easiest implementation that is useful.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit confusing.


## Minium Viable Product
TerryHowe marked this conversation as resolved.
Show resolved Hide resolved

The backup command will initially just support writing to a directory and the restore command will only support reading from a directory.

### oras backup

The backup command will initially only support reading a list of files from the command line and writing to a directory:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

List of files or list or artifacts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


```bash
oras backup --output ./mirror registry.k8s.io/kube-apiserver-arm64:v1.31.0 registry.k8s.io/kube-controller-manager-arm64:v1.31.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will here be an option to suppress output?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially, there will be no output to stdout.

```

The generated directory structure is `<specified-directory>/<repository>`.
The command above puts the OCI layout for the Kubernetes API server in `mirror/kube-api-server-arm64`.
The directory structure with intermediate blobs removed:

```bash
$ find mirror
mirror
mirror/kube-apiserver-arm64
mirror/kube-apiserver-arm64
mirror/kube-apiserver-arm64/ingest
mirror/kube-apiserver-arm64/oci-layout
mirror/kube-apiserver-arm64/blobs
mirror/kube-apiserver-arm64/blobs/sha256
mirror/kube-apiserver-arm64/blobs/sha256/3f4e2c5863480125882d92060440a5250766bce764fee10acdbac18c872e4dc7
...
mirror/kube-apiserver-arm64/blobs/sha256/4f80fb2b9442dbecd41e68b598533dcaaf58f9d45cce2e03a715499aa9f6b676
mirror/kube-apiserver-arm64/index.json
mirror/kube-controller-manager-arm64
mirror/kube-controller-manager-arm64
mirror/kube-controller-manager-arm64/ingest
mirror/kube-controller-manager-arm64/oci-layout
mirror/kube-controller-manager-arm64/blobs
mirror/kube-controller-manager-arm64/blobs/sha256
mirror/kube-controller-manager-arm64/blobs/sha256/3f4e2c5863480125882d92060440a5250766bce764fee10acdbac18c872e4dc7
...
mirror/kube-controller-manager-arm64/blobs/sha256/4f80fb2b9442dbecd41e68b598533dcaaf58f9d45cce2e03a715499aa9f6b676
mirror/kube-controller-manager-arm64/index.json
$
```

The directory `mirror` must exist for the initial feature.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it support the creation of a directory if it does not exist?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking not for the initial release because later we want to support writing to a tgz file and this will help differentiate file output vs directory output. Some option could be added later.

Each image will be stored in a subdirectory which matches the repository name.

### oras restore

Initially, the restore command will only support reading a directory and writing to a registry:
TerryHowe marked this conversation as resolved.
Show resolved Hide resolved

```bash
oras restore --input ./mirror localhost:15000/my-mirror
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will there be flags to support authentication and TLS for the remote registry?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, all the normal registry flags. I added some text about that.

```

The above backup example would result in:
```bash
localhost:15000/my-mirror/kube-apiserver-arm64:v1.31.0
localhost:15000/my-mirror/kube-controller-manager-arm64:v1.31.0
```

A namespace in the registry will be optional.
The registry in the above example could be specified as `localhost:15000`.


Any directory in the input that does not contain an `index.json` shall be silently ignored.

## Compressed Tar File Support

After the command supports directory write and read, support will be added for write and read to a compressed tar file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this trying to state that future enhancements after the initial release will add support for compressed archives?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

The directory structure in the tar file would be the same as in the directory output.

```bash
oras backup --output ./mirror.tgz registry.k8s.io/kube-apiserver-arm64:v1.31.0 registry.k8s.io/kube-controller-manager-arm64:v1.31.0
```

There will be no validation on file name.
The file name does not need to end in `tgz`, but the format will be compressed tar file.
If the file exists, it will be overwritten.

```bash
oras restore --input ./mirror.tgz localhost:15000/my-mirror
```
If the specified source is a file, the format is assumed to be a compressed tar file.

## Backup file input from a file

The backup command will support an `--input filename` argument which will be a file.
TerryHowe marked this conversation as resolved.
Show resolved Hide resolved
The format of the contents of the file is a list of images names separated by newlines.
If the `--input` argument is specified, no images may be specified on the command line.

## Backup file input from standard input

The backup command will support the `--input -` argument to read a list of images from standard input.
The format of the input is a list of images names separated by newlines.

## Restore file input from standard input

The restore command will support the `--input -` argument to read a compressed tar input from standard input.

## Optimize blobs

Add the feature to create blobs that are duplicated between images as hard links in the directory output and compressed tar files.
TerryHowe marked this conversation as resolved.
Show resolved Hide resolved
Loading