Cache and remote structure in the next major version

[iesahin]

This is a rather detailed discussion on the cache and remote structure I have started in #6653. Please look for technical and UI related downsides and difficulties in implementation in this proposal.

Requirements

• Cache and remote structure should be identical as they are now. They can both serve to multiple projects. They can reside in various different file systems. They can have different URLs depending on the access method.
• Cache/remote should know to which projects they serve. For this we need DVC project GUID that are generated during dvc init and tracked in Git.
• Cache/remote should know which files it contains and their path in the DVC project.
• Cache/remote should support more than one hash algorithm.
• Cache/remote should support different hierarchy models, like deeper directory hierarchies in SSH based remotes, and flat directories in Google Drive.
• For multiple projects, shared content should not be duplicated.
• Cache/remote should support splitting large files and merging small files.
• Cache/remote should support unfinished transfers and should be able to replay them.

Preliminary design

This is preliminary and possible will undergo drastic changes after reviews.

Content directories

Content directories are those that contain the file contents as a whole, in parts or as merged in TAR files.

They are named after the hash value of content or TAR file they contain.

When the file is large, it looks like

1234567890ABCDEF123456/
   whole
   whole.meta
   part-001
   part-002
   part-003
   ...
   part-999
   parts.meta

When the file is small, the directory looks like

1234567890ABCDEF123456/
    in-archive.meta

When the file is an archive file, the directory looks like

1234567890ABCDEF123456/
    archive.tar
    archive.meta

Content directory hierarchies

These content directories can all reside in hierarchies of various depth:

content/
   12345.../
   ABCDE.../
   FED787.../
   ...

to

content
   12/
       34/
          56/
             123456.../
    AB/
      CD/
          ABCD99../
      ABEE66.../
      ABF123.../

are possible.

When a DVC command looks for a file in the contents, they can either look for the exact directory, or the 2-digit directory that may contain the searched one.

Operation Logs

An oplog is similar to Git's reflog, but it contains the operations performed by a DVC repository in a remote.

PUT data/my-data.xml ABC1234...29810
PUT model/my-model.xml 12390A...2891
DELETE model/my-model.xml ABC812039...8329
...

These logs contain GET/PUT/DELETE instructions, the path of the file in DVC remote and the hash value that shows the content directory. It can be in JSON for easier parsing, here I used a flat structure.

These logs are named per-DVC-project-GUID + per-access-id + timestamp. So, when I access to a remote from 192.168.1.1, with a project GUID of ZZ90-FF11, it will create two files in /META/logs/.

dvc-begin==ZZ90-FF11==192.168.1.1==189798739837.log that may contain some information about the transaction, like the command initialized it. This is the first file created by DVC in dvc push and similar operations. It signals the beginning of a transaction. It may also contain a plan for the transaction if its known.

dvc-end==ZZ90-FF11==192.168.1.1==189798729837.log contains all the operations successfully performed by DVC at the end of the transaction.

When a dvc-begin... file is present the '/META/logs/but notdvc-end..., it tells the transaction wasn't properly executed. DVC can try to reproduce the command or delete this dvc-begin-...` file if the operation was cancelled intentionally.

Another kind of file, named dvc-base==... can be created time to time to list all the previous operations and to list the files. A command like dvc remote fsck to merge all dvc-begin... and dvc-end... files into dvc-base= files can allow DVC to load the most recent dvc-base=... and later dvc-begin=... and dvc-end=... files to get all the files available in a remote, with their hash values and paths in the repository.

These oplogs are duplicated in all cache and remotes, given the unique filenames. If there are N users of a repository, with M remotes, it's possible to get a list of each of N user's cache status and the files in M remotes without making a network request. So if a certain file is known to be available in remote A, but not remote B, DVC can automatically ask the file from remote A.

TBC