RFC: Data Mirroring Script

[tessamerrill]

In the inevitable case that we expand to more users around the country (and world), we'd want to utilize donated storage to have download mirrors closer to end-users.

I am proposing a data mirroring script that syncs (to the best ability) all data between storage servers.

What to consider:

• Users will have differing storage amounts
  • To mitigate this, allow users to set the max storage for their mirror
• Users may want to change where the data is stored
  • To mitigate this, allow users to set where the mirror data is stored
• How would data be synced? Is there a "master" copy, or can all users modify the data?
• Will data be encrypted, or is all data available to everyone?

Repository: https://github.com/LLEB-ME/fileserver

[doamatto]

Data stored on a per-user basis should be encrypted. This is law and I'll make that more clear in the fedlex soon (dictating how user data should be handled by all persons hosting services and all that; a GDPR-esque policy, if you will.)

Data stored by users into things like media libraries can be encrypted, but do not need to be. They are not user identifiable by any means.

[doamatto]

@cyckl Can you make a mockup for a basic interface for:

• simple file ops (seeing whats stored in the media folder, seeing what you have stored, downloading per file online)
• managing the server, if accessed by proper user (establishing region based on future policy— see Notion, setting max mirror size, other basic settings)
• managing user settings (establishing home region for data, optting-out of storage regions/servers, et al.; account settings are saved for sso.gouv.fa or similar)

[tessamerrill]

Going off that, do we want a basic API and/or website to control everything, as well as a CLI? Also, once the IdP is set up, I can implement that too.

[doamatto]

The basic API and website can held off for now, others can help implement those; the server administration stuff should still have CLI "manage.py"-esque settings. As long as those call functions, building APIs for that shouldn't be too hard.

IdP is/will be done via OIDC/OAUTH2, so you should be able to have dummy logic for either or. LDAP is also an option since we'll likely keep it for all intensive purposes, but it's best to use OAUTH scopes and use the SSO service.

[tessamerrill]

Okay cool.

I'll authenticate with SSO/OAuth/whatever in the style that Tailscale does it, if it's possible. I'll likely split the project into two, a daemon and a CLI.

Suggestions are appreciated.

[doamatto]

Pretend Tailscale doesn't exist for all intensive purposes of development. IdP, nor SSO, will ever be handled through it. At most, we'll move from GitHub to our SSO solution for authenticating with Tailscale. This is something I've considered for awhile and have had scattered talks with the folks over there about enabling.

The split is likely necessary and sensible. You could also have those settings configurable with flags to the daemon, requiring restarts and downtime, which is a fine "issue".

[tessamerrill]

I only considered a daemon for detached processes (i.e. docker run -d) but honestly we could be just fine without it. I'll make sure the program is dockerized, in case we want to use that.

[doamatto]

Dockerised is fine. Container plans are scattered at best right now; I'll formulate some concrete plans to propose sometime in the month. For ref., see #4.

[tessamerrill]

Alright, I'll start looking into this today. I'll update this issue with any findings.

[tessamerrill]

After some research, here's what I'm likely going to end up implementing:

• rsync transfer layer
• gin backend / api server
• some custom client
• each instance is both a server and client, with the primary instance being authoritative.
  • this creates a sort of "mesh"
  • with this, maybe all the servers could pass around their details?
    • example, server1.example.com joins the mesh at mesh.example.com
    • mesh.example.com then tells all other clients of server1.example.com
    • other clients could initialize rsync connections?