Evaluate alternatives to Ubuntu LTS as server OS

[eloquence]

After the transition to Ubuntu 20.04 as the server OS (#4768), we will have bought ourselves some breathing room to carefully consider long-term alternatives.

Properties we'll want to consider will likely include:

• size of base OS
• built in support for hardened kernels or other security hardening
• support for OS immutability
• LTS cycle and cost, if any
• quality of documentation
• packaging format & quality of packages
• community health, project governance & sustainability

As part of this evaluation, we'll also need to revisit whether or not to use or support containerization or virtualization in some form on the server, e.g., to obtain the security benefits described in #2890.

We'll likely evaluate these properties from the perspective of minimizing the maintenance burden for administrators, and simplifying OS-level updates.

See previous issues:

• CoreOS - Figure out install bootstrap story for bare-metal #2544, Container-based OS - user configuration story #2552, Containerized SD server - application update story #2547, Harden CoreOS and container runtime security #2546, Fedora Atomic as base OS for SD #3492, Deploy SecureDrop with libostree #2966

See architecture discussion during Xenial migration:
https://github.com/freedomofpress/securedrop/wiki/Architecture-Meetings-2018-06-14

[zklosko]

I have a crazy idea, but it's so crazy it just might work. (Also, new to the project so take this with a grain of salt.)

For security reasons, it might be best to have an immutable OS, similar to if we provided Docker images. For even more security reasons, it looks like a good focus to keep SD on a separate, isolated system. Possibly even a disposable system...

So what if, in addition to Ubuntu dev, we look into dev for Raspbian or OSes that run in RAM instead of installing to disk (such as SmartOS)? This would require us to release an image which could be flashed onto an SD card/USB drive and then could be booted from and would run in a live environment, just like Tails. Sure, Debian has an inconsistent release schedule, but using an immutable OS might allow us to move at more of our own pace because the OS resets on reboot.

This automatically brings up the issue of losing all of your settings and data upon reboot. What if one SD card/USB stick was used for the OS + applications and the data was stored on a second SD card/USB stick? Or having a partition for persistence on the one SD card/USB stick with the remainder of the card/stick/drive for a live/immutable OS? Something small and easily destroy-able to protect the data on it. (Movie plot idea: a journalist needs to communicate something private with a whistleblower, so they whip out a Raspberry Pi Zero W and transfer the information over an ad-hoc WiFi network. The feds begin to close in. The journalist casually breaks the Pi Zero in half and tosses it into the water while they slip the microSD card into the sewer.)

Again, I'm new here (I spend most of my time working on the Libretime project), so take all this with a grain of salt. I just think with the advancements of ways to invade our privacy, we need to make advancements in ways to protect our privacy.

Sincerely,
-Zack

[ageis]

My vote is for Alpine.

We used Alpine with a grsecurity RBAC to perform the Zcash parameter generation ceremony. It could only run a Rust executable and nothing else. NCC Group was not able to even dump the memory.

[legoktm]

We're coming up on the focal EOL so we want to get this discussion started again; also in the context of what we'll need for future SecureDrop Protocol work.

I re-read just about all the previous discussion that I could find, so here are my personal, rough requirements for what we're looking for:

must have

• A base operating system/distro/stack that allows us to build SecureDrop parts without having to do things that aren't SecureDrop
• Good security support that doesn't require manual intervention from us
  • i.e. we are not responsible for manually patching security things
• Relatively long-term stable base
  • we shouldn't be constantly having to keep up with a distro that it interferes with the work we want to do
  • also means it shouldn't be experimental (up for debate where ostree/CoreOS-type things fall)
• free as in cost and open source
  • so not RHEL, SUSE, Oracle, etc. but CentOS/Rocky/Alma would probably be OK
• compatible with existing NUC hardware
• allows us to (mostly) control the full stack, from firmware/kernel up
  • this is kind of in response to Containerized SD server - application update story #2547; in the current SD architecture, we want to have full control over the OS-level things like kernel and whatnot. just shipping a docker image and leaving it up to people on how to run it isn't acceptable

nice to have:

• alignment with our other projects
  • SDW uses Qubes, Fedora and Debian. We're also still using Tails (which is Debian based).
  • notably Ubuntu is not used anywhere else in our stack
• bridge to SecureDrop Protocol-powered server
• not require manual intervention for each OS upgrade
• future upgrades are relatively streamlined
• broader hardware support

candidates

Here are the candidates that I've seen mentioned/heard, I kind of want to turn the above list into a matrix against the below:

traditional linux OS

• Ubuntu LTS (status quo)
• Debian
• Alpine
• CentOS/Rocky Linux/AlmaLinux

non-traditional / misc

My intuition is that we shouldn't be adopting anything non-traditional for stability and longevity purposes, but also find that projects like ostree do align better with the properties we want we want.

• Fedora Atomic/ostree
• Fedora CoreOS (same-ish as the previously discussed CoreOS, just now under the Fedora branding)
• OCI container
• something bare metal(??)

[huertanix]

Relatively long-term stable base

Dang, I guess that takes Hannah Montana Linux out of the running.

Bush-era jokes aside, Debian seems mostly ok?

• Base size can be relatively small
• SD kin already working with Debian templates in Qubes would be familiar with the server OS
• SD admins who don't come with a Linux background would have one less OS to learn the nuances of (granted, Debian-based Ubuntu isn't hugely different)
• Much of the package management architecture would need minimal changes, maybe
• IIRC, Debian's 5-year maintenance cycle seems similar to Ubuntu's at first glance? Not sure if there's a major advantage or disadvantage there
• Similarly, there would likely still need to be some manual intervention on dist upgrades every half decade unless there's some new feature I'm not aware of that would obviate this
• Debian's package offerings can be a little less bleeding edge than say, Arch, and that could limit certain features we want but aren't available in stable packages
• The packages are p stable, as labeled
• Docs seem ok in my experience
• Debian's likely going to outlive most other distros given that it's been around since I was playing pogs at recess
• SELinux is supported
• On immutability, I'm not aware of any Debian-based projects utilizing an immutable architecture; Was going to point to SteamOS but it seems they scrapped Debian for Arch :<

Alpine's tiny install base and makes it really compelling, but it's a relatively new distro, and I'm not familiar enough with it to really chime in, but I'd love to hear from folks who are; Seems like a cool distro.

[legoktm]

I didn't get to follow-up here after we discussed it amongst the team at our retreat earlier this year, mea culpa. We agreed that given the timeframe and upcoming SecureDrop Protocol-based server, we would continue ahead with the next Ubuntu LTS 24.04 (Noble) with an in-place upgrade and then keep our options open for when the Protocol is ready.