Skip to content

[Experimental] Focal Upgrades via SSH

Jump to bottom
rocodes edited this page Apr 30, 2021 · 6 revisions

In-place upgrade to Ubuntu 20.04 via ssh

Disclaimer

These are not officially-recommended steps. These steps are error-prone and could lead to breakage if performed incorrectly. Please double-check all commands carefully. We strongly recommend you follow our official upgrade guide instead. Please contact us before attempting these steps to discuss your migration scenario. If you do not contact us first, we may not be able to provide you with assistance.

These steps are meant for experienced administrators only. We still recommend reinstalling from a clean version of Ubuntu 20.04 once physical access to your servers is possible again, to ensure your system is in a clean state.

Acknowledgments

Thank you to @rmol.

Prerequisites

  • Admin Workstation is on latest Tails and latest SecureDrop app code (as of 2021/04/30, Tails 4.18 and SecureDrop 1.8.1)
  • SecureDrop servers are on latest SecureDrop app code
  • SSH access to the servers is working (test with ssh app hostname and ssh mon hostname)

On the Admin Workstation

  • Back up the app server: ./securedrop-admin backup

Via SSH, on the Servers

  • SSH to each of the servers and elevate to root (sudo su) to perform the following instructions.

  • Update apt sources from Xenial to Focal:

    sed -i 's/xenial/focal/g' /etc/apt/sources.list /etc/apt/sources.list.d/apt_freedom_press.list

  • Start upgrade:

    apt update

    Then

    apt -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"  dist-upgrade

    Accept all of the default prompts and settings by typing Enter (or 'y' if asked if you want to install). Sometimes tmux renders the configuration menus poorly; press 'Enter' even so to continue through the upgrade process.

    This command will likely fail partway through with errors. If so, run

    apt-get --fix-broken install

    Again, accept all of the default prompts and settings by pressing Enter (or 'y' if asked if you want to install).

    Once this completes, run

    apt -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"  dist-upgrade

    again. You will have to cycle back and forth between this command and apt-get --fix-broken install until the upgrade completes without errors and there is nothing left to upgrade.

    On Mon, this will mean 0 upgraded, 0 newly installed, 0 to remove, and 0 not upgraded. On App, this will mean 0 upgraded, 0 newly installed, 0 to remove, and 1 not upgraded, with ntp showing as being "kept back".

  • Fix DNS and time services:

    systemctl disable systemd-resolved

    apt install systemd-timesyncd

  • Install Focal-specific packages on the Application Server:

    apt install -y --allow-downgrades ossec-agent=3.6.0+focal securedrop-app-code=1.8.1+focal \ 
securedrop-config=0.1.4+1.8.1+focal securedrop-keyring=0.1.4+1.8.1+focal  \
securedrop-ossec-agent=3.6.0+1.8.1+focal tor=0.4.5.7-1~focal+1

    The Tor restart will mean you need to disconnect and reconnect your SSH session when the terminal appears unresponsive.

  • Install Focal-specific packages on the Monitor server:

    apt install -y --allow-downgrades ossec-server=3.6.0+focal \ 
securedrop-config=0.1.4+1.8.1+focal securedrop-keyring=0.1.4+1.8.1+focal \
securedrop-ossec-server=3.6.0+1.8.1+focal tor=0.4.5.7-1~focal+1

The Tor restart will mean you need to disconnect and reconnect your SSH session when the terminal appears unresponsive.

  • Remove packages and files not used on Focal:

    rm /etc/apt/security.list /etc/cron-apt/action.d/5-security
rm -r /etc/network/if-up.d/load_iptables /etc/network/iptables
apt -y autoremove
apt -y purge cron-apt ntp ntpdate paxctl

  • Switch from ifupdown to netplan:

    apt install netplan.io
mv /etc/network/interfaces /etc/network/interfaces.orig
grep -E -v -e '\s+(network|broadcast)' /etc/network/interfaces.orig > /etc/network/interfaces
ENABLE_TEST_COMMANDS=1 netplan migrate
apt remove ifupdown

  • Reboot each server.

    reboot

In an Admin Workstation Terminal:

Run

cd ~/Persistent/securedrop
./securedrop-admin sdconfig

and ensure v2 onion services are disabled, if they were not already.

Then run

./securedrop-admin install

Once the install command completes successfully, your upgrade process is complete.

We still recommend reinstalling cleanly on Ubuntu 20.04 when it is possible to obtain physical access to your servers.

To test upgraded systems from your Admin Workstation Terminal:

git checkout develop

 ./securedrop-admin setup -t
USE_FOCAL=1 ./securedrop-admin --force verify

Our test suite will run. You should see only 1 failure: app/test_apparmor.py::test/apparmor_enforced[paramiko:/app-/usr/sbin/tcpdump]. This is expected since tcpdump is not installed.

Then, to return to a regular production setup,

rm -r ~/Persistent/securedrop/admin/.venv3

Check out the latest SecureDrop app code (as of 2021/04/30, SecureDrop 1.8.1)

git tag -v 1.8.1

Verify the tag, then

git checkout 1.8.1
./securedrop-admin setup
Clone this wiki locally