Skip to content

QA Testing

Jump to bottom
Erik Moeller edited this page Apr 20, 2022 · 7 revisions

The following instructions are for Release QA processes only and should not be used on production hardware or in production contexts.

dom0 Testing

Scenario Overview

Three scenarios are supported through the environment key in config.json:

  • prod: prod RPM server, prod APT server
  • staging: dev RPM server, dev APT server
  • dev: local salt provisioning logic, dev APT server.

Clean install testing

In order to test latest release candidate and dom0 changes, we should use the staging environment key and install as per the install docs.

Upgrade install testing

In order to test the upgrade in place scenario, the best way would be to install a workstation using the production, and then switching over the configuration to staging.

All commands to be typed in dom0:

# edit /srv/salt/sd/config.json and change `environment` key from `prod` to `staging`
# update dom0 configuration (RPM sources)
sudo qubesctl --show-output --targets dom0 state.highstate
# update sys-firewall configuration
sudo qubesctl --show-output --targets sys-firewall state.highstate
# update TemplateVMs to use test apt repo(call fpf-apt-test-repo to make minimal changes)
sudo qubesctl --show-output --max-concurrency 2 --skip-dom0 --targets sd-log-buster-template,securedrop-workstation-buster,sd-proxy-buster-template,sd-app-buster-template,sd-devices-buster-template,sd-viewer-buster-template,whonix-gw-15 state.sls fpf-apt-test-repo 
# run the GUI updater, if necessary, remove the update status:
rm ~/.securedrop_launcher/sdw-update-status

The next time the preflight updater is run, it will apply all updates, in dom0 and TemplateVMs, using apt-test.freedom.press and yum-test.securedrop.org.

Component-level Testing

Release candidate Testing

  1. Download the rc package you want to test from https://apt-test.freedom.press
  2. Install it in the appropriate TemplateVM
  3. Power down the TemplateVM. Wait for this operation to finish.
  4. Reboot the AppVM
  5. Now you can start running tests against the component rc

Tip: If you are unsure about the available or installed packages, you can run the command apt-cache policy <package> in a TemplateVM or AppVM, e.g., apt-cache policy securedrop-client. It will tell you the exact version of the package that is available, as well as the package that will be installed next time the VM is updated.

Preflight Testing

The goal of preflight testing is ensure that the release has been properly deployed, and is ready for final deployment to production instances. For preflight testing, we host the new packages at https://apt-qa.freedom.press/, rather than https://apt.freedom.press/. The exact same packages, Release file, and corresponding detached signature, generated with the prod key, are used. We deploy those artifacts to apt-qa to provide final verification that the release will function well for real-world prod instances.

The following is using securedrop-client as an example.

  1. Update config.json to prod if it isn't already
  2. Run sdw-admin --apply
  3. Edit the apt sources list in sd-small-buster-template to point to apt-qa
  4. Run sudo apt update && sudo apt install securedrop-client
  5. Restart the template and sd-app if it's running
  6. Double click on the securedrop-client Desktop icon
  7. Confirm the correct version the the client appears on the login window
  8. Login and make sure your changes work

Post-release Testing

  1. If not already on prod, update config.json to prod and provision VMs by running sdw-admin --apply
  2. Force a run of the updater in dom0: /opt/securedrop/launcher/sdw-launcher.py --skip-delta 0
Clone this wiki locally