Usage of Molecule and Ansible for development

We use Ansible heavily in SecureDrop project. Except the actual installation process, we use Molecule along with ansible for the other steps. All the available molecule scenarios can be found at ./molecule/ directory.

• builder-xenial
• builder-focal
• testinfra
• vagrant-packager
• fetch-tor-packages
• libvirt-staging-xenial
• libvirt-staging-focal
• qubes-staging-focal
• qubes-staging-xenial
• upgrade
• virtualbox-staging-xenial

Builder Xenial and builder Focal

These 2 scenarios build the debian packages for the SecureDrop server code and related packages. For the following, we will examine the builder-focal scenario.

Let us dig into the files in the scenario.

ansible-override-vars.yml

The file contains two variables which are used in conditions to mark that we are building for Focal. These are used inside of the actual package building roles.

aptpreferences.conf

Apt file

create.yml

This creates all the containers defined in the molecule.yml file.

destroy.yml

Destroys the containers created in the create step

Dockerfile

Dockerfile for the builder container image, this needs to pushed to the container repository

image_hash

Hash of the container image

Makefile

The make file

molecule.yml

Main file which defines all configurations, from where to find the Ansible roles, and what all steps should be takes for this scenario.

playbook.yml

After the containers are running, this runs the various roles to build the actual packages. It also does the initial package testing. Check the various roles in "Build SecureDrop application Debian package from local repository." task to find which package gets build via which container.

push.sh

To push the container image.

Building the main app-code package

Path: install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/

All steps are in the tasks/main.yml.

Building ossec-server and ossec-agent package

Path; install_files/ansible-base/roles/build-ossec-deb-pkg

• Using python requests package, it first downloads the ossec source tarball.
• We use our own systemd service (two different files) for both server and agent on Focal. On Xenial we are still using the old sysv script.
• Remember that the same role is being used to build both agent and server, configuration changes dynamically based on conditions.

- name: Copy our systemd based service file for ossec-server
  copy:
    src: ossec.service
    dest: "{{ ossec_build_dir }}/etc/systemd/system/ossec.service"
  when:
    - ansible_host.endswith("-sd-generic-ossec-server")
    - ansible_distribution_release == "focal"

- name: Copy our systemd based service file for ossec-agent
  copy:
    src: ossec-agent.service
    dest: "{{ ossec_build_dir }}/etc/systemd/system/ossec.service"
  when:
    - ansible_host.endswith("-sd-generic-ossec-agent")
    - ansible_distribution_release == "focal"

Building securedrop-ossec-server and securedrop-ossec-agent

Path: install_files/ansible-base/roles/build-generic-pkg/tasks/main.yml

- role: build-generic-pkg
  tags: securedrop-ossec-server
  package_name: securedrop-ossec-server
  when: ansible_host.endswith("-sd-generic-ossec-server2") or ansible_host == "localhost"

- role: build-generic-pkg
  tags: securedrop-ossec-agent
  package_name: securedrop-ossec-agent
  when: ansible_host.endswith("-sd-generic-ossec-agent2") or ansible_host == "localhost"

Building of other 3 generic packages

The same build-generic-pkg also builds next 3 packages, securedrop-keyring, securedrop-grsec, securedrop-config.

Copying the packages into tester container for verification

As part of the task: Give dpkg verify container access to debs we copy over the fresh debian packages into the tester container. This is in the playbook.yml file.

Actual verification of the packages

This is defined in the verifier section of the molecule.yml file. The testinfra runs ../builder-xenial/tests/ tests for the same.

Fetching latest Tor packages

This is being done via fetch-tor-packages scenario. The following things happen in this:

• 2 containers started for Xenial and Focal (molecule/fetch-tor-packages/molecule.yml)
• Then the playbook.yml sets up Tor Apt repository
• Download the Tor debs via apt-get download command
• Sync them back into host machine

Building Focal staging

This scenario assumes that you already built the packages for the Focal and they are available in ./build/focal/ directory.

As a first step, create.yml creates two vms based on the configuration from molecule.yml file.

For the converge, we should look at ./install_files/ansible-base/securedrop-staging.yml playbook.

Server provisioning

When running against a staging or production host, the following playbooks will be applied to the host (generated with ansible-playbook-grapher)