-
Notifications
You must be signed in to change notification settings - Fork 685
Tooling, automation and developer experience
Kunal Mehta edited this page Jan 6, 2023
·
30 revisions
Work in progress
This page serves two purposes:
- Describe abstractly what our interactive (CLI) and automated (CI) tools are for across the repositories that compose the SecureDrop project.
- This is in contrast to any given repository's readme, which describes concretely how to use these tools in that context.
- Track (a) inconsistencies to resolve across repositories and (b) new features, goals, or principles to implement.
While some of this material may eventually be reflected in the developer documentation, for now it is a (public) "living document" rather than a releasable documentation artifact.
- It should be easy to run the same thing that CI is running. Conversely, CI runs what you're running locally.
- Standard tooling (e.g. ShellCheck) should be available opportunistically: even if a project has no shell scripts yet, ShellCheck should kick in automatically if it acquires one.
- We're using Make as a command-runner, not for any real build logic.
- Make's main pro is that it's universal, everyone knows "make test", etc. and already has it installed.
- We do use make as a build tool for client localization targets, but that's not necessary.
- We should use tool-specific configuration (e.g. Black, pytest) rather than committing complex command invocations to Make targets.
- Makes it easier for new developers who already have experience using the tool to also use the tool here.
- Helps with other tooling, e.g. IDEs that know how to use the tool in the standard way (e.g. PyCharm can run
black
to autoformat every time you save a file), but won't know about our wrapper that contains our configuration arguments.
Across projects/repositories, we use CI to accomplish the following tasks in the following ways (including the following exceptions/gaps):
Repository | securedrop |
securedrop-client |
securedrop-proxy |
securedrop-export |
securedrop-log |
securedrop-workstation |
securedrop-updater |
---|---|---|---|---|---|---|---|
CI image | debian |
circleci/python:3.7-buster |
debian |
debian |
|||
Python version | |||||||
Runs ShellCheck | yes | ||||||
...on any and all shell files present, or passes if none | |||||||
Runs mypy | yes, config partially in shell wrapper | yes, config in Makefile | yes, config in Makefile | ||||
...in strict mode, including Qt | |||||||
Runs Black+isort | yes, config in Makefile | yes | yes, config in Makefile | yes black w/ config in Makefile, |
|||
Runs flake8 | yes | yes | yes | yes | |||
Runs safety | yes, complex Makefile command | yes, complex Makefile command | yes, complex Makefile command | yes, complex Makefile command | yes, complex Makefile command | ||
Extracts source strings for localization | yes | no | no | no | |||
...if babel.cfg is present, otherwise passes |
|||||||
Caches CI jobs | yes, but inefficiently | ||||||
CI jobs are grouped in workflows | |||||||
CI jobs are parameterized in matrices | |||||||
... |
Across projects/repositories, we use Makefiles and shell scripts to accomplish the following tasks in the following ways (including the following exceptions/gaps):
Repository | securedrop |
securedrop-client |
... |
---|---|---|---|
Create a Python venv | make venv |
make venv |
|
Set up Git hooks | make add-hooks |
make hooks |
|
Run in a development environment | make dev |
./run.sh |
|
Lint (etc.) | |||
Run tests | make test |
make test |
|
... |