Use ruff for linting and code formatting

[jkarasti]

Replace black and isort with ruff for linting and code formatting and fixes everything found by ruffs default configuration. Also add one fix to rule them all from freedomofpress/securedrop-tooling/issues/11 and a complementary check target that runs ruff check, ruff format and mypy useful for CI. Note that this "reverts" #904.

This is ready for merge as is but not complete as I'd I'd like to enable more lint rules.

Closes #254

[almet]

Thanks @jkarasti for the work on this :-)

Here is some feedback:

• To me, make check conveys less meaning than make lint, so we might maybe keep the last one.
• We are currently using isort in addition to ruff, but ruff can do this for us. I think we should do this before merging.

Thanks a lot 👍 let me know when you need another review :-)

[jkarasti]

Here is some feedback:

* To me, `make check` conveys less meaning than `make lint`, so we might maybe keep the last one.

My thinking was to have a complementary target for fix so that there would be one target used to check for linting, formatting and typing issues and one target used to fix linting and formatting issues (mypy doesen't have a way to automatically apply fixes it seems). I can switch it back if you would prefer lint instead. These could also be lint/lint-check and lint-fix depending on if we want to follow what Securedrop devs are doing or not.

* We are currently using `isort` in addition to ruff, but ruff can [do this for us](https://docs.astral.sh/ruff/rules/#output-format). I think we should do this before merging.

I've enabled the isort rules.

[almet]

Thanks for enabling the I rules for isort, and for explaining your thinking.

Rather than having detailed targets, regrouping them behind the fix and lint targets seems to cut it while being self-explanatory. What do you think?

It's what the securedrop folks are doing, and having the same naming seems the right thing to do. Here's what's inside their makefile:

.PHONY: lint
lint:
	@ruff check .
	@ruff format --check .

.PHONY: fix
fix:
	@ruff check . --fix
	@ruff format .

Also, thanks for enabling the isort rule. The linter is currently failing with:

dangerzone/args.py:109: error: Unused "type: ignore" comment  [unused-ignore]

[jkarasti]

Thanks for enabling the I rules for isort, and for explaining your thinking.

Rather than having detailed targets, regrouping them behind the fix and lint targets seems to cut it while being self-explanatory. What do you think?

It's what the securedrop folks are doing, and having the same naming seems the right thing to do. Here's what's inside their makefile:

.PHONY: lint
lint:
	@ruff check .
	@ruff format --check .

.PHONY: fix
fix:
	@ruff check . --fix
	@ruff format .

I was thinking to either have the targets be check and fix or lint-check and lint-fix since, to me lint and fix didn't sound immediately obvious together. Might be a bit too pedantic on my part. Thanks for pointing out how the Securedrop folks have their Makefile targets set up, I followed suit and consolidated the targets into just lint and fix. I also went ahead and did the same with mypy. I think having the separate mypy-host and mypy-tests targets just brings unnecessary complexity.

Also noticed that the CI gets confused and breaks when running the ruff commands and then mypy from a makefile target. It tries to run mypy directly instead of the target so MYPY_ARGS is ignored.

Also, thanks for enabling the isort rule. The linter is currently failing with:

dangerzone/args.py:109: error: Unused "type: ignore" comment  [unused-ignore]

That's a bit odd, the error is coming from mypy which this PR shouldn't touch. (Other than tweaking the makefile targets now, but the error was happening before I did those changes too.) I'm not able to reproduce the error locally either. e.g running this succeeds:

poetry run mypy --ignore-missing-imports --disallow-incomplete-defs --disallow-untyped-defs --show-error-codes --warn-unreachable --warn-unused-ignores --exclude tests/test_docs_large/*.py dangerzone/args.py 

[almet]

I've been able to reproduce the error locally by only installing the dependencies in the lint and test group. I'm not sure why mypy doesn't behave the same way, but:

poetry install --only lint,test
poetry run make lint
<snip>
dangerzone/args.py:109: error: Unused "type: ignore" comment  [unused-ignore]
Found 1 error in 1 file (checked 22 source files)

This seem to point to the fact that black was bringing a dependency (click) that was missing when it was replaced with ruff. Adding back click fixes the issue.

I took the liberty to add a commit on top of yours, when the CI is green I will merge :-)

[almet]

See jkarasti#1

[jkarasti]

I've been able to reproduce the error locally by only installing the dependencies in the lint and test group. I'm not sure why mypy doesn't behave the same way, but:

poetry install --only lint,test
poetry run make lint
<snip>
dangerzone/args.py:109: error: Unused "type: ignore" comment  [unused-ignore]
Found 1 error in 1 file (checked 22 source files)

This seem to point to the fact that black was bringing a dependency (click) that was missing when it was replaced with ruff. Adding back click fixes the issue.

Thanks. I managed to reproduce this now. Turns out click ships its own type studs (Since version 8.0.0, there's also a types-click for earlier versions.), so it's needed in the lint group to get those.

Also, while poking around the codebase with mypys --install-types switch, mypy wants to install some additional dependencies: types-colorama, and types-setuptools or types-pygments (mypy seems to changes which of the latter two it suggest, I'm not sure why yet.) Installing those removes a couple of errors so shall I open a PR to add those?

[almet]

Also, while poking around the codebase with mypys --install-types switch, mypy wants to install some additional dependencies: types-colorama, and types-setuptools or types-pygments (mypy seems to changes which of the latter two it suggest, I'm not sure why yet.) Installing those removes a couple of errors so shall I open a PR to add those?

Thanks, that would be appreciated.

The CI is red because of #1028, so I'll close your PR and reopen a new one from a branch on this repo to circumvent this, until it is resolved.