CI check to validate compatibility of requirements

[kamil-certat]

Some bots have incompatible dependency requirements. It should be possible to write a CI check that tries to resolve all required dependency together and fail if it's not possible.

[gethvi]

I have this in my makefile for this (using pip-compile from pip-tools package):

requirements-all:
	rm intelmq/bots/collectors/blueliv/REQUIREMENTS.txt
	pip-compile --quiet setup.py intelmq/bots/*/*/REQUIREMENTS.txt --output-file requirements-all.txt
	git restore intelmq/bots/collectors/blueliv/REQUIREMENTS.txt

requirements-dev:
	pip-compile --extra=dev setup.py --output-file=requirements-dev.txt

The problem with blueliv collector is that it's requirements of requests package has upper boundry requests>=2.4.0,<= 2.5.1.

[kamil-certat]

Hah, I thought exactly about using pip-compile ;)

[gethvi]

It's also useful for building docker image as it is much faster (from my experience) to compile requirements and just run pip install -r requirements.txt only once. :)

[kamil-certat]

It is, the pip needs to resolve dependencies only once - and with very specific versions. This process take some time.