build: move to PEP 621 project

[Molkree]

fixes #1595

TODO list for this PR

• [ ] Move doc dependencies to dependency-group: let's make it simple and keep out of the scope of this PR. Simply moving dependencies to the main project.toml might be a bad idea because they conflict with Python version specified in it. Namely our doc dependency Sphinx==8.2.3 requires Python>=3.11 but cve-bin-tool itself only requires >=3.9. This results in unresolvable dependencies. In practice docs are completely separate project, we can add pyproject.toml for docs only later on.

  [dependency-groups]
  doc = ["doc-dep1", "doc-dep2"]

• update test/test_requirements.py, it expects requirements.txt to be present
• [ ] update how we generate requirements.csv as far as I understand, these are handcrafted, please correct me if I'm wrong

[terriko]

Thanks for working on this! I don't have any brain power to give it this week (this is probably the only day i'm going to do any PR review) but hopefully one of the other maintainers can step in. I've at least set the checks to run for now.

[Molkree]

I have replaced toml dependency with tomli
toml is abandonded, it hasn't seen any releases since 2020
tomli is a community recognized replacement.
Note that this is only required for Python<3.11 (3.9 and 3.10 at the time of writing, and 3.9 will be hopefully dropped next month as well).
Change is required to update one test to read project dependencies from pyproject.toml. toml is so old it can't read our (valid) pyproject.toml.

We also used toml to read cve-bin-tool config, so updating our dependency to something working will benefit there as well.

I understand that this will probably slow down approval of this PR but it's a net positive (for ~1 year until 3.10 goes out of support).
tomli is already used by numerous dev tools in our dependency tree but I don't think it will help with acceptance much because it was always optional and not included in SBOM.

[Molkree]

Hm, I'm really not sure why Linux Tests (3.9) failed while building wheel 🤔
I can build it just fine locally with Python 3.9 and it succeeded in my repository.

@terriko, can you rerun the tests? I think I'm finished with this PR

[Copilot]

Pull Request Overview

Move the project metadata and configuration to PEP 621 in pyproject.toml, remove legacy setup tooling, and update tests, docs, and CI to align with the new structure.

• Adopt PEP 621: add project metadata, dependencies, scripts, and tool configs to pyproject.toml.
• Remove legacy files (setup.py, dev-requirements.txt, bandit.conf, .coveragerc, .mypy.ini) and update CI/docs accordingly.
• Update tests to read dependencies from pyproject.toml and modernize context manager syntax.

Reviewed Changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pyproject.toml	New single source of truth for project metadata, dependencies, and tool configs.
test/test_requirements.py	Read dependencies from pyproject.toml; refactor helpers; add guards and typing.
test/test_package_list_parser.py	Grouped context managers using parentheses for readability.
test/test_output_engine.py	Grouped patches with parentheses for readability.
test/test_extractor.py	Grouped patches with parentheses for readability.
CONTRIBUTING.md	Update install instructions and Bandit usage to reflect pyproject.toml.
.pre-commit-config.yaml	Point Bandit to pyproject.toml and ensure bandit[toml] is available.
.github/workflows/*	Replace requirements.txt references with pyproject.toml and install extras via .[dev].
doc/how_to_guides/*	Update cache keys from requirements.txt to pyproject.toml.
setup.py, dev-requirements.txt, bandit.conf, .coveragerc, .mypy.ini	Removed legacy files no longer needed with PEP 621.
MANIFEST.in	Drop requirements.txt from sdist since dependencies are now in pyproject.toml.
cve_bin_tool/config.py	Switch to tomllib/tomli consistently and simplify TOML loading.

Comments suppressed due to low confidence (1)

test/test_requirements.py:118

• String identity comparison is used here; use equality instead. Replace is with == to ensure correct behavior when file is a distinct but equal string.

            if file is HTML_DEP_CSV:

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Molkree]

Rerun unfortunately didn't help.
What I see is that library build succeeds on Python 3.13 ("Long tests on Python 3.13"), on Python 3.12 ("Windows long tests") but fails on Python 3.9 ("Linux tests (3.9)"). Other versions (3.10 and 3.11) are unknown because they are cancelled after 3.9 fails.

I built locally using 3.9 because I picked it as the lowest currently supported version, worked fine. Also worked fine in my fork (log).

The only thing that comes to mind now is caching, I see it's enabled for setup-python action with cache: 'pip'. I see the cache is restored in the runs on this PR, but there's no cache in my repo/locally.
I'll need to check how and what is cached.

[Molkree]

Latest hypothesis: This is just an intermittent error with GAD data source. I noticed that all tests run on push to branch and on pull request and on the same commit in PR in my fork test for Python 3.11 failed on push but succeeded for PR (same code in both) which strongly suggests it's an issue with GAD. Rerun fixed on push check as well. Copying relevant part here:

      creating build/lib/cache/gad/gemnasium-db-master-pypi/pypi
      error: can't copy 'cache/gad/gemnasium-db-master-pypi/pypi/proxy.py': doesn't exist or not a regular file

This is related to this part of code base I guess: GAD data source. The error is quite obscure to be honest, maybe there's a better way to catch it during build.

@terriko, yet another rerun (only failed jobs) should fix it I hope.

Initial hypothesis: Cache key is taken from requirements.txt or pyproject.toml (docs). It's just a file hash. But I have changed dependencies as mentioned above: toml was replaced with tomli. Let alone the fact that requirements.txt was deleted altogether and pyproject.toml should be hashed now. I don't understand how the hash of deps can stay the same 🤔

@terriko, can you delete the 3.9 cache and then rerun the test on this PR, please? It can be done from this page. That should be done in quick succession so that some other PR/commit doesn't recreate the 3.9 cache with a non-PEP 517 build. Current cache has key setup-python-Linux-x64-24.04-Ubuntu-python-3.9.24-pip-c88d8bf62fa962a4911c7d077a32b08bf74fdacf9e5f68b9de5fbad88bbede2b (we might need to do the same for 3.10 and 3.11, or maybe not because 3.12 and 3.13 worked fine).

P.S. I also wonder if deleting cache and letting it be recreated from this PR will break it for all other PRs/commits... In this case, we can once again delete the cache or just merge this ASAP