Use a script to populate parts of tox.ini

[sentrivana]

Right now, our tox.ini has to be kept up-to-date with new releases manually. This is not feasible as we have tens of test suites.

This is the first step towards fully automating the process. This PR adds a script that's capable of figuring out which releases of a package are available, are supported, and should be tested.

How it works

For each package, test the minimum supported version (defined by MIN_VERSION in integrations/__init__.py, the Python versions supported by both the SDK and the package, and the date of the release -- only packages not older than 5 years will be considered).

Additionally, if the package has majors, test the last version of each major as well as the first release in the last major.

If the package doesn't have majors, take the first and last supported release and two in between.

The future

Currently, the script is only responsible for generating the tox.ini entries for ~28 of our integration test suites.

More to follow in follow up PRs.

Ideally at some point the Tox entries for all integrations are generated by this script. We can then also get rid of the -latest test suites once the script is running and regenerating tox.ini on an automated basis (since it'll catch newest releases and add them to tox.ini on its own).

Relates #3808

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.15%. Comparing base (4ae94a5) to head (64ced2f).
Report is 1 commits behind head on master.

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3920      +/-   ##
==========================================
- Coverage   80.21%   80.15%   -0.07%     
==========================================
  Files         139      139              
  Lines       15394    15394              
  Branches     2596     2596              
==========================================
- Hits        12349    12339      -10     
- Misses       2202     2206       +4     
- Partials      843      849       +6     

Files with missing lines	Coverage Δ
sentry_sdk/integrations/__init__.py	77.01% <ø> (ø)

... and 10 files with indirect coverage changes

[szokeasaurusrex]

I started reviewing this PR – I am fully on board with the idea of automating our tox setup, but this PR is quite large and difficult to review.

Would you be open to splitting this into multiple smaller PRs? I think, for example, the changes to the Python versions in the .github directory could be in their own PR, adding the script could possibly also be its own thing, and we might even consider making PRs for each individual integration we move over to the script (that level of division might admittedly be overkill though if moving an individual integration to the script is a trivial change)

[szokeasaurusrex]

General question (not related to this PR): Have we thought about using uv pip here? In my experience, it is much faster than pip

[sentrivana]

See #3920 (comment), we can consider this, but not in this PR.

[szokeasaurusrex]

Why are we removing 3.13 here?

[sentrivana]

See #3920 (comment) (also for the other CI YAML changes)

[szokeasaurusrex]

Could you please explain why we need to delete these lines (and the lines in the other files)? It is unclear to me from the PR description.

[sentrivana]

Explanation here: #3974 (comment)

Basically, the script doesn't generate a -latest target anymore, just pinned versions. This is fine since it always makes sure to include the latest version out there. So if we run this script often enough (which is the plan), it effectively substitutes the -latest category.

[szokeasaurusrex]

Why are we removing 3.8 and 3.11?

[szokeasaurusrex]

Why do we remove versions here, but add them later in the file?

[szokeasaurusrex]

Same general idea of using uv pip instead of pip (just wondering if it is possible)

[sentrivana]

It probably is possible but would be a bit of a larger overhaul. I think @antonpirker was already looking into switching to uv and was also happy about the performance. So this is something we can definitely consider.

[szokeasaurusrex]

Should we consider formalizing these formats as TypedDicts, or by using some other structured data type?

[sentrivana]

We can do this in a follow up PR, might be nice.

[szokeasaurusrex]

What does this do?

[sentrivana]

Internally it creates a dict of args -> return value that it then uses as a cache on consequent calls. Here it helps us avoid making extra API calls. See https://docs.python.org/3/library/functools.html#functools.cache

[szokeasaurusrex]

some more feedback

[szokeasaurusrex]

Where will this show up? Is it meant to be run locally, or in a GitHub action? Just want to make sure it is not too easy to miss this message.

[sentrivana]

The goal would eventually be to run this in a GH action. The log is mostly informative, if we notice it at some point and have the bandwidth, we can act on it. Not a big deal if we miss it. If we at some point decide we care more, we can figure out ways to make it more prominent.

[szokeasaurusrex]

Why does this need to be an inner function?

[sentrivana]

It's only used as the key function for the bisect couple lines after, and nowhere else.

[szokeasaurusrex]

A comment might be helpful here, to describe what these are trying to do.

Also, maybe we should use a set for indexes instead of a list, since depending on the value of releases, this could have duplicates

[sentrivana]

Will add a comment in #3971

The indexes might be duplicates, but they're only used for building the filtered_releases set, where any duplicates will be ignored. If indexes was a huge list we could optimize but in this case it doesn't really make a difference.

[szokeasaurusrex]

Perhaps we should pin these to specific versions to avoid breaking the script in the event of a breaking change in these packages

[sentrivana]

I think this is fine to leave as is. We're only using very basic public APIs. If anything changes we can pin the version then.

[sentrivana]

Closing this in favor of #3971 & friends to avoid confusion, but will get back to you @szokeasaurusrex regarding the comments

[sentrivana]

@szokeasaurusrex Regarding the CI YAML changes, these are still generated automatically by the split_gh_tox_actions script from the tox.ini. Since the tox.ini had changed (for some integrations at least) and is now possibly testing different versions of frameworks on different Python versions, this then translates to CI YAML changes.

The Python changes, in general, are due to the script auto-determining the Python versions to use from the package's metadata on PyPI. I think, for instance, we were testing some packages on 3.13 before they were officially supporting it (= they had 3.13 in their PyPI classifiers). The script also auto-selects a handful of Python versions to actually test and this is now automated instead of done manually (and possibly inconsistently), also leading to CI YAML changes.