configure warnings as errors in the test suite

[graingert]

Feature or enhancement

Proposal:

It's very easy (especially when deprecating something) to introduce a lot of noisy test output with new warnings, eg: #127949 (comment)

I think it would be useful to either configure warnings as errors in the test suite (and then capture unraisable/threadexcept errors using a mechanism like pytest has) or tee any warnings produced, and fail CI if any have been produced that were not captured with catch_warnings or assertWarns etc like https://pypi.org/project/pytest-max-warnings/

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/its-too-easy-to-introduce-noise-in-cpythons-test-suite-output-resourcewarnings-and-deprecationwarnings/75234

Linked PRs

• gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings #128726
• gh-128770: fix ResourceWarning in test_pyrepl #128906
• [3.13] gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings (GH-128726) #128935
• Revert "gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings (#128726)" #128936
• gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings, and internal test warnings that are now logged #128973
• [3.13] gh-128770: fix ResourceWarning in test_pyrepl (GH-128906) #129179
• [3.13] gh-128770: raise warnings as errors in test suite - except for test_s… #131802
• [3.12] gh-128770: raise warnings as errors in test suite - except for test_s… #131824

[sobolevn]

See #110012 test_socket raises warnings right now.

[graingert]

See #128816 test_doctest also has a warning

[graingert]

See #110012 test_socket raises warnings right now.

I'm thinking we mark these as default with catch_warnings in the test for now?

[kulikjak]

Hi, after this change (specifically #131802), I am seeing the following errors when running test_peg_generator:

======================================================================
ERROR: test_with_stmt_with_paren (test.test_peg_generator.test_c_parser.TestCParser.test_with_stmt_with_paren)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/Python-3.13.3/Lib/test/test_peg_generator/test_c_parser.py", line 113, in setUp
    cmd = support.missing_compiler_executable()
  File "/build/Python-3.13.3/Lib/test/support/__init__.py", line 1916, in missing_compiler_executable
    if spawn.find_executable(cmd[0]) is None:
       ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/usr/lib/python3.13/vendor-packages/setuptools/_distutils/spawn.py", line 87, in find_executable
    warnings.warn(
    ~~~~~~~~~~~~~^
        'Use shutil.which instead of find_executable', DeprecationWarning, stacklevel=2
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
DeprecationWarning: Use shutil.which instead of find_executable

I tried replacing spawn.find_executable with shutil.which as the warning suggests, but then I get a different distutils warning and test still fails:

======================================================================
ERROR: test_with_stmt_with_paren (test.test_peg_generator.test_c_parser.TestCParser.test_with_stmt_with_paren)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/Python-3.13.3/Lib/test/test_peg_generator/test_c_parser.py", line 395, in test_with_stmt_with_paren
    self.run_test(grammar_source, test_source)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/Python-3.13.3/Lib/test/test_peg_generator/test_c_parser.py", line 134, in run_test
    self.build_extension(grammar_source)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/build/Python-3.13.3/Lib/test/test_peg_generator/test_c_parser.py", line 131, in build_extension
    generate_parser_c_extension(grammar, Path('.'), library_dir=self.library_dir)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/Python-3.13.3/Tools/peg_generator/pegen/testutil.py", line 107, in generate_parser_c_extension
    compile_c_extension(
    ~~~~~~~~~~~~~~~~~~~^
        str(source),
        ^^^^^^^^^^^^
    ...<3 lines>...
        library_dir=library_dir,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/build/Python-3.13.3/Tools/peg_generator/pegen/build.py", line 98, in compile_c_extension
    from setuptools._distutils.dep_util import newer_group
  File "/usr/lib/python3.13/vendor-packages/setuptools/_distutils/dep_util.py", line 9, in __getattr__
    warnings.warn(
    ~~~~~~~~~~~~~^
        "dep_util is Deprecated. Use functions from setuptools instead.",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        DeprecationWarning,
        ^^^^^^^^^^^^^^^^^^^
        stacklevel=2,
        ^^^^^^^^^^^^^
    )
    ^
DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead.

This is with 3.13 only, which makes sense as that is the only one which we have setuptools for also installed.

I wonder what is different about my setup considering that nobody else reported this?