Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursion Crashes Under Windows in Python 3.12 and 3.13 #1096

Closed
mrbean-bremen opened this issue Dec 3, 2024 Discussed in #1095 · 4 comments
Closed

Recursion Crashes Under Windows in Python 3.12 and 3.13 #1096

mrbean-bremen opened this issue Dec 3, 2024 Discussed in #1095 · 4 comments
Labels

Comments

@mrbean-bremen
Copy link
Member

Discussed in #1095

Originally posted by MikeTheHammer December 3, 2024
I think this is a bug, but it might be something I'm doing wrong.

I've run the following smoke test under Linux (Ubuntu 24.04) and Windows 10 LTSC, using Pythons 3.9 - 3.13 .

def test__pyfakefs_smoke(fs):
    filename = r"C:\source_file"
    fs.create_file(filename)
    with open(filename) as source_file:
        assert True

Under Linux, this test passes on all Pythons 3.9 - 3.13. Under Windows, it passes on 3.9 - 3.11, but crashes with a RecursionError: maximum recursion depth exceeded on Python 3.12 and 3.13.

On Python 3.12, the stack trace is:

============================= test session starts =============================
platform win32 -- Python 3.12.4, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\Users\Hammer\pyfakefs_smoke
plugins: pyfakefs-5.7.2
collected 1 item

test_pyfakefs_smoke.py F                                                 [100%]

================================== FAILURES ===================================
____________________________ test__pyfakefs_smoke _____________________________

filename = 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv312\\Scripts\\pytest.exe\\__main__.py'
module_globals = None

    def updatecache(filename, module_globals=None):
        """Update a cache entry and return its list of lines.
        If something's wrong, print a message, discard the cache entry,
        and return an empty list."""
    
        if filename in cache:
            if len(cache[filename]) != 1:
                cache.pop(filename, None)
        if not filename or (filename.startswith('<') and filename.endswith('>')):
            return []
    
        fullname = filename
        try:
>           stat = os.stat(fullname)
E           FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv312\\Scripts\\pytest.exe\\__main__.py'

C:\Program Files\Python312\Lib\linecache.py:93: FileNotFoundError

That error repeats 74 times, followed by:

During handling of the above exception, another exception occurred:

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000019388945EE0>

    def test__pyfakefs_smoke(fs):
        filename = r"C:\source_file"
        fs.create_file(filename)
>       with open(filename) as source_file:

C:\Users\Hammer\pyfakefs_smoke\test_pyfakefs_smoke.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python312\Lib\traceback.py:232: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python312\Lib\traceback.py:395: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python312\Lib\traceback.py:438: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python312\Lib\traceback.py:323: in line
    self._line = linecache.getline(self.filename, self.lineno)
C:\Program Files\Python312\Lib\linecache.py:30: in getline
    lines = getlines(filename, module_globals)
C:\Program Files\Python312\Lib\linecache.py:46: in getlines
    return updatecache(filename, module_globals)
C:\Program Files\Python312\Lib\linecache.py:101: in updatecache
    data = cache[filename][0]()
<frozen zipimport>:196: in get_source
    ???
<frozen zipimport>:534: in _get_data
    ???
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_io.py:124: in open_code
    return self._io_module.open_code(path)
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python312\Lib\traceback.py:232: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python312\Lib\traceback.py:395: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python312\Lib\traceback.py:438: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python312\Lib\traceback.py:323: in line
    self._line = linecache.getline(self.filename, self.lineno)
E   RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)
=========================== short test summary info ===========================
FAILED test_pyfakefs_smoke.py::test__pyfakefs_smoke - RecursionError: maximum...
============================== 1 failed in 0.64s ==============================

The stack trace in 3.13 is similar:

============================= test session starts =============================
platform win32 -- Python 3.13.0, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\Users\Hammer\pyfakefs_smoke
plugins: pyfakefs-5.7.2
collected 1 item

test_pyfakefs_smoke.py F                                                 [100%]

================================== FAILURES ===================================
____________________________ test__pyfakefs_smoke _____________________________

filename = 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv313\\Scripts\\pytest.exe\\__main__.py'
module_globals = None

    def updatecache(filename, module_globals=None):
        """Update a cache entry and return its list of lines.
        If something's wrong, print a message, discard the cache entry,
        and return an empty list."""
    
        # These imports are not at top level because linecache is in the critical
        # path of the interpreter startup and importing os and sys take a lot of time
        # and slows down the startup sequence.
        import os
        import sys
        import tokenize
    
        if filename in cache:
            if len(cache[filename]) != 1:
                cache.pop(filename, None)
        if not filename or (filename.startswith('<') and filename.endswith('>')):
            return []
    
        fullname = filename
        try:
>           stat = os.stat(fullname)

C:\Program Files\Python313\Lib\linecache.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = ('C:\\Users\\Hammer\\pyfakefs_smoke\\venv313\\Scripts\\pytest.exe\\__main__.py',)
kwargs = {}, should_use_original = True

    @functools.wraps(f)
    def wrapped(*args, **kwargs):
        should_use_original = FakeOsModule.use_original
    
        if not should_use_original and args:
            self = args[0]
            fs: FakeFilesystem = self.filesystem
            if self.filesystem.patcher:
                skip_names = fs.patcher.skip_names
                if is_called_from_skipped_module(
                    skip_names=skip_names,
                    case_sensitive=fs.is_case_sensitive,
                ):
                    should_use_original = True
    
        if should_use_original:
            # remove the `self` argument for FakeOsModule methods
            if args and isinstance(args[0], FakeOsModule):
                args = args[1:]
>           return getattr(os, f.__name__)(*args, **kwargs)
E           FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv313\\Scripts\\pytest.exe\\__main__.py'

C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_os.py:1454: FileNotFoundError

This is repeated 60 times, followed by:

During handling of the above exception, another exception occurred:

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000024459FD9E80>

    def test__pyfakefs_smoke(fs):
        filename = r"C:\source_file"
        fs.create_file(filename)
        assert os.path.exists(filename)
>       with open(filename) as source_file:

C:\Users\Hammer\pyfakefs_smoke\test_pyfakefs_smoke.py:7: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python313\Lib\traceback.py:260: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python313\Lib\traceback.py:445: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python313\Lib\traceback.py:492: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python313\Lib\traceback.py:369: in line
    self._set_lines()
C:\Program Files\Python313\Lib\traceback.py:350: in _set_lines
    lines.append(linecache.getline(self.filename, lineno).rstrip())
C:\Program Files\Python313\Lib\linecache.py:25: in getline
    lines = getlines(filename, module_globals)
C:\Program Files\Python313\Lib\linecache.py:41: in getlines
    return updatecache(filename, module_globals)
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_filesystem_unittest.py:702: in updatecache
    return self.linecache_updatecache(filename, module_globals)
C:\Program Files\Python313\Lib\linecache.py:108: in updatecache
    data = cache[filename][0]()
C:\Program Files\Python313\Lib\linecache.py:189: in get_lines
    return get_source(name, *args, **kwargs)
<frozen zipimport>:196: in get_source
    ???
<frozen zipimport>:617: in _get_data
    ???
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_io.py:124: in open_code
    return self._io_module.open_code(path)
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python313\Lib\traceback.py:260: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python313\Lib\traceback.py:445: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python313\Lib\traceback.py:492: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python313\Lib\traceback.py:369: in line
    self._set_lines()
E   RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)
=========================== short test summary info ===========================
FAILED test_pyfakefs_smoke.py::test__pyfakefs_smoke - RecursionError: maximum...
============================== 1 failed in 1.20s ==============================

Versions:

  • pytest 8.3.4

  • pyfakefs 5.7.2

  • Linux Ubuntu 24.04 - Linux 6.8.0-49-generic x86_64:

    • Python 3.9.19
    • Python 3.10.14
    • Python 3.11.9
    • Python 3.12.4
    • Python 3.13.0
  • Windows 10 LTSC 21H2 (OS Build 19044.5131) :

    • Python 3.9.13
    • Python 3.10.11
    • Python 3.11.9
    • Python 3.12.4
    • Python 3.13.0
@mrbean-bremen
Copy link
Member Author

I have seen a somewhat similar behavior with #1086, but labeled it as a limitation due to another context.
But if that happens with such a basic test, it is certainly a bug, at least as far as I can see now.

I did not really understand the cause of the problem (it has to do with the virtual environment and the Python path, but I was not able to find a good solution to prevent this).

An easy workaround to fix this is to use python -m pytest instead of calling pytest directly. This is generally considered good practice, but the direct call should also work. I cannot promise a timely fix, as I already have been struggling with this without success, but I will see what I can do.

@MikeTheHammer
Copy link

Yes, running it as python -m pytest works for me with both 3.12 and 3.13.

@mrbean-bremen
Copy link
Member Author

mrbean-bremen commented Dec 11, 2024

@MikeTheHammer - I added a workaround that should mitigate the issue. Can you please check if the main branch works for you?

MarshalX added a commit to cycodehq/cycode-cli that referenced this issue Dec 11, 2024
@mrbean-bremen
Copy link
Member Author

I've made a new patch release with the workaround. I'm not happy with the solution and will try to understand the underlying issue better, but I'm closing this issue for now. Please re-open if the issue persists with the current version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants