Astroid 2.12 breaks sphinx autoapi (at least for Airflow)

[potiuk]

Steps to reproduce

1. Checkout apache airlfow (https://github.com/apache/airflow)
2. Install airflow with doc extras pip install ".[doc]"
3. Make sure to use latest astroid version pip install --upgrade astroid (2.12.1 should be installed)
4. export PYTHONPATH=$(pwd)
5. ./docs/build_docs.py --package-filter apache-airflow --one-pass-only --docs-only
6. docs build fails with 111 errors similar to:

------------------------------ Error  95 --------------------
  File "/usr/local/lib/python3.7/site-packages/astroid/nodes/node_ng.py", line 182, in infer
------------------------------ Error  96 --------------------
  File "/usr/local/lib/python3.7/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 1842, in _is_metaclass

7. downgrade astroid to 2.11.7: pip install astroid==2.11.7
8. ./docs/build_docs.py --package-filter apache-airflow --one-pass-only --docs-only
9. Observe the build succeeds.

NOTE. The above might fail because of libenchant could not be installed (especially on MacOS silicon)
You can reproduce it easily using our CI docker image - with all dependencies needed installed:

1. docker run -it ghcr.io/apache/airflow/main/ci/python3.7:2af19f16a4d94e749bbf6c7c4704e02aac35fc11
2. ./docs/build_docs.py --package-filter apache-airflow --one-pass-only --docs-only
3. docs build fails with 111 errors similar to:

------------------------------ Error  95 --------------------
  File "/usr/local/lib/python3.7/site-packages/astroid/nodes/node_ng.py", line 182, in infer
------------------------------ Error  96 --------------------
  File "/usr/local/lib/python3.7/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 1842, in _is_metaclass

4. downgrade astroid to 2.11.7: pip install astroid==2.11.7
5. ./docs/build_docs.py --package-filter apache-airflow --one-pass-only --docs-only
6. Observe the build succeeds.

Also downgrading to 2.12.0 leads to failures.
Example failure in CI: https://github.com/apache/airflow/runs/7291353339?check_suite_focus=true#step:9:6115

Current behavior

Version 2.12.* of astroid causes sphinx autoapi to fail
Version 2.11.7 of astroid and before works.

Expected behavior

• Version 2.12.* of astroid should work with sphinx autoapi

python -c "from astroid import __pkginfo__; print(__pkginfo__.version)"
2.12.1

[DanielNoord]

Thanks for the report! Failing action.

The offending line is something like this:

packages/astroid/interpreter/_import/spec.py", line 204, in find_module
submodule_path = sys.modules.__path__
AttributeError: module 'http.client' has no attribute '__path__'

Either we incorrectly classify http.client as a namespace package:
https://github.com/PyCQA/astroid/blob/c9425824b06adc654ca2d5f7b1471f65f74f64f6/astroid/interpreter/_import/spec.py#L203

Or it is a namespace package with a mangled __path__ attribute.

@jacobtylerwalls Does this ring a bell?

[jacobtylerwalls]

Thanks for the report. Looks like we need to add a hasattr guard. I noticed the failing job is running python 3.7? Is it reproducible on later versions?

[potiuk]

Yep. same problems on 3.10 at least.

[potiuk]

BTW. I you have a candidate to test, I am happy to test it. For now we limited astroid to < 2.12 so we are good, but we do not like to keep the upper-binding on our dependencies :).

[jacobtylerwalls]

@potiuk If you wanted to test #1714 that would be fantastic. I'm worried you might run into benjaminp/six#261, but there is a workaround listed there.

[potiuk]

On it.

[potiuk]

Yep. Confirmed it fixes the problem (though some tests are failing for you :P ) .

apache-airflow                                              : Building documentation
apache-airflow                                              : Running sphinx. The output is hidden until an error occurs.
apache-airflow                                              : Finished docs building successfully
Documentation build is successful
root@bdc147c580ac:/opt/airflow# pip freeze |  grep astroid
astroid @ git+https://github.com/jacobtylerwalls/astroid.git@3df008431b1801e0b38896530481fbe93baf0317

[jacobtylerwalls]

Nice!

(though some tests are failing for you :P )

yeah, I was hoping to get a reproducer without urllib3, but my effort to repro with just six failed...

[jacobtylerwalls]

@potiuk:
With encouragement from @DanielNoord I found a more correct implementation and pushed it to #1714, so you may be interested to try testing it again.

[potiuk]

Sorrry for the delay. Had to catch up with stuff :)

Yep. Installed astroid from tip of https://github.com/jacobtylerwalls/astroid/tree/namespace-path-crash and it also solves the problem: