[BUG] pip.installed for libvirt-python is (still/yet again) broken

[johnnybubonic]

Description
...C'mon, now.

Related:

• virt module (libvirt) broken on EL9 #65694
• [BUG] mysqlclient installation (via salt-pip) fails on 3006.6 #65980

This is broken in 3007.1 on a fresh install:

          ID: Ensure Python Modules for libvirt (Common)                                                                                                                                                                                      
    Function: pip.installed                                                                                                                                                                                                                   
        Name: libvirt-python                                                                                                                                                                                                                  
      Result: False                                                                                                                                                                                                                           
     Comment: Failed to install packages: libvirt-python. Error: Collecting libvirt-python                                                                                                                                                    
                Downloading libvirt-python-10.10.0.tar.gz (241 kB)                                                                                                                                                                            
                   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 241.1/241.1 kB 4.4 MB/s eta 0:00:00                                                                                                                                                
                Installing build dependencies: started                                                                                                                                                                                        
                Installing build dependencies: finished with status 'done'                                                                                                                                                                    
                Getting requirements to build wheel: started                                                                                                                                                                                  
                Getting requirements to build wheel: finished with status 'error'   error: subprocess-exited-with-error                                                                                                                       
                                                                                                                                                                                                                                              
                × Getting requirements to build wheel did not run successfully.                                                                                                                                                               
                │ exit code: 1                                                                                                                                                                                                                
                ╰─> [14 lines of output]                                                                                                                                                                                                      
                    Error in sitecustomize; set PYTHONVERBOSE for traceback:                                                                                                                                                                  
                    AssertionError:                                                                                                                                                                                                           
                    Fatal Python error: init_import_site: Failed to import the site module                                                                                                                                                    
                    Python runtime state: initialized                                                                                                                                                                                         
                    Traceback (most recent call last):                                                                                                                                                                                        
                      File "/opt/saltstack/salt/lib/python3.10/site.py", line 627, in <module>                                                                                                                                                
                        main()                                                                                                                                                                                                                
                      File "/opt/saltstack/salt/lib/python3.10/site.py", line 620, in main                                                                                                                                                    
                        execsitecustomize()                                                                                                                                                                                                   
                      File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 969, in wrapper                                                                                                                         
                        import sitecustomize                                                                                                                                                                                                  
                      File "/tmp/pip-build-env-9n0996zo/site/sitecustomize.py", line 22, in <module>                                                                                                                                          
                        assert not path in sys.path                                                                                                                                                                                           
                    AssertionError                                                                                                                                                                                                            
                    [end of output]                                                                                                                                                                                                           
                                                                                                                                                                                                                                              
                note: This error originates from a subprocess, and is likely not a problem with pip.                                                                                                                                          
              error: subprocess-exited-with-error                                                                                                                                                                                             
                                                                                                                                                                                                                                              
              × Getting requirements to build wheel did not run successfully.                                                                                                                                                                 
              │ exit code: 1                                                                                                                                                                                                                  
              ╰─> See above for output.                                                                                                                                                                                                       
                                                                                                                                                                                                                                              
              note: This error originates from a subprocess, and is likely not a problem with pip.                                                                                                                                            
     Started: 04:47:48.997682                                                                                                                                                                                                                 
    Duration: 5393.856 ms                                                                                                                                                                                                                     
     Changes:

The fix I found and mention here, YET AGAIN, works fine. pip.installed, however, (and salt-pip) remains broken.

This has been broken since 3006.6.

Setup
Try to do a pip.installed on a 3007.1 minion for libvirt-python (after installing all prereq packages; namely, libvirt-devel) on EL 9.5.

Please be as specific as possible and give set-up details.

• on-prem machine
• [Hyper-V] VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

See above.

Expected behavior

I expect pip.installed to install packages from pip.

Screenshots
N/A

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: almalinux 9.5 Teal Serval
        locale: utf-8
       machine: x86_64
       release: 5.14.0-503.15.1.el9_5.x86_64
        system: Linux
       version: AlmaLinux 9.5 Teal Serval

Additional context

[jblac]

Running into basically the same problem with the mysqlclient package.

Running the bento/ubuntu-24.04 image in vagrant with salt 3007.1 and with the following states:

python_mysql:
  pip.installed:
    - name: mysqlclient

and pip being installed with:

{% from "general/pip/map.jinja" import pip with context %}
install_pip_extensions:
  pkg.installed:
    - pkgs: {{  pip.dev_pkgs }}
install_pip:
  pkg.installed:
    - name: {{ pip.pip_pkg }}
install_boto3:
  pkg.installed:
    - pkgs: {{  pip.extra_pkgs }}

The map.jinja is as follows:

    {% set pip = salt['grains.filter_by']({
  'Debian': {
    'pip_pkg': 'python3-pip',
    'dev_pkgs': [
      'pkg-config',
      'python3-dev',
      'default-libmysqlclient-dev',
    ],
    'extra_pkgs': [
    ]
  }
 }, merge=salt['pillar.get']('pip:lookup')) %}

and I have no pillar data setup. With this when running vagrant up I get:

          ID: python_mysql
   Function: pip.installed
       Name: mysqlclient
     Result: False
    Comment: Failed to install packages: mysqlclient. Error: Collecting mysqlclient
               Downloading mysqlclient-2.2.6.tar.gz (91 kB)
                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.1/91.1 kB 4.7 MB/s eta 0:00:00
               Installing build dependencies: started
               Installing build dependencies: finished with status 'done'
               Getting requirements to build wheel: started
               Getting requirements to build wheel: finished with status 'error'   error: subprocess-exited-with-error
               
               × Getting requirements to build wheel did not run successfully.
               │ exit code: 1
               ╰─> [14 lines of output]
                   Error in sitecustomize; set PYTHONVERBOSE for traceback:
                   AssertionError:
                   Fatal Python error: init_import_site: Failed to import the site module
                   Python runtime state: initialized
                   Traceback (most recent call last):
                     File "/opt/saltstack/salt/lib/python3.10/site.py", line 627, in <module>
                       main()
                     File "/opt/saltstack/salt/lib/python3.10/site.py", line 620, in main
                       execsitecustomize()
                     File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 969, in wrapper
                       import sitecustomize
                     File "/tmp/pip-build-env-74bl48gb/site/sitecustomize.py", line 22, in <module>
                       assert not path in sys.path
                   AssertionError
                   [end of output]
               
               note: This error originates from a subprocess, and is likely not a problem with pip.
             error: subprocess-exited-with-error
             
             × Getting requirements to build wheel did not run successfully.
             │ exit code: 1
             ╰─> See above for output.
             
             note: This error originates from a subprocess, and is likely not a problem with pip.
    Started: 18:57:26.067938
   Duration: 6451.258 ms

Now if I SSH in and run export PYTHONVERBOSE=true && salt-pip install mysqlclient I get the following error:

     Fatal Python error: init_import_site: Failed to import the site module
      Python runtime state: initialized
      Traceback (most recent call last):
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 883, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/opt/saltstack/salt/lib/python3.10/site.py", line 627, in <module>
          main()
        File "/opt/saltstack/salt/lib/python3.10/site.py", line 620, in main
          execsitecustomize()
        File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 969, in wrapper
          import sitecustomize
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 883, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/tmp/pip-build-env-xpte2lii/site/sitecustomize.py", line 22, in <module>
          assert not path in sys.path
      AssertionError
      # destroy sitecustomize

No clue how to resolve that issue, though just as a test I modified my mysqlclient installation to the following and I get the same error as above:

python_mysql:
  pip.installed:
    - name: mysqlclient
    - env_vars:
        PYTHONVERBOSE: True

[jblac]

Alright, I may have figured this out but I'm not entirely sure and no clue how I can repair it permanently so some insight there would be helpful.

I looked at the site.py file, specifically around line 620 which looks like:

618     if not sys.flags.isolated:                                                                                                                                                                                                                                                                                           
619         enablerlcompleter()                                                                                                                                                                                                                                                                                              
620     execsitecustomize()                                                                                                                                                                                                                                                                                                  
621     if ENABLE_USER_SITE:                                                                                                                                                                                                                                                                                                 
622         execusercustomize()

Now initially I thought the call to execsitecustomize() just wasn't indented correctly, and that's kinda true but not the solution to the problem it just errors out in the execsitecustomize() function instead of where it's called. However, getting rid of line 620 completely fully allows me to run sudo salt-call --local state.apply db.mysql.pythonlib successfully.

UPDATE:

Temporarily I added a new state in my pip/init.sls:

fix_sitepy:
  file.managed:
    - name: /opt/saltstack/salt/lib/python3.10/site.py
    - source: salt://general/pip/files/site.py
    - runas: root

I copied the site.py from /opt/saltstack/salt/lib/python3.10/site.py to my /vagrant/salt/srv/general/pip/files directory which has line 620 removed from it. Then I destroyed my VM and brought it back up and it successfully installed the mysqlclient and the later states for mysql_user and mysql_grants both work properly.