pip-compile with certain packages throws pip._internal.exceptions.InstallationError

[YKdvd]

When compiling a requirements.in file with certain packages listed ("mod_wsgi" and "edl" are the two examples I've found so far), pip-compile throws pip._internal.exceptions.InstallationError.

Environment Versions

1. OS Type: Windows 10, Ubuntu 16.04
2. Python version: 2.7.17 (Windows), 2.7.12 (Ubuntu)
3. pip version: 20.0.2, 19.3.1
4. pip-tools version: 4.20, 4.30, 4.40

Steps to replicate

1. Create requirements.in file with any/all of these "bad" packages: "edl", "mod_wsgi"
2. pip-compile --verbose --rebuild requirements.in

Expected result

Successful compilation to requirements.txt.
I've compiled files with "edl" and especially "mod_wsgi" in the past, but perhaps long enough ago that it was before 4.20/19.3.1, which is the oldest combo I've tried reproducing this with.
I'll see if I can find other examples of "bad" packages that trigger this, and also see if I can figure out something in their setup.py etc. that might be causing this.
...

Actual result

Errors out with pip._internal.exceptions.InstallationError when it reaches the first one of the "bad" packages above.

(piptemp) V:\repos\test\build>pip-compile --verbose --rebuild temp.in
Using indexes:
  https://pypi.org/simple

                          ROUND 1
Current constraints:
  mod_wsgi (from -r temp.in (line 5))
  timecode (from -r temp.in (line 1))

Finding the best candidates:
  found candidate mod-wsgi==4.7.0 (constraint was <any>)
  found candidate timecode==1.2.0 (constraint was <any>)

Finding secondary dependencies:
  timecode==1.2.0 not in cache, need to check index
  timecode==1.2.0           requires -
  mod-wsgi==4.7.0 not in cache, need to check index
No handlers could be found for logger "pip.subprocessor"
Traceback (most recent call last):
  File "C:\Python27\Lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\user.name\PyEnvs\piptemp\Scripts\pip-compile.exe\__main__.py", line 7, in <module>
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\piptools\scripts\compile.py", line 358, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\piptools\resolver.py", line 168, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\piptools\resolver.py", line 263, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\piptools\resolver.py", line 358, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\piptools\repositories\local.py", line 70, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\piptools\repositories\pypi.py", line 272, in get_dependencies
    download_dir, ireq, wheel_cache
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\piptools\repositories\pypi.py", line 231, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\legacy_resolve.py", line 365, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\legacy_resolve.py", line 313, in _get_abstract_dist_for
    req, self.session, self.finder, self.require_hashes
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\operations\prepare.py", line 224, in prepare_linked_requirement
    req, self.req_tracker, finder, self.build_isolation,
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\operations\prepare.py", line 49, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\distributions\source\legacy.py", line 39, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\req\req_install.py", line 563, in prepare_metadata
    self.metadata_directory = metadata_generator(self)
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\operations\generate_metadata.py", line 124, in _generate_metadata_legacy
    command_desc='python setup.py egg_info',
  File "c:\users\user.name\pyenvs\piptemp\lib\site-packages\pip\_internal\utils\subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[atugushev]

Hello @YKdvd,

Thanks for the issue! I believe that's because the packages have own issues with setuptools or you probably missed some of the requirements. See the tracebacks below.

1. Package edl has issue with setup.py, were import edl cause import timecode before its installation. There is a tracking issue pip install edl doesn't automatically include timecode library simonh10/python-edl#5.

$ echo edl | pip-compile - -qo-
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/albert/Projects/pip-tools/.venv/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-req-tracker-buu1cwsg/edl/setup.py'"'"'; __file__='"'"'/private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-req-tracker-buu1cwsg/edl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-req-tracker-buu1cwsg/edl/pip-egg-info
     cwd: /private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-req-tracker-buu1cwsg/edl/
Complete output (7 lines):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-req-tracker-buu1cwsg/edl/setup.py", line 4, in <module>
    import edl
  File "/private/var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/pip-req-tracker-buu1cwsg/edl/edl/__init__.py", line 9, in <module>
    import timecode
ModuleNotFoundError: No module named 'timecode'

2. To install mod_wsgi you must have Apache installed. Run apt install apache2-dev on Debian/Ubuntu to fix the issue.

$ echo mod_wsgi | pip-compile - -qo-
Command errored out with exit status 1:
 command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-tracker-w2mhr7w5/mod-wsgi/setup.py'"'"'; __file__='"'"'/tmp/pip-req-tracker-w2mhr7w5/mod-wsgi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-tracker-w2mhr7w5/mod-wsgi/pip-egg-info
     cwd: /tmp/pip-req-tracker-w2mhr7w5/mod-wsgi/
Complete output (5 lines):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-tracker-w2mhr7w5/mod-wsgi/setup.py", line 168, in <module>
    'missing Apache httpd server packages.' % APXS)
RuntimeError: The 'apxs' command appears not to be installed or is not executable. Please check the list of prerequisites in the documentation for this package and install any missing Apache httpd server packages.

Also, try to install these packages directly by pip, you'll see the same tracebacks.

[atugushev]

I'll close this since it doesn't look like there's any action required regarding pip-tools, but reach us out if you have any other questions. Thanks for the issue!