Pipfile with private index AND optional dependency doesn't work #6187

scastria · 2024-07-05T03:51:38Z

I have been using pipenv 2022.9.21 for a long time and decided to try the latest and greatest. The latest 2024.0.1 appears broken when it comes to locating packages in my private AWS CodeArtifact repo.

Issue description

pipenv install fails with this Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
"my_utils[markdown]" = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

I thought it might be a failure in resolving the env variable referenced in the alternate source index called data_engineering so I hardcoded it in the Pipfile. No change.

Expected result

my_utils package is downloaded successfully

Actual result

Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠏ Locking packages...False
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement my_utils==0.1.391 (from versions: none)
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 645, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 612, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 592, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 932, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 700, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 457, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for my_utils==0.1.391

Traceback (most recent call last):
  File "/opt/homebrew/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 653, in do_init
    do_lock(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 737, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

pipenv install

$ pipenv --support

Pipenv version: '2024.0.1'

Pipenv location: '/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv'

Python location: '/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/bin/python'

OS Name: 'posix'

User pip version: '24.0'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.12.4',
 'os_name': 'posix',
 'platform_machine': 'arm64',
 'platform_python_implementation': 'CPython',
 'platform_release': '23.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 23.5.0: Wed May  1 20:12:58 PDT '
                     '2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000',
 'python_full_version': '3.12.4',
 'python_version': '3.12',
 'sys_platform': 'darwin'}

System environment variables:

MANPATH
TERM_PROGRAM
SHELL
TERM
HISTSIZE
HOMEBREW_REPOSITORY
TMPDIR
GOBIN
TERM_PROGRAM_VERSION
ADOPS_PAT
OLDPWD
TERM_SESSION_ID
TF_VAR_postman_localhost_api_key
FORGE_EMAIL
USER
LDAP_USERNAME
CODEARTIFACT_AUTH_TOKEN
SSH_AUTH_SOCK
PATH
__CFBundleIdentifier
PWD
JAVA_HOME
LANG
LDAP_PASSWORD
FORGE_API_TOKEN
XPC_FLAGS
PS1
ENVIRONMENT
XPC_SERVICE_NAME
SHLVL
HOME
SHELL_SESSION_HISTORY
HOMEBREW_PREFIX
LOGNAME
LOCAL_MYSQL_ROOT_PASSWORD
GOPATH
INFOPATH
HOMEBREW_CELLAR
BITBUCKET_PAT
HISTTIMEFORMAT
_
__CF_USER_TEXT_ENCODING
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/Users/shawncastrianni/.go/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/opt/liquibase
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /Users/shawncastrianni/GIT/data_api_one

Contents of Pipfile ('/Users/shawncastrianni/GIT/data_api_one/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
"my_utils[markdown]" = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

The text was updated successfully, but these errors were encountered:

scastria · 2024-07-05T03:54:48Z

I also tried verify_ssl = false, but no change

scastria · 2024-07-05T04:02:24Z

I discovered a BIG clue to this. It seems to be related to the optional dependency of the my_utils package called markdown. If I remove mention of the optional dependency like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
my_utils = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

then the pipenv install command worked. It seems the optional dependency causes a problem. The optional dependency IS VALID as pipenv 2022.9.21 works just fine.

scastria · 2024-07-05T04:04:00Z

Another test is that I removed the my_utils dependency completely from the Pipfile and instead specified ONLY "redis[hiredis]" = "==4.6.0" and that worked too. So optional dependencies work against PyPI, but they do NOT work against private indicies.

scastria · 2024-07-05T15:30:57Z

I wasn't aware of the extras keyword in Pipfile. If I change the Pipfile to use extras, it DOES work. So I guess now the question is should the extras syntax be required or can the other syntax without extras be used?

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
my_utils = {version="==0.1.391", extras=["markdown"], index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

matteius · 2024-08-03T16:28:06Z

Currently the extras key should be defined, I believe when you run pipenv install "my_utils[markdown]" it should be generating the correct Pipfile entry -- its not clear to me from your report if your initial entry was self input or generated from the install command? Current expectations is the key is just the package name (without extras).

scastria · 2024-08-05T14:55:43Z

Let me try to clarify:

With pipenv 2022.9.21:

Pipfile contains {redis = {version="==4.6.0", extras=["hiredis"]} and pipenv install WORKS
Pipfile contains "redis[hiredis]" = "==4.6.0" and pipenv install WORKS
Pipfile contains my private package my_utils = {version="0.1.391", extras=["markdown"], index="my_private_index"} and pipenv install WORKS
Pipfile contains my private package "my_utils[markdown]" = {version="0.1.391", index="my_private_index"} and pipenv install WORKS

With pipenv 2024.0.1:

Pipfile contains {redis = {version="==4.6.0", extras=["hiredis"]} and pipenv install WORKS
Pipfile contains "redis[hiredis]" = "==4.6.0" and pipenv install WORKS
Pipfile contains my private package my_utils = {version="0.1.391", extras=["markdown"], index="my_private_index"} and pipenv install WORKS
Pipfile contains my private package "my_utils[markdown]" = {version="0.1.391", index="my_private_index"} and pipenv install DOES NOT WORK

elanou · 2024-08-21T22:04:29Z

2024.* broke something. All my installs fail with it trying to source setuptools from a private repo I have defined as a source. Stepping down to latest 2023.* fixes it.

× pip subprocess to install build dependencies did not run successfully. [pipenv.exceptions.InstallError]: │ exit code: 1 [pipenv.exceptions.InstallError]: ╰─> [3 lines of output] [pipenv.exceptions.InstallError]: Looking in indexes: <REDACTED> [pipenv.exceptions.InstallError]: ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none) [pipenv.exceptions.InstallError]: ERROR: No matching distribution found for setuptools>=40.8.0 [pipenv.exceptions.InstallError]: [end of output]

matteius · 2024-10-19T22:47:34Z

Could this issue be checked against this branch? #6276

scastria changed the title ~~Pipfile with private index doesn't work~~ Pipfile with private index AND optional dependency doesn't work Jul 5, 2024

matteius added the triage label Aug 25, 2024

oz123 added the Category: Private PyPIs 😎 Problem relates to private PyPI usage. label Oct 4, 2024

matteius mentioned this issue Oct 19, 2024

install should not do full lock resolution, as that is the purpose of pipenv lock #6276

Merged

2 tasks

matteius closed this as completed in #6276 Oct 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipfile with private index AND optional dependency doesn't work #6187

Pipfile with private index AND optional dependency doesn't work #6187

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading

matteius commented Aug 3, 2024

scastria commented Aug 5, 2024

elanou commented Aug 21, 2024

matteius commented Oct 19, 2024

Pipfile with private index AND optional dependency doesn't work #6187

Pipfile with private index AND optional dependency doesn't work #6187

Comments

scastria commented Jul 5, 2024 • edited Loading

Issue description

Expected result

Actual result

Steps to replicate

scastria commented Jul 5, 2024 • edited Loading

scastria commented Jul 5, 2024 • edited Loading

scastria commented Jul 5, 2024 • edited Loading

scastria commented Jul 5, 2024 • edited Loading

matteius commented Aug 3, 2024

scastria commented Aug 5, 2024

elanou commented Aug 21, 2024

matteius commented Oct 19, 2024

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading

scastria commented Jul 5, 2024 •

edited

Loading