Cannot install backoff: tomlkit-0.5.3.tar.gz is already being built

[glasserc]

I'm trying to build a project with a lot of dependencies, but I think I've narrowed it down to backoff.

> pypi2nix -v -V 3.7 -e backoff
pypi2nix v2.0.0
 running ...

Stage1: Downloading wheels and creating wheelhouse ...
|-> nix-shell /nix/store/a7rzipwndw9jcw44538aicjk6p3hqicv-python3.7-pypi2nix-2.0.0/lib/python3.7/site-packages/pypi2nix/pip.nix --arg requirements_files [ "/tmp/pypi2nix/d41d8cd98f00b204e9800998ecf8427e/697475eb603a42a7e7ef73faba71ae1c.txt" ] --argstr project_dir /tmp/pypi2nix/d41d8cd98f00b204e9800998ecf8427e --argstr download_cache_dir /tmp/pypi2nix/cache/download --argstr wheel_cache_dir /tmp/pypi2nix/cache/wheel --arg extra_build_inputs [  ] --argstr extra_env  --argstr python_version python37 --arg setup_requires [  ] --arg wheels_cache [  ] -I /home/ethan/.nix-defexpr/channels:nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels --show-trace --pure --run exit
    
    ===================================================================
    download source distributions to: /tmp/pypi2nix/cache/download
    ===================================================================
    
    Looking in links: file:///tmp/pypi2nix/cache/download, file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index
    Collecting backoff (from -r /tmp/pypi2nix/d41d8cd98f00b204e9800998ecf8427e/697475eb603a42a7e7ef73faba71ae1c.txt (line 1))
      Downloading https://files.pythonhosted.org/packages/84/1d/1b7c1502d91f86c3bd38f693c9c037bf45d2684eaf071426da27a8a2761c/backoff-1.7.1.tar.gz
      Saved /tmp/pypi2nix/cache/download/backoff-1.7.1.tar.gz
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'
      Complete output from command /nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-ep0znsmb --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12:
      Looking in links: file:///tmp/pypi2nix/cache/download, file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index
      Collecting poetry>=0.12
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
      Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry>=0.12)
      Collecting cachy<0.3,>=0.2 (from poetry>=0.12)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
      Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
      Collecting html5lib<2.0,>=1.0 (from poetry>=0.12)
      Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
      Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12)
      Collecting pyparsing<3.0,>=2.2 (from poetry>=0.12)
      Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry>=0.12)
      Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry>=0.12)
      Collecting requests<3.0,>=2.18 (from poetry>=0.12)
      Collecting shellingham<2.0,>=1.1 (from poetry>=0.12)
      Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'error'
        Complete output from command /nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-dp9tatpk --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12a3:
        Looking in links: file:///tmp/pypi2nix/cache/download, file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index
        Collecting poetry>=0.12a3
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
        Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry>=0.12a3)
        Collecting cachy<0.3,>=0.2 (from poetry>=0.12a3)
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
        Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12a3)
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
        Collecting html5lib<2.0,>=1.0 (from poetry>=0.12a3)
        Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12a3)
        Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12a3)
        Collecting pyparsing<3.0,>=2.2 (from poetry>=0.12a3)
        Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry>=0.12a3)
        Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry>=0.12a3)
        Collecting requests<3.0,>=2.18 (from poetry>=0.12a3)
        Collecting shellingham<2.0,>=1.1 (from poetry>=0.12a3)
        Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12a3)
        Exception:
        Traceback (most recent call last):
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/cli/base_command.py", line 143, in main
            status = self.run(options, args)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/commands/install.py", line 318, in run
            resolver.resolve(requirement_set)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/resolve.py", line 102, in resolve
            self._resolve_one(requirement_set, req)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/resolve.py", line 256, in _resolve_one
            abstract_dist = self._get_abstract_dist_for(req_to_install)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
            self.require_hashes
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/operations/prepare.py", line 297, in prepare_linked_requirement
            with self.req_tracker.track(req):
          File "/nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/lib/python3.7/contextlib.py", line 112, in __enter__
            return next(self.gen)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/req/req_tracker.py", line 74, in track
            self.add(req)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/req/req_tracker.py", line 46, in add
            % (link, fp.read()))
        LookupError: file:///tmp/pypi2nix/cache/download/tomlkit-0.5.3.tar.gz is already being built: tomlkit<0.6.0,>=0.5.1 from file:///tmp/pypi2nix/cache/download/tomlkit-0.5.3.tar.gz (from poetry>=0.12)
      
        ----------------------------------------
      Command "/nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-dp9tatpk --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12a3" failed with error code 2 in None
      
      ----------------------------------------
    Command "/nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-ep0znsmb --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12" failed with error code 1 in None
    
    ===================================================================
    create wheels from source distributions without going online
    ===================================================================
    
    Looking in links: file:///tmp/pypi2nix/cache/wheel, file:///tmp/pypi2nix/cache/download, file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index
    Collecting backoff (from -r /tmp/pypi2nix/d41d8cd98f00b204e9800998ecf8427e/697475eb603a42a7e7ef73faba71ae1c.txt (line 1))
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'
      Complete output from command /nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-9vg_8dy8 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///tmp/pypi2nix/cache/wheel --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12:
      Looking in links: file:///tmp/pypi2nix/cache/wheel, file:///tmp/pypi2nix/cache/download, file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index
      Collecting poetry>=0.12
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
      Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry>=0.12)
      Collecting cachy<0.3,>=0.2 (from poetry>=0.12)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
      Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
      Collecting html5lib<2.0,>=1.0 (from poetry>=0.12)
      Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
      Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12)
      Collecting pyparsing<3.0,>=2.2 (from poetry>=0.12)
      Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry>=0.12)
      Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry>=0.12)
      Collecting requests<3.0,>=2.18 (from poetry>=0.12)
      Collecting shellingham<2.0,>=1.1 (from poetry>=0.12)
      Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'error'
        Complete output from command /nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-cekbgw15 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///tmp/pypi2nix/cache/wheel --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12a3:
        Looking in links: file:///tmp/pypi2nix/cache/wheel, file:///tmp/pypi2nix/cache/download, file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index
        Collecting poetry>=0.12a3
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
        Collecting cachecontrol[filecache]<0.13.0,>=0.12.4 (from poetry>=0.12a3)
        Collecting cachy<0.3,>=0.2 (from poetry>=0.12a3)
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
        Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12a3)
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
        Collecting html5lib<2.0,>=1.0 (from poetry>=0.12a3)
        Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12a3)
        Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12a3)
        Collecting pyparsing<3.0,>=2.2 (from poetry>=0.12a3)
        Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry>=0.12a3)
        Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry>=0.12a3)
        Collecting requests<3.0,>=2.18 (from poetry>=0.12a3)
        Collecting shellingham<2.0,>=1.1 (from poetry>=0.12a3)
        Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12a3)
        Exception:
        Traceback (most recent call last):
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/cli/base_command.py", line 143, in main
            status = self.run(options, args)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/commands/install.py", line 318, in run
            resolver.resolve(requirement_set)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/resolve.py", line 102, in resolve
            self._resolve_one(requirement_set, req)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/resolve.py", line 256, in _resolve_one
            abstract_dist = self._get_abstract_dist_for(req_to_install)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
            self.require_hashes
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/operations/prepare.py", line 297, in prepare_linked_requirement
            with self.req_tracker.track(req):
          File "/nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/lib/python3.7/contextlib.py", line 112, in __enter__
            return next(self.gen)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/req/req_tracker.py", line 74, in track
            self.add(req)
          File "/nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/base/pip/_internal/req/req_tracker.py", line 46, in add
            % (link, fp.read()))
        LookupError: file:///tmp/pypi2nix/cache/download/tomlkit-0.5.3.tar.gz is already being built: tomlkit<0.6.0,>=0.5.1 from file:///tmp/pypi2nix/cache/download/tomlkit-0.5.3.tar.gz (from poetry>=0.12)
      
        ----------------------------------------
      Command "/nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-cekbgw15 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///tmp/pypi2nix/cache/wheel --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12a3" failed with error code 2 in None
      
      ----------------------------------------
    Command "/nix/store/rk8wzrrly4h13y3pmcsidhxbgfa9h8d5-python3-3.7.1/bin/python3.7m -m pip install --ignore-installed --no-user --prefix /run/user/1000/pip-build-env-9vg_8dy8 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///tmp/pypi2nix/cache/wheel --find-links file:///tmp/pypi2nix/cache/download --find-links file:///nix/store/zizbp0l4rj20afmki38abxgq2bhqm0hz-pypi2nix-bootstrap/index -- poetry>=0.12" failed with error code 1 in None
Do you want to report above issue (a browser will open with prefilled details of issue)? [y/N]:

I'm running pypi2nix master.

[glasserc]

I think part of the problem is that tomlkit depends on poetry and poetry also depends on tomlkit. pip seems able to install this cycle under normal conditions but I'm not sure how.

[glasserc]

It seems like this is (at least in part) due to the --no-binary :all: flags on the pip commands. The pip documentation even says: "Note that some packages are tricky to compile and may fail to install when this option is used on them." I guess that might cover circular dependencies. I don't really understand enough about pypi2nix to understand why the --no-binary :all: flag is there, but I was able to take it out and eventually get things working.

[garbas]

I encounter the same problem with backoff package. problem is that backoff package switched to poetry for their packaging which is currently not working with pypi2nix. I haven't yet have time to dig deeper into this. For now I just pinned backoffto 1.6.0 where installation from source was still possible (mozilla/release-services@bdebc78#diff-cec043b792559a4f14d84ecb57b6d630R6)

--no-binary :all: is needed since we must require all packages to be built from source, wheels might depend on some path that does not exist (eg. /usr/...).

[garbas]

you can reproduce the above you can simple run:

$ pip install tomlkit --no-binary :all:

This means you can not install tomlkit from source distribution

[garbas]

Reported the problem upstream python-poetry/poetry#760 and litl/backoff#67

[cawilliamson]

Is there any workaround for this issue available at the moment?

I really want to deploy Apache Airflow (latest version) on NixOS using Nix but until this is resolved I simply can't progress with it and am forced to use an older version.

[reversefold]

I discuss my workaround in this comment on another ticket: python-poetry/poetry#760 (comment)

This is for if you are directly calling pip and able to set the --no-binary command-line arguments yourself. I don't know if pypi2nix supports this or not, I just got here by searching for the error. Hope it helps, though.