Bug 1767003

Summary: pipenv does not work on Fedora 31 with Python 2.7
Product: [Fedora] Fedora Reporter: Lumír Balhar <lbalhar>
Component: pipenvAssignee: Miro Hrončok <mhroncok>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 31CC: m.cyprian, mhroncok, python-sig
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pipenv-2018.11.26-12.fc30 pipenv-2018.11.26-12.fc29 pipenv-2018.11.26-12.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-09 22:39:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lumír Balhar 2019-10-30 13:15:33 UTC
Description of problem:
pipenv does not seem to work after fresh install in Docker container with Fedora 31.

Version-Release number of selected component (if applicable):
pipenv-2018.11.26-8.fc31.noarch

How reproducible:
Always

Steps to Reproduce:
1. # dnf install pipenv python27

2. create simple Pipfile:
# cat Pipfile 
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "==2.20.0"

[dev-packages]
pytest = ">=2.8.0"

[requires]
python_version = "2.7"

3. # pipenv install

Actual results:
Creating a virtualenv for this project…
Pipfile: /Pipfile
Using /usr/bin/python2.7 (2.7.17) to create virtualenv…
⠏ Creating virtual environment...Already using interpreter /usr/bin/python2.7
New python executable in /root/.local/share/virtualenvs/-x-v5uFv0/bin/python2.7
Also creating executable in /root/.local/share/virtualenvs/-x-v5uFv0/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7

✔ Successfully created virtual environment! 
Virtualenv location: /root/.local/share/virtualenvs/-x-v5uFv0
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
✘ Locking Failed! 
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 97, in main
    from pipenv.vendor.vistir.compat import ResourceWarning
  File "/usr/lib/python3.7/site-packages/pipenv/__init__.py", line 22, in <module>
    from pipenv.vendor.vistir.compat import ResourceWarning, fs_str
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/compat.py", line 36, in <module>
    from pathlib2 import Path
ImportError: No module named pathlib2
File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 97, in main
    from pipenv.vendor.vistir.compat import ResourceWarning
  File "/usr/lib/python3.7/site-packages/pipenv/__init__.py", line 22, in <module>
    from pipenv.vendor.vistir.compat import ResourceWarning, fs_str
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/compat.py", line 36, in <module>
    from pathlib2 import Path
ImportError: No module named pathlib2

Expected results:
Everything works without errors, environment is created and dependencies installed

Additional info:
Manual installation of python3-pathlib2 does not fix the problem:

# pipenv install
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
✘ Locking Failed! 
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 97, in main
    from pipenv.vendor.vistir.compat import ResourceWarning
  File "/usr/lib/python3.7/site-packages/pipenv/__init__.py", line 22, in <module>
    from pipenv.vendor.vistir.compat import ResourceWarning, fs_str
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/compat.py", line 36, in <module>
    from pathlib2 import Path
  File "/usr/lib/python3.7/site-packages/pathlib2/__init__.py", line 52, in <module>
    from scandir import scandir as os_scandir
ImportError: No module named scandir
File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 97, in main
    from pipenv.vendor.vistir.compat import ResourceWarning
  File "/usr/lib/python3.7/site-packages/pipenv/__init__.py", line 22, in <module>
    from pipenv.vendor.vistir.compat import ResourceWarning, fs_str
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/compat.py", line 36, in <module>
    from pathlib2 import Path
  File "/usr/lib/python3.7/site-packages/pathlib2/__init__.py", line 52, in <module>
    from scandir import scandir as os_scandir
ImportError: No module named scandir

python3-scandir is installed as a dependency of pipenv.

It also seems that pipenv-2018.11.26-7 on Fedora 30 has the same bug. And finally, in rawhide with the same Pipfile and pipenv-2018.11.26-11.fc32 I get:

# pipenv install
Creating a virtualenv for this project…
Pipfile: /tmp/Pipfile
Using /usr/bin/python2.7 (2.7.17) to create virtualenv…
⠼ Creating virtual environment...Already using interpreter /usr/bin/python2.7
New python executable in /root/.local/share/virtualenvs/tmp-XVr6zr33/bin/python2.7
Also creating executable in /root/.local/share/virtualenvs/tmp-XVr6zr33/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2.7

✔ Successfully created virtual environment! 
Virtualenv location: /root/.local/share/virtualenvs/tmp-XVr6zr33
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
✘ Locking Failed! 
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.8/site-packages/pipenv/resolver.py", line 97, in main
    from vistir.compat import ResourceWarning
  File "/usr/lib/python3.8/site-packages/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.8/site-packages/vistir/compat.py", line 13, in <module>
    from .backports.tempfile import NamedTemporaryFile as _NamedTemporaryFile
  File "/usr/lib/python3.8/site-packages/vistir/backports/__init__.py", line 6, in <module>
    from .tempfile import NamedTemporaryFile
  File "/usr/lib/python3.8/site-packages/vistir/backports/tempfile.py", line 15, in <module>
    from backports.weakref import finalize
ImportError: No module named backports.weakref
File "/usr/lib/python3.8/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.8/site-packages/pipenv/resolver.py", line 97, in main
    from vistir.compat import ResourceWarning
  File "/usr/lib/python3.8/site-packages/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.8/site-packages/vistir/compat.py", line 13, in <module>
    from .backports.tempfile import NamedTemporaryFile as _NamedTemporaryFile
  File "/usr/lib/python3.8/site-packages/vistir/backports/__init__.py", line 6, in <module>
    from .tempfile import NamedTemporaryFile
  File "/usr/lib/python3.8/site-packages/vistir/backports/tempfile.py", line 15, in <module>
    from backports.weakref import finalize

What I am doing wrong?

Comment 1 Miro Hrončok 2019-10-30 13:41:17 UTC
The bloody tool uses the bundled pathlib2 even when running on Python 3 but creating Python 2 environments.

We need to re-bundle it.

Comment 2 Miro Hrončok 2019-10-30 13:51:54 UTC
I've installed python3-pathlib2 and the problem seems gone for me. Cannot longer reproduce form a fresh directory:

$ rpm -q pipenv
pipenv-2018.11.26-8.fc31.noarch

$ rpm -q python3-pathlib2
python3-pathlib2-2.3.4-2.fc31.noarch

$ mkdir new
$ cd new

$ cat > Pipenv << EOF
> [[source]]
> url = "https://pypi.python.org/simple"
> verify_ssl = true
> name = "pypi"
> 
> [packages]
> requests = "==2.20.0"
> 
> [dev-packages]
> pytest = ">=2.8.0"
> 
> [requires]
> python_version = "2.7"
> EOF

$ pipenv install
Installing dependencies from Pipfile.lock (dfae9f)…
  🎃   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Comment 3 Miro Hrončok 2019-10-30 13:53:39 UTC
OK, this reprduces entirely random:

[12345]$ pipenv install
Creating a virtualenv for this project…
Pipfile: /home/churchyard/tmp/12345/Pipfile
Using /usr/bin/python2 (2.7.17) to create virtualenv…
⠋ Creating virtual environment...Already using interpreter /usr/bin/python2
New python executable in /home/churchyard/.local/share/virtualenvs/12345-Zmm6krPD/bin/python2
Also creating executable in /home/churchyard/.local/share/virtualenvs/12345-Zmm6krPD/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python2

✔ Successfully created virtual environment! 
Virtualenv location: /home/churchyard/.local/share/virtualenvs/12345-Zmm6krPD
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
✘ Locking Failed! 
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 97, in main
    from pipenv.vendor.vistir.compat import ResourceWarning
  File "/usr/lib/python3.7/site-packages/pipenv/__init__.py", line 22, in <module>
    from pipenv.vendor.vistir.compat import ResourceWarning, fs_str
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/compat.py", line 36, in <module>
    from pathlib2 import Path
  File "/usr/lib/python3.7/site-packages/pathlib2/__init__.py", line 52, in <module>
    from scandir import scandir as os_scandir
ImportError: No module named scandir
File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/pipenv/resolver.py", line 97, in main
    from pipenv.vendor.vistir.compat import ResourceWarning
  File "/usr/lib/python3.7/site-packages/pipenv/__init__.py", line 22, in <module>
    from pipenv.vendor.vistir.compat import ResourceWarning, fs_str
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/__init__.py", line 4, in <module>
    from .compat import (
  File "/usr/lib/python3.7/site-packages/pipenv/vendor/vistir/compat.py", line 36, in <module>
    from pathlib2 import Path
  File "/usr/lib/python3.7/site-packages/pathlib2/__init__.py", line 52, in <module>
    from scandir import scandir as os_scandir
ImportError: No module named scandir

Comment 4 Miro Hrončok 2019-10-30 14:03:52 UTC
OK, so pipenv runs /usr/lib/python3.7/site-packages/pathlib2/__init__.py with Python 2.7.
It cannot find scandir, because it doesn't look for it in /usr/lib64/python3.7/site-packages/

I'll just bundle pathlib2, scandirs, backports.lru_cache again.

Comment 5 Miro Hrončok 2019-10-30 14:16:55 UTC
Fix attempt: https://src.fedoraproject.org/rpms/pipenv/pull-request/19

Comment 6 Miro Hrončok 2019-10-30 16:44:06 UTC
Could you please test pipenv on various Fedoras?

f32: https://src.fedoraproject.org/rpms/pipenv/pull-request/19
f31: https://src.fedoraproject.org/rpms/pipenv/pull-request/20
f30: https://src.fedoraproject.org/rpms/pipenv/pull-request/21
f29: https://src.fedoraproject.org/rpms/pipenv/pull-request/22


Note that the patches are slightly different, but I kept the release number same on purpose to make further cherry-picking easier.

Comment 7 Lumír Balhar 2019-10-31 07:16:37 UTC
Tested all Fedoras with the original reproducer and it seems to be fixed. I am going to test it more on my F31 because I have some tasks for S2I I need pipenv for.

Comment 8 Lumír Balhar 2019-10-31 07:56:20 UTC
Tested on Fedora 31 with Python 2.7, 3.6 and 3.7 in virtual environments and everything works.

Comment 9 Miro Hrončok 2019-10-31 09:09:44 UTC
Thanks!

Comment 10 Fedora Update System 2019-10-31 10:16:26 UTC
FEDORA-2019-b9d986e330 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-b9d986e330

Comment 11 Fedora Update System 2019-10-31 10:16:30 UTC
FEDORA-2019-b5f53f4280 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-b5f53f4280

Comment 12 Fedora Update System 2019-11-02 02:26:20 UTC
pipenv-2018.11.26-12.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-de521ba933

Comment 13 Fedora Update System 2019-11-02 04:13:19 UTC
pipenv-2018.11.26-12.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-b9d986e330

Comment 14 Fedora Update System 2019-11-03 06:00:53 UTC
pipenv-2018.11.26-12.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-b5f53f4280

Comment 15 Fedora Update System 2019-11-09 22:39:04 UTC
pipenv-2018.11.26-12.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2019-11-10 01:06:50 UTC
pipenv-2018.11.26-12.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2019-11-11 01:05:24 UTC
pipenv-2018.11.26-12.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.