Bug 1396158

Summary: fails to remove the f24 version of the python2-requests after updating to f25
Product: [Fedora] Fedora Reporter: mock <mark>
Component: python-requestsAssignee: Fedora Infrastructure SIG <infra-sig>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: cstratak, davidshumway, infra-sig, jeremy, rbean, sagarun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-17 18:07:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description mock 2016-11-17 15:44:03 UTC
Description of problem:


Version-Release number of selected component (if applicable):

python2-requests-2.10.0-2.fc24.noarch

also...

$ uname -a
Linux liberia 4.8.6-300.fc25.x86_64 #1 SMP Tue Nov 1 12:36:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/system-release
Fedora release 25 (Twenty Five)


How reproducible:

too easy to reproduce. 


Steps to Reproduce:
1. sudo dnf upgrade-to python2-requests-2.10.0-4.fc25.noarch
2.
3.

Actual results:

this output was what i got when trying the update:

----
Dependencies resolved.
================================================================================
 Package                 Arch          Version              Repository     Size
================================================================================
Upgrading:
 python2-requests        noarch        2.10.0-4.fc25        fedora        107 k

Transaction Summary
================================================================================
Upgrade  1 Package

Total download size: 107 k
Downloading Packages:
--------------------------------------------------------------------------------
Total                                           162 kB/s | 107 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Upgrading   : python2-requests-2.10.0-4.fc25.noarch                       1/2 
error: unpacking of archive failed on file /usr/lib/python2.7/site-packages/requests/packages/chardet: cpio: rename
  Verifying   : python2-requests-2.10.0-4.fc25.noarch                       1/2 
  Verifying   : python2-requests-2.10.0-2.fc24.noarch                       2/2 

Failed:
  python2-requests.noarch 2.10.0-4.fc25                                         

Complete!
----


Expected results:

upgraded python2-requests to 2.10.0-4.fc25.noarch


Additional info:

Comment 1 Jeremy Cline 2016-11-17 16:21:20 UTC
*** Bug 1396160 has been marked as a duplicate of this bug. ***

Comment 2 Jeremy Cline 2016-11-17 16:46:21 UTC
Hey, thanks for the report.

I wasn't able to reproduce this with a fresh F24 box that I dnf system-upgraded. Did you by chance use pip to install anything as root?

Comment 3 mock 2016-11-17 18:01:10 UTC
(In reply to Jeremy Cline from comment #2)
> Hey, thanks for the report.
> 
> I wasn't able to reproduce this with a fresh F24 box that I dnf
> system-upgraded. Did you by chance use pip to install anything as root?

yes, i had installed a few things using pip. i don't remember them all, but i know i did add pip (upgrade), boto, awscli, and maybe moto. i was a super to install those.

i did try to pip install the requests package via pip AFTER setting up the fpaste output (posted in the Actual results section of the ticket), and that wasn't working either. 

should i try downgrading via pip or do i need physically wipe out the effected modules from site-packages and try downloading again? is this just a permissions problem on the directory for the requests module?

Comment 4 mock 2016-11-17 18:06:08 UTC
well, i just tried that, and it worked.

i moved the requests directory from the /usr/lib/python2.7/site-packages directory to a tmp directory and then issued the dnf upgrade again.

it pulled the pacakge down and removed the old one without problems.

i think we can close this.

----
$ sudo dnf upgrade --refresh
Atlassian Hipchat                                                                       194 kB/s |  57 kB     00:00    
nuvolaplayer repository, component stable (fc23 x86_64)                                  19 kB/s |  11 kB     00:00    
Opera packages                                                                           35 kB/s |  12 kB     00:00    
Paper Project (Fedora_24)                                                                25 kB/s | 144 kB     00:05    
Failed to synchronize cache for repo 'google-chrome', disabling.
Failed to synchronize cache for repo 'home_Horst3180', disabling.
Dependencies resolved.
========================================================================================================================
 Package                           Arch                    Version                        Repository               Size
========================================================================================================================
Upgrading:
 python2-requests                  noarch                  2.10.0-4.fc25                  fedora                  107 k

Transaction Summary
========================================================================================================================
Upgrade  1 Package

Total download size: 107 k
Is this ok [y/N]: y
Downloading Packages:
python2-requests-2.10.0-4.fc25.noarch.rpm                                               545 kB/s | 107 kB     00:00    
------------------------------------------------------------------------------------------------------------------------
Total                                                                                    59 kB/s | 107 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Upgrading   : python2-requests-2.10.0-4.fc25.noarch                                                               1/2 
  Cleanup     : python2-requests-2.10.0-2.fc24.noarch                                                               2/2 
  Verifying   : python2-requests-2.10.0-4.fc25.noarch                                                               1/2 
  Verifying   : python2-requests-2.10.0-2.fc24.noarch                                                               2/2 

Upgraded:
  python2-requests.noarch 2.10.0-4.fc25                                                                                 

Complete!
----

Comment 5 Jeremy Cline 2016-11-17 18:18:07 UTC
Glad you got it sorted!

Yeah, pip and dnf do not play well together since neither one is aware of the other, and this is especially painful with requests because upstream bundles all their libraries inside their package and we unbundle them. I would generally recommend to not sudo pip install if you can avoid it, since you'll occasionally hit issues like this (or suddenly certain imports don't work).

Comment 6 davidshumway 2017-03-08 00:27:22 UTC
Yes So this also solved the problem of upgrading from 24 to 25 for python2-requests-2.10.0-4.fc25.noarch.

>>"i moved the requests directory from the /usr/lib/python2.7/site-packages directory to a tmp directory and then issued the dnf upgrade again."

However, I then moved the site-packages folder back to /usr/lib/python2.7/site-packages and did "dnf upgrade --refresh". Is this correct? It is odd because there are some software programs which require python which are now missing dependencies. For example in a scrapy program the following message appears when running scrapyd-deploy (which did not appear prior to this): "{"status": "error", "message": "ImportError: No module named cssselect"}".

Comment 7 davidshumway 2017-03-08 00:44:16 UTC
Or for example:

>>$ pip install cssselect
>>Traceback (most recent call last):
>>  File "/usr/bin/pip", line 7, in <module>
>>    from pip import main
>>ImportError: No module named pip

Pip was here before moving site-packages around.

Comment 8 Jeremy Cline 2017-03-08 00:47:01 UTC
Oh, you definitely don't want to move it back. You also definitely should not mix ``sudo pip install`` with dnf-installed packages.

Comment 9 davidshumway 2017-03-08 00:49:32 UTC
>>Oh, you definitely don't want to move it back. You also definitely should not mix ``sudo pip install`` with dnf-installed packages.

So by not moving it back now how does python know what modules are available? For example pip was available and now is no longer available.

Comment 10 davidshumway 2017-03-08 00:49:52 UTC
Thanks for your help btw...

Comment 11 davidshumway 2017-03-08 00:51:57 UTC
For example /usr/lib/python2.7/site-packages/ now contains:

>>examples  requests  requests-2.10.0-py2.7.egg-info  tweepy  tweepy-2.3.0-py2.7.egg-info

Comment 12 Jeremy Cline 2017-03-08 00:55:23 UTC
Did you move the entire ``site-packages`` directory?

Comment 13 davidshumway 2017-03-08 00:57:01 UTC
Hm. Yes, probably. In the site-packages folder that has been moved to /tmp there are 572 items.

Comment 14 davidshumway 2017-03-08 00:59:17 UTC
Is it enough to simply move the 572 files back to the /usr/lib/python2.7/site-packages/ folder?

Comment 15 Jeremy Cline 2017-03-08 01:03:11 UTC
You can try that, but if you used dnf to upgrade packages after you moved site-packages, things might be broken. Maybe try downgrading any packages you upgraded before you moved everything, move everything back, move _only_ the requests directory, upgrade again, and make sure everything works?

Comment 16 davidshumway 2017-03-08 01:18:37 UTC
Okay...

>>$ sudo cp -Rf /tmp/site-packages/* /usr/lib/python2.7/site-packages/
>>cp: cannot overwrite non-directory '/usr/lib/python2.7/site-packages/requests/packages/chardet' with directory '/tmp/abc/requests/packages/chardet'
>>cp: cannot overwrite non-directory '/usr/lib/python2.7/site-packages/requests/packages/urllib3' with directory '/tmp/abc/requests/packages/urllib3'

And now things are working again. Pip works. Other programs with dependencies work.

Comment 17 davidshumway 2017-03-08 01:23:27 UTC
Well almost everything. Seeing this again:

>>$ ./retweet.py 
>>Traceback (most recent call last):
>>  File "./retweet.py", line 3, in <module>
>>    import tweepy
>>  File "build/bdist.linux-x86_64/egg/tweepy/__init__.py", line 14, in <module>
>>  File "build/bdist.linux-x86_64/egg/tweepy/api.py", line 12, in <module>
>>  File "build/bdist.linux-x86_64/egg/tweepy/binder.py", line 11, in <module>
>>  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 64, in <module>
>>    from . import utils
>>  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 26, in <module>
>>    from .compat import (
>>ImportError: cannot import name integer_types

Which is what was happening after upgrading from Fedora 24 to Fedora 25.

Comment 18 davidshumway 2017-03-08 01:32:19 UTC
Did not downgrade anything. May try moving only requests directory and upgrading again...

Comment 19 davidshumway 2017-03-08 02:05:15 UTC
Maybe will try moving these and then upgrading...

>>requests  requests-2.10.0-py2.7.egg-info  tweepy  tweepy-2.3.0-py2.7.egg-info

Comment 20 davidshumway 2017-03-08 03:34:18 UTC
Hm. Well pip upgrade makes tweepy work again:

>>sudo pip install --upgrade requests
>>Collecting requests
>>  Downloading requests-2.13.0-py2.py3-none-any.whl (584kB)
>>    100% || 585kB 642kB/s 
>>Installing collected packages: requests
>>  Found existing installation: requests 2.12.4
>>    Uninstalling requests-2.12.4:
>>      Successfully uninstalled requests-2.12.4
>>Successfully installed requests-2.10.0

As noted in previous comment tweepy error was:

>>Traceback (most recent call last):
>>  File "./retweet.py", line 3, in <module>
>>    import tweepy
>>  File "build/bdist.linux-x86_64/egg/tweepy/__init__.py", line 14, in <module>
>>  File "build/bdist.linux-x86_64/egg/tweepy/api.py", line 12, in <module>
>>  File "build/bdist.linux-x86_64/egg/tweepy/binder.py", line 11, in <module>
>>  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 64, in <module>
>>    from . import utils
>>  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 26, in <module>
>>    from .compat import (
>>ImportError: cannot import name integer_types

Comment 21 Jeremy Cline 2017-03-08 14:01:07 UTC
Okay. Just so you know though, upstream requests bundles several libraries (which we unbundle at the moment), so if you ever 'sudo pip install requests', you won't be able to receive updates to requests from Fedora and other odd things might happen.

There's work being done to make using ``sudo pip`` safe to do, but it's not done yet so at the moment, pip and dnf step on each others toes.

Comment 22 davidshumway 2017-03-08 14:21:02 UTC
Okay. So just did:

>>$sudo pip uninstall requests
>>$sudo dnf reinstall python2-requests tweepy

And tweepy works again.