Bug 1710576

Summary: /etc/cron.hourly/0yum-hourly.cron crashes - unhashable type: 'dict'
Product: [Fedora] Fedora Reporter: Tom Chiverton <bugzilla.redhat.com>
Component: python-urlgrabberAssignee: Michal Domonkos <mdomonko>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: dmach, mdomonko, ngompa13, packaging-team-maint, sheltren
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-urlgrabber-4.0.0-3.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-06 01:05:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tom Chiverton 2019-05-15 19:42:17 UTC
Description of problem:
yum-hourly cron task fails

Version-Release number of selected component (if applicable):

# rpm -qf /etc/yum/yum-cron-hourly.conf
yum-cron-3.4.3-521.fc30.noarch
# rpm -qa urlgrabber
python2-urlgrabber-4.0.0-1.fc30.noarch

How reproducible:
every time 

Steps to Reproduce:
1. run /etc/cron.hourly/0yum-hourly.cron

Actual results:
crash

Expected results:
no crash

Additional info:

# /etc/cron.hourly/0yum-hourly.cron 
Traceback (most recent call last):
  File "/usr/sbin/yum-cron", line 713, in <module>
    main()
  File "/usr/sbin/yum-cron", line 710, in main
    base.updatesCheck()
  File "/usr/sbin/yum-cron", line 602, in updatesCheck
    self.populateUpdateMetadata()
  File "/usr/sbin/yum-cron", line 409, in populateUpdateMetadata
    self.pkgSack # honor skip_if_unavailable
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1095, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 799, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 344, in populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
    self.retrieveAllMD()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
    dl = repo._async and repo._commonLoadRepoXML(repo)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1460, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1234, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1014, in _getFile
    result = self.grab.urlgrab(misc.to_utf8(relative), local,
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 698, in <lambda>
    grab = property(lambda self: self._getgrab())
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 693, in _getgrab
    self._setupGrab()
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 659, in _setupGrab
    failure_callback=mirror_failure)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 288, in __init__
    self.mirrors.sort(key=estimate, reverse=True)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 282, in estimate
    speed, fail = _TH.estimate(m['mirror'])
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2547, in estimate
    host = urlparse.urlsplit(baseurl).netloc.split(b'@')[-1] or baseurl
  File "/usr/lib64/python2.7/urlparse.py", line 176, in urlsplit
    cached = _parse_cache.get(key, None)
TypeError: unhashable type: 'dict'

Comment 1 Tom Chiverton 2019-05-16 17:10:30 UTC
Found an upstream bug (linked), closed sometime ago saying they don't care.
so fedora should drop urlgrabber? Means rewriting core yum features..

Comment 2 Michal Domonkos 2019-05-17 08:15:30 UTC
Fedora actually *is* dropping yum (the original implementation, version 3) starting with F31.

That said, in F30, we still have yum-3, so this should really be fixed.  I'll take that on.

Comment 3 Michal Domonkos 2019-05-20 13:55:18 UTC
Fixed upstream: https://github.com/rpm-software-management/urlgrabber/pull/13

Comment 4 Fedora Update System 2019-05-21 16:30:18 UTC
python-urlgrabber-4.0.0-3.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-372212061d

Comment 5 Fedora Update System 2019-05-22 01:44:20 UTC
python-urlgrabber-4.0.0-3.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-372212061d

Comment 6 Tom Chiverton 2019-05-23 20:52:48 UTC
Sorted.

Comment 7 Tom Chiverton 2019-05-27 14:40:35 UTC
However, dnf upgrade now reports issues :

[root@bookcase boinc]# dnf upgrade
ROCm                                                                                            16 kB/s | 2.9 kB     00:00    
Fedora Modular 30 - x86_64                                                                      14 kB/s |  24 kB     00:01    
Fedora Modular 30 - x86_64 - Updates                                                            45 kB/s |  23 kB     00:00    
Fedora 30 - x86_64 - Updates                                                                    47 kB/s |  24 kB     00:00    
Fedora 30 - x86_64                                                                              29 kB/s |  24 kB     00:00    
RPM Fusion for Fedora 30 - Free - Updates                                                       57 kB/s |  10 kB     00:00    
RPM Fusion for Fedora 30 - Free                                                                 24 kB/s |  10 kB     00:00    
Dependencies resolved.

 Problem: package fuse-2.9.9-3.fc30.x86_64 requires fuse-common = 3.4.2, but none of the providers can be installed
  - cannot install both fuse-common-3.5.0-1.fc30.x86_64 and fuse-common-3.4.2-6.fc30.x86_64
  - cannot install both fuse-common-3.4.2-3.fc30.x86_64 and fuse-common-3.5.0-1.fc30.x86_64
  - cannot install the best update candidate for package fuse-common-3.4.2-6.fc30.x86_64
  - cannot install the best update candidate for package fuse-2.9.9-3.fc30.x86_64
===============================================================================================================================
 Package                         Architecture               Version                          Repository                   Size
===============================================================================================================================
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 fuse-common                     x86_64                     3.4.2-3.fc30                     fedora                       19 k
 fuse-common                     x86_64                     3.5.0-1.fc30                     updates                     7.4 k

Transaction Summary
===============================================================================================================================
Skip  2 Packages

Nothing to do.
Complete!
[root@bookcase boinc]# dnf upgrade --best --allowerasing
Last metadata expiration check: 0:00:30 ago on Mon 27 May 2019 15:36:09 BST.
Error: 
 Problem: cannot install the best update candidate for package fuse-2.9.9-3.fc30.x86_64
  - problem with installed package fuse-2.9.9-3.fc30.x86_64
  - package fuse-2.9.9-3.fc30.x86_64 requires fuse-common = 3.4.2, but none of the providers can be installed
  - cannot install both fuse-common-3.5.0-1.fc30.x86_64 and fuse-common-3.4.2-6.fc30.x86_64
  - cannot install both fuse-common-3.4.2-3.fc30.x86_64 and fuse-common-3.5.0-1.fc30.x86_64
  - cannot install the best update candidate for package fuse-common-3.4.2-6.fc30.x86_64
  - problem with installed package fuse-common-3.4.2-6.fc30.x86_64
(try to add '--skip-broken' to skip uninstallable packages)
[root@bookcase boinc]# dnf upgrade --best --allowerasing --skip-broken
Last metadata expiration check: 0:01:13 ago on Mon 27 May 2019 15:36:09 BST.
Error: 
 Problem: cannot install the best update candidate for package fuse-2.9.9-3.fc30.x86_64
  - problem with installed package fuse-2.9.9-3.fc30.x86_64
  - package fuse-2.9.9-3.fc30.x86_64 requires fuse-common = 3.4.2, but none of the providers can be installed
  - cannot install both fuse-common-3.5.0-1.fc30.x86_64 and fuse-common-3.4.2-6.fc30.x86_64
  - cannot install both fuse-common-3.4.2-3.fc30.x86_64 and fuse-common-3.5.0-1.fc30.x86_64
  - cannot install the best update candidate for package fuse-common-3.4.2-6.fc30.x86_64
  - problem with installed package fuse-common-3.4.2-6.fc30.x86_64
[root@bookcase boinc]# dnf remove fuse-common
Dependencies resolved.
===============================================================================================================================
 Package                          Architecture             Version                            Repository                  Size
===============================================================================================================================
Removing:
 fuse-common                      x86_64                   3.4.2-6.fc30                       @updates                    38  
Removing dependent packages:
 borgbackup                       x86_64                   1.1.10-2.fc30                      @updates                   4.8 M
Removing unused dependencies:
 fuse                             x86_64                   2.9.9-3.fc30                       @fedora                    224 k
 python3-llfuse                   x86_64                   1.3.5-2.fc30                       @fedora                    1.5 M

Transaction Summary
===============================================================================================================================
Remove  4 Packages

Freed space: 6.5 M
Is this ok [y/N]: n
Operation aborted.
[root@bookcase boinc]#

Comment 8 Tom Chiverton 2019-05-27 14:43:46 UTC
Looks like you need the fuse package from updates-testing as well:

[root@bookcase boinc]# sudo dnf upgrade --enablerepo=updates-testing fuse
Last metadata expiration check: 0:00:50 ago on Mon 27 May 2019 15:40:57 BST.
Dependencies resolved.
===============================================================================================================================
 Package                 Architecture              Version                            Repository                          Size
===============================================================================================================================
Upgrading:
 fuse                    x86_64                    2.9.9-7.fc30                       updates-testing                     79 k

Transaction Summary
===============================================================================================================================
Upgrade  1 Package

Total download size: 79 k
Is this ok [y/N]: y
Downloading Packages:
fuse-2.9.9-7.fc30.x86_64.rpm                                                                   280 kB/s |  79 kB     00:00    
-------------------------------------------------------------------------------------------------------------------------------
Total                                                                                           86 kB/s |  79 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                       1/1 
  Upgrading        : fuse-2.9.9-7.fc30.x86_64                                                                              1/2 
update
  Cleanup          : fuse-2.9.9-3.fc30.x86_64                                                                              2/2 
  Running scriptlet: fuse-2.9.9-3.fc30.x86_64                                                                              2/2 
  Verifying        : fuse-2.9.9-7.fc30.x86_64                                                                              1/2 
  Verifying        : fuse-2.9.9-3.fc30.x86_64                                                                              2/2 

Upgraded:
  fuse-2.9.9-7.fc30.x86_64                                                                                                     

Complete!
[root@bookcase boinc]# dnf update
Last metadata expiration check: 0:06:13 ago on Mon 27 May 2019 15:36:09 BST.
Dependencies resolved.
===============================================================================================================================
 Package                         Architecture               Version                          Repository                   Size
===============================================================================================================================
Upgrading:
 fuse-common                     x86_64                     3.5.0-1.fc30                     updates                     7.4 k

Transaction Summary
===============================================================================================================================
Upgrade  1 Package

Total download size: 7.4 k
Is this ok [y/N]: y
Downloading Packages:
fuse-common-3.4.2-6.fc30_3.5.0-1.fc30.x86_64.drpm                                               26 kB/s | 4.6 kB     00:00    
[DRPM] fuse-common-3.4.2-6.fc30_3.5.0-1.fc30.x86_64.drpm: done                                                                
-------------------------------------------------------------------------------------------------------------------------------
Total                                                                                          8.7 kB/s | 4.6 kB     00:00     
Delta RPMs reduced 0.0 MB of updates to 0.0 MB (38.1% saved)
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                       1/1 
  Upgrading        : fuse-common-3.5.0-1.fc30.x86_64                                                                       1/2 
  Cleanup          : fuse-common-3.4.2-6.fc30.x86_64                                                                       2/2 
  Verifying        : fuse-common-3.5.0-1.fc30.x86_64                                                                       1/2 
  Verifying        : fuse-common-3.4.2-6.fc30.x86_64                                                                       2/2 

Upgraded:
  fuse-common-3.5.0-1.fc30.x86_64                                                                                              

Complete!
[root@bookcase boinc]# dnf update
Last metadata expiration check: 0:06:43 ago on Mon 27 May 2019 15:36:09 BST.
Dependencies resolved.
Nothing to do.
Complete!
[root@bookcase boinc]#

Comment 9 Michal Domonkos 2019-05-27 15:06:57 UTC
Please note that DNF does not use python-urlgrabber at all.

Comment 10 Ben Cotton 2019-05-28 18:55:09 UTC
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 11 Fedora Update System 2019-06-06 01:05:58 UTC
python-urlgrabber-4.0.0-3.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.