Bug 1181501

Summary: offline update failed in an unexpected way - Failed to find curl;7.37.0-12.fc21;x86_64;updates
Product: [Fedora] Fedora Reporter: Tim Waugh <twaugh>
Component: libhifAssignee: Richard Hughes <rhughes>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: awilliam, jonathan, kalevlember, kparal, mattdm, mcatanzaro+wrong-account-do-not-cc, michael, miminar, pschindl, rdieter, rhughes, sergio, sgallagh, smparrish, wdc
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libhif-0.1.8-4.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-20 21:03:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tim Waugh 2015-01-13 10:24:36 UTC
Description of problem:
Since a few days ago (Jan 9th) offline updates have been consistently failing. Instead I've been applying updates with dnf.

After attempting an offline update by restarting, on next login I get this message:

==
Failed To Update

The offline update failed in an unexpected way.
Detailed errors from the package manager follow:

Failed to find curl;7.37.0-12.fc21;x86_64;updates
==

However, before restarting I had taken the precaution of noting the contents of /var/cache/PackageKit/metadata/updates/packages/:

total 7580
-rw-r--r--. 1 root root  276420 Jan 12 23:05 curl-7.37.0-12.fc21.x86_64.rpm
-rw-r--r--. 1 root root  360360 Jan 12 23:05 dbus-1.8.14-1.fc21.x86_64.rpm
-rw-r--r--. 1 root root   58520 Jan 12 23:05 dbus-devel-1.8.14-1.fc21.x86_64.rpm
-rw-r--r--. 1 root root  174388 Jan 12 23:05 dbus-libs-1.8.14-1.fc21.i686.rpm
-rw-r--r--. 1 root root  170936 Jan 12 23:05 dbus-libs-1.8.14-1.fc21.x86_64.rpm
-rw-r--r--. 1 root root   51816 Jan 12 23:05 dbus-x11-1.8.14-1.fc21.x86_64.rpm
-rw-r--r--. 1 root root  246864 Jan 12 23:05 libcurl-7.37.0-12.fc21.x86_64.rpm
-rw-r--r--. 1 root root  121280 Jan 12 23:05 libsmbclient-4.1.14-2.fc21.x86_64.rpm
-rw-r--r--. 1 root root   89468 Jan 12 23:05 libwbclient-4.1.14-2.fc21.x86_64.rpm
-rw-r--r--. 1 root root  527068 Jan 12 23:05 samba-client-4.1.14-2.fc21.x86_64.rpm
-rw-r--r--. 1 root root  710756 Jan 12 23:05 samba-common-4.1.14-2.fc21.x86_64.rpm
-rw-r--r--. 1 root root 4467684 Jan 12 23:05 samba-libs-4.1.14-2.fc21.x86_64.rpm
-rw-r--r--. 1 root root  367268 Jan 12 23:06 spice-glib-0.27-3.fc21.x86_64.rpm
-rw-r--r--. 1 root root   58116 Jan 12 23:06 spice-gtk3-0.27-3.fc21.x86_64.rpm

After update that directory no longer exists.

Version-Release number of selected component (if applicable):
PackageKit-1.0.3-4.fc21.x86_64

How reproducible:
100% - happens every time for me.

Comment 1 Kamil Páral 2015-01-15 12:55:00 UTC
*** Bug 1182156 has been marked as a duplicate of this bug. ***

Comment 2 Kamil Páral 2015-01-15 13:19:28 UTC
See bug 1182156 to see some logs.

By trial and error, I've found out that the problem is in libhif. With libhif-0.1.6-2.fc21.x86_64 offline upgrades work for me without any problem. With libhif-0.1.7-1.fc21.x86_64, I always get "failed to find" error.

Reproducer:
1. Install F21 Workstation from Live.
2. Do offline upgrade to latest packages, everything works.
3. yum downgrade devassistant (or something else)
4. try to do offline upgrade, it fails
5. yum downgrade libhif
6. reboot (just to be sure)
7. try to do offline upgrade, it succeeds
8. repeat steps 3-7 until you're sure it's not an accident

Comment 3 Kamil Páral 2015-01-15 13:27:32 UTC
libhif-0.1.7-1.fc21.x86_64 got submitted stable on Jan 6. And I'm afraid *everyone* who updated since Jan 6 has now their offline upgrades broken. And you won't get a fixed version, since... their update system is broken. They'll need to use yum/dnf. I'd love be be wrong.

Comment 4 Petr Schindler 2015-01-15 13:46:30 UTC
Yep, it's broken.

I tried to update both with libhif 0.1.6.2 and 0.1.7.1 and it is as Kamil said. With 0.1.7 offline updates don't work. 0.1.6 is ok.

Comment 5 Stephen Gallagher 2015-01-15 14:17:18 UTC
Adding another datapoint, I can also confirm this bug.

So, we have a pretty serious problem here for anyone that has already hit this bug.

I'd like to recommend that if the fix is not yet known, we should ask rel-eng to untag this build from the stable repository until it can be fixed (so that anyone not already impacted by it won't end up broken).

As for those who are already stuck, they'll likely need to force a yum/dnf downgrade for the time being.

Comment 6 Richard Hughes 2015-01-15 20:10:42 UTC
I've spent the last ~7 hours debugging this, and this is the core issue: https://github.com/Tojaj/librepo/issues/41 -- it's a huge pain it slipped through all my own testing and the updates QA process. Frustratingly, the libhif self tests passed, but because they are run on a machine with network access. :/ For the future, I'll enable them on koji and also run them locally with 'unshare -n' to remove any connectivity.

I've worked around this upstream with https://github.com/hughsie/libhif/commit/359b609fc37e84063b125a88b3ed97b6f0ea4beb and I'm building new packages for F22 and F21 now. I guess the only thing I can do is apologise, but it seems like anyone who installed libhif-0.1.7-1 needs to update to libhif-0.1.7-3 using either dnf, yum or gnome-packagekit.

Once you've installed the new libhif packages you'll either need to reboot or do:

gdbus call --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit --method org.freedesktop.PackageKit.SuggestDaemonQuit 

and then verify that the packagekitd process has quit.

Comment 7 Richard Hughes 2015-01-15 20:23:28 UTC
(In reply to Richard Hughes from comment #6)
> anyone who installed libhif-0.1.7-1 needs to update to libhif-0.1.7-3
> using either dnf, yum or gnome-packagekit.

Actually, it's slightly better than this as you can use gnome-software just by clicking on the http:// link to the rpm. It actually works surprisingly well.

Comment 8 Fedora Update System 2015-01-15 20:27:44 UTC
libhif-0.1.7-3.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libhif-0.1.7-3.fc21

Comment 9 Fedora Update System 2015-01-17 05:42:05 UTC
Package libhif-0.1.7-3.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libhif-0.1.7-3.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-0771/libhif-0.1.7-3.fc21
then log in and leave karma (feedback).

Comment 10 Fedora Update System 2015-01-19 19:08:03 UTC
libhif-0.1.8-1.fc21, PackageKit-1.0.4-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/PackageKit-1.0.4-1.fc21,libhif-0.1.8-1.fc21

Comment 11 Fedora Update System 2015-01-20 21:03:07 UTC
libhif-0.1.8-1.fc21, PackageKit-1.0.4-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 12 Adam Williamson 2015-01-20 23:10:16 UTC
For reference of anyone with broken offline updates, you can install the update with:

su -c 'yum update --advisory=FEDORA-2015-0921'

or:

sudo yum update --advisory=FEDORA-2015-0921

At present the update is syncing to mirrors so it may not be immediately available; if the above doesn't install anything, wait a few hours and try again. Sorry again for the trouble.

Comment 13 Fedora Update System 2015-02-03 15:55:17 UTC
libhif-0.1.8-3.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libhif-0.1.8-3.fc21

Comment 14 Fedora Update System 2015-02-05 05:24:28 UTC
libhif-0.1.8-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 15 Sergio Basto 2015-02-09 01:31:44 UTC
I have  libhif updated  I reboot the system but seems to me still not working 

pkcon offline-status

   Command failed: no update results available

Comment 16 Kamil Páral 2015-02-09 09:36:41 UTC
Sergio, that command should give you the outcome of the last performed offline upgrade process. If you want to actually perform it, see `man pkcon` to find out how to trigger it (I'm not sure at the moment), or fire up gnome-software, go to Updates, and hit the refresh button. It should download the updates and ask you to reboot to perform the offline upgrade. On the next boot, you can use `pkcon offline-status` to see the outcome.

Please also note that there were multiple update breakages lately, so you might need to update everything related to PackageKit, libhif and librepo manually using yum first.

Comment 17 Sergio Basto 2015-02-17 18:15:39 UTC
Hi , today after do: 

pkcon -c -1 refresh force


The updates appears on my systray !  it works huf !