Description of problem: Offline updates fail immediately with "cannot download xxx.rpm" error. This happened to me twice in two subsequent weeks (I update once a week). This is what I do: 1. I open Software, look at Updates tab. 2. Just to make sure I have the latest updates, I click the Refresh button. 3. I wait until all the metadata and new packages are downloaded. 4. I select Reboot & install updates. 5. PC reboots. In the offline update environment, immediately after initialization I see an error like this: failed to update system: cannot download a/autocorr-cs-4.3.6.2-6.fc21.noarch.rpm to /var/cache/PackageKit/metadata/updates/packages/: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f21&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org] 6. PC reboots back into GNOME. Software tells me that offline update failed. 7. If I go to Software -> Updates and again hit Reboot & install updates, the same story repeats itself. 8. If I check /var/cache/PackageKit and look for the downloaded rpms, there are *none*, in any subdirectory. 9. I go to Software -> Updates, hit Refresh button. All of those packages are downloaded again. They are clearly not cached anywhere, but downloaded again, because it takes a long time (and I can see it in pkmon). 10. I hit Reboot & install updates, and this time the offline update proceeds correctly. The first time I thought it's some kind of repository fluke, because the subsequent force & download & offline update worked. But it happened again, a week later. There's something wrong in there. I'm not sure how exactly to reproduce it. My assumption is that gnome-software or packagekit sometimes don't save the files into a correct directory, or perhaps remove them immediately afterwards. But I haven't really checked this before doing the initial offline update, only after the offline update failed. And after it failed, the directories in /var/cache/PackageKit were empty (no rpms). Version-Release number of selected component (if applicable): PackageKit-1.0.4-1.fc21.x86_64 libsolv-0.6.4-3.fc21.x86_64 libhif-0.1.8-4.fc21.x86_64 rpm-4.12.0.1-5.fc21.x86_64 librepo-1.7.13-1.fc21.x86_64 hawkey-0.5.3-2.fc21.x86_64
Created attachment 1010632 [details] journal from failed offline update This important snippet is: Apr 03 10:06:29 kraken pk-offline-update[551]: percentage 10% Apr 03 10:06:30 kraken pk-offline-update[551]: percentage 14% Apr 03 10:06:30 kraken pk-offline-update[551]: sent msg to plymouth 'Installing Updates - 14%' Apr 03 10:06:30 kraken pk-offline-update[551]: status download Apr 03 10:06:30 kraken pk-offline-update[551]: package downloading autocorr-cs-1:4.3.6.2-6.fc21.noarch (updates) Apr 03 10:06:30 kraken pk-offline-update[551]: status finished Apr 03 10:06:30 kraken PackageKit[563]: update-packages transaction /1034_dbadacdb from uid 0 finished with failed after 6177ms Apr 03 10:06:30 kraken pk-offline-update[551]: writing failed results Apr 03 10:06:30 kraken pk-offline-update[551]: failed to update system: cannot download a/autocorr-cs-4.3.6.2-6.fc21.noarch.rpm to /var/cache/PackageKit/metadata/updates/packages/: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f21&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org] Apr 03 10:06:40 kraken pk-offline-update[551]: rebooting
Created attachment 1010633 [details] packagekit transactions The first transaction is when I forced refresh in Software, the second transaction is the offline update.
Created attachment 1010635 [details] filesystem printout after failed offline update This is how /var/cache/PackageKit/ looked like after the failed offline update. No RPMs anywhere. But /var/lib/PackageKit/prepared-update correctly contained all of their names and versions.
I might have finally some good info to debug this, because I've had packagekit running in verbose mode in the last week. Two days ago, this problem happened again - PackageKit was trying to update packages which were missing on the disk. The error message is no longer "cannot download foo" but it is "failed to update system: Failed to find foo", but the problem seems to be the same: There are certain times when gnome-software announces "updates are available", but the files are missing from the disk. Maybe it's a race condition. On Oct 10, I tried to perform offline upgrade and I received this during the upgrade process: Oct 10 09:57:58 medusa pk-offline-update[545]: percentage 9% Oct 10 09:58:00 medusa packagekitd[588]: 09:58:00 PackageKit emitting error-code package-not-found, 'Failed to find NetworkManager;1:1.0.6-6.fc22;x86_64;updates' Oct 10 09:58:00 medusa packagekitd[588]: 09:58:00 PackageKit backend was running for 2272 ms Oct 10 09:58:00 medusa PackageKit[588]: update-packages transaction /264_acecabaa from uid 0 finished with failed after 2272ms Oct 10 09:58:00 medusa pk-offline-update[545]: percentage 10% Oct 10 09:58:00 medusa packagekitd[588]: 09:58:00 PackageKit emitting finished 'failed', 2272 Oct 10 09:58:00 medusa packagekitd[588]: 09:58:00 PackageKit transaction now finished Oct 10 09:58:00 medusa packagekitd[588]: 09:58:00 PackageKit 1 transactions in list, 0 committed but not finished Oct 10 09:58:00 medusa pk-offline-update[545]: status finished Oct 10 09:58:00 medusa pk-offline-update[545]: writing failed results Oct 10 09:58:00 medusa pk-offline-update[545]: failed to update system: Failed to find NetworkManager;1:1.0.6-6.fc22;x86_64;updates I did not mess with dnf or any other packager before doing this, this seems to be solely PackageKit's fault. The prepared-update file includes "NetworkManager;1:1.0.6-6.fc22;x86_64;updates" (with 208 other updates), but on the disk there are just 4: /var/cache/PackageKit/metadata/fedora/packages /var/cache/PackageKit/metadata/fedora/packages/libconfig-1.4.9-7.fc22.x86_64.rpm /var/cache/PackageKit/metadata/fedora/packages/libffado-2.2.1-3.fc22.x86_64.rpm /var/cache/PackageKit/metadata/fedora/packages/libxml++-2.38.0-1.fc22.x86_64.rpm /var/cache/PackageKit/metadata/fedora/packages/recordmydesktop-0.3.8.1-14.fc22.x86_64.rpm Where are the rest of the RPMs? When I search the packagekit journal for "1.0.6-6.fc22", I see this: Oct 05 19:58:27 medusa packagekitd[933]: 19:58:27 PackageKit emit package available, NetworkManager;1:1.0.6-6.fc22;x86_64;updates, Network connection manager and user applications ... Oct 05 19:58:28 medusa packagekitd[933]: 19:58:28 PackageKit UpdatePackages method called: NetworkManager;1:1.0.6-6.fc22;x86_64;updates&NetworkManager-libnm;1:1.0.6-6.fc22;x86_64;updates&NetworkManager... ... Oct 05 19:58:30 medusa packagekitd[933]: 19:58:30 Hif checking if /var/cache/PackageKit/metadata/updates/packages/NetworkManager-1.0.6-6.fc22.x86_64.rpm already exists... ... Oct 05 19:59:06 medusa packagekitd[933]: 19:59:06 Hif downloading n/NetworkManager-1.0.6-6.fc22.x86_64.rpm to /var/cache/PackageKit/metadata/updates/packages/ Oct 05 19:59:06 medusa packagekitd[933]: 19:59:06 PackageKit-Hif got state download with hint NetworkManager;1:1.0.6-6.fc22;x86_64;updates Oct 05 19:59:06 medusa packagekitd[933]: 19:59:06 PackageKit emit package downloading, NetworkManager;1:1.0.6-6.fc22;x86_64;updates, ... Oct 06 20:19:29 medusa packagekitd[933]: 20:19:29 PackageKit UpdatePackages method called: NetworkManager;1:1.0.6-6.fc22;x86_64;updates&NetworkManager-libnm;1:1.0.6-6.fc22;x86_64;updates&NetworkManager... ... Oct 06 20:21:22 medusa packagekitd[933]: 20:21:22 PackageKit setting role for /259_beccccac to refresh-cache Oct 06 20:21:22 medusa packagekitd[933]: 20:21:22 PackageKit-Hif Deleting contents of updates as forced ... Oct 06 20:21:22 medusa packagekitd[933]: 20:21:22 Hif deleting file /var/cache/PackageKit/metadata/updates/packages/NetworkManager-1.0.6-6.fc22.x86_64.rpm ################### <--- OH WOW WHAT? ... Oct 06 20:22:10 medusa packagekitd[933]: 20:22:10 PackageKit emit package available, NetworkManager;1:1.0.6-6.fc22;x86_64;updates, Network connection manager and user applications ... Oct 06 20:22:10 medusa packagekitd[933]: 20:22:10 PackageKit UpdatePackages method called: NetworkManager;1:1.0.6-6.fc22;x86_64;updates&NetworkManager-libnm;1:1.0.6-6.fc22;x86_64;updates&NetworkManager... ... Oct 06 20:22:13 medusa packagekitd[933]: 20:22:13 Hif checking if /var/cache/PackageKit/metadata/updates/packages/NetworkManager-1.0.6-6.fc22.x86_64.rpm already exists... ... Oct 06 20:22:50 medusa packagekitd[933]: 20:22:50 Hif downloading n/NetworkManager-1.0.6-6.fc22.x86_64.rpm to /var/cache/PackageKit/metadata/updates/packages/ Oct 06 20:22:50 medusa packagekitd[933]: 20:22:50 PackageKit-Hif got state download with hint NetworkManager;1:1.0.6-6.fc22;x86_64;updates Oct 06 20:22:50 medusa packagekitd[933]: 20:22:50 PackageKit emit package downloading, NetworkManager;1:1.0.6-6.fc22;x86_64;updates, ... Oct 10 09:50:35 medusa packagekitd[933]: 09:50:35 Hif deleting file /var/cache/PackageKit/metadata/updates/packages/NetworkManager-glib-1.0.6-6.fc22.x86_64.rpm ################# <--- AND AGAIN ... Oct 10 09:57:58 medusa packagekitd[588]: 09:57:58 PackageKit UpdatePackages method called: NetworkManager;1:1.0.6-6.fc22;x86_64;updates&NetworkManager-libnm;1:1.0.6-6.fc22;x86_64;updates&NetworkManager... ... Oct 10 09:58:00 medusa packagekitd[588]: 09:58:00 PackageKit emitting error-code package-not-found, 'Failed to find NetworkManager;1:1.0.6-6.fc22;x86_64;updates' ... Oct 10 10:00:04 medusa packagekitd[875]: 10:00:04 PackageKit GetUpdateDetail method called: NetworkManager;1:1.0.6-6.fc22;x86_64;updates&NetworkManager-libnm;1:1.0.6-6.fc22;x86_64;updates&NetworkManager... It seems that PackageKit is deleting its own files for some reason, but that doesn't invalidate the whole "system updates are available" notification. It also doesn't invalidate the updates view if gnome-software, and it doesn't invalidate "install updates on reboot/poweroff" checkbox. But if user does that, of course if fails to find the necessary RPM files. I don't know if it is related, but I have noticed several times that when some downloading error occurs during forced update refresh, the *whole* set is downloaded again, even updates that were already downloaded before. Sometimes maybe even an error doesn't need to occur, you just hit the "refresh" button and everything seems to be re-downloading again. I wonder if it is somehow related to this bug. PackageKit-1.0.8-2.fc22.x86_64 libhif-0.2.0-5.fc22.x86_64 dnf-1.1.1-2.fc22.noarch gnome-software-3.16.5-1.fc22.x86_64 gnome-shell-3.16.3-1.fc22.x86_64 hawkey-0.6.0-1.fc22.x86_64 librepo-1.7.16-1.fc22.x86_64
Created attachment 1082092 [details] gnome-software offering updates This shows that updates are displayed, with icons and descriptions and everything. They seem to be ready.
Created attachment 1082093 [details] prepared-update file prepared-update file (retrieved after offline upgrade failed)
Created attachment 1082094 [details] filesystem printout files available on disk (retrieved after offline upgrade failed)
Created attachment 1082096 [details] journal from failed offline update
Created attachment 1082097 [details] packagekit journal This packagekit journal is with verbose mode and it spans several days before the offline update was triggered. The updates were several times downloaded and deleted. I'm not sure whether the "updates ready" notification was there for the whole time. But when I clicked it on Oct 10, the RPM files were missing from the disk.
Created attachment 1082098 [details] list of installed packages (rpm -qa)
I think I might have tracked this down: commit 2bdc001f784ade91fd5fd2823e5ddc4f776fc89e Author: Richard Hughes <richard> Date: Tue Oct 13 10:30:34 2015 +0100 Do not force the cache refresh If we're only installing updates once per week it doesn't make sense to delete the already-downloaded packages if refresh-cache is on the same schedule. Perhaps fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1208861 It can't explain a lot of the symptoms, but it's a step in the right direction never-the-less.
Richard, any updates regarding this? I see the commit in 3.16 branch only, is that intentional? For future reference, I attach a record of our IRC conversation. <hughsie> kparal, i think you're right, it's when the update schedule and refresh schedule happen to overlap in the wrong way <hughsie> kparal, the downsides of my patch is that the size of the PK cache could balloon if you never actually use PK for updates <hughsie> i.e. if PK autodownloads them, then you run dnf update in the meantime <hughsie> i guess the fix there is to look for older versions of the same package name in the cache when downloading updates <kparal> if we keep deleting old caches, can't we at least emit a signal to stop announcing and displaying new updates in gnome-software/gnome-shell, then delete the cache, download anew, and then announce again? <hughsie> kparal, i'm just trying some test code <kparal> also, couldn't you first get a list of available updates, and then only delete those which are no longer available (e.g. either unpushed or already installed)? that would solve all problems - files would not get missing, and the cache would not grow indefinitely if people use only dnf <kparal> also it solves concurrency issues - as long as you download new updates first, it does not matter if people trigger offline update in the meantime. once all new updates are downloaded, you refresh gnome-software/gnome-shell, and then delete old files <hughsie> i think that's more fragile tbh <kparal> should not have any race condition this way I still think that the proposed solution is valid (slight adjustments made): 1. if PK founds out there are new updates waiting, emit "cancel updates" signal, which should remove an "updates ready" notification and the checkbox in poweroff window 2. get a list of to-be-updated rpm files 3. remove everything in PK cache except for to-be-updated rpm files (could have been downloaded earlier, so this will save re-downloading - bandwidth and time; will also prune the dir from outdated rpm files) 4. download to-be-updated rpm files (skip those already present, ideally including checking their rpm checksum to make sure they're correct) 5. emit "new updates ready" signal, which should trigger a notification and add the checkbox in poweroff window
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
I think the improvements proposed in comment 12 are still valid and not yet implemented.
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle. Changing version to '25'.
This message is a reminder that Fedora 25 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '25'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 25 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.