Bug 1276886 - Downloaded packages are deleted too quickly, leading to frustration
Downloaded packages are deleted too quickly, leading to frustration
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: dnf-plugin-system-upgrade (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Will Woods
Fedora Extras Quality Assurance
https://github.com/rpm-software-manag...
https://fedoraproject.org/wiki/Common...
: CommonBugs
: 1279071 1281305 1283601 1283818 1287570 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-31 19:36 EDT by Mads Kiilerich
Modified: 2016-07-01 11:16 EDT (History)
15 users (show)

See Also:
Fixed In Version: dnf-1.0.2-3.fc22
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-01 11:16:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mads Kiilerich 2015-10-31 19:36:06 EDT
I ran
dnf system-upgrade --releasever 23 download
and it downloaded 3 GB data ... and then transaction test failed because conflict with a custom package.
After fixing that and running the command again, it downloads everything again.
(I think I have seen the same when lacking disk space.)
I would expect it to keep the downloaded files after these minor failures and reuse them.

dnf-1.1.3-1.fc22.noarch
dnf-plugin-system-upgrade-0.4.1-1.fc22.noarch
Comment 1 Peter van Hooft 2015-11-03 16:20:49 EST
Confirm - same release of dnf and dnf-plugin-system-upgrade. In my case it downloaded 8GB and restarted downloading everything again after a conflict.
Comment 2 Zbigniew Jędrzejewski-Szmek 2015-11-03 17:01:58 EST
We're working on it ;)
Comment 3 clockskew 2015-11-04 01:28:03 EST
I can also confirm this failure while using dnf system-upgrade from a fully updated Fed 22, and attempting a Fed 23 upgrade.

In my case, I was forced to re-download 4.1 Gb following what I thought was a minor package conflict.

This behavior is frustrating (to be unbelievably kind), specifically when the package error throws no useful information concerning a resolution.

My larger questions are why was this new upgrade tool released without warning or discussion on Fedora's website - and why was it released without sufficient testing?

I also note Fedora's use of this unintelligible directive, theoretically in 'English':

"3.  Download the updated packages. There are two options which it is usually a good idea to set:

    sudo dnf system-upgrade download --releasever=23 [--distro-sync] [--best] [--allowerasing] "

That was a direct quote from:

https://fedoraproject.org/wiki/DNF_system_upgrade

as displayed immediately following the Fed 23.iso release avaliability on:

https://getfedora.org/

System-Upgrade may turn out to an excellent tool following several more iterations, but it does not currently appear ready for the user base, at large.
Comment 4 clockskew 2015-11-04 04:43:26 EST
Adding balance, completeness, and perspective:

The forced double-download issue I mentioned above occurred while trying to upgrade a Fed 22 Workstation.  It is also important to note that despite the considerable wasted time, that Workstation upgrade did ultimately succeed.

I have now also had the the chance to upgrade a different Fed 22 system using:

dnf system-upgrade download --releasever=23

When I upgraded my fully updated Fed 22 Server to Fed 23 Server, the entire upgrade initiated, proceeded, and successfully completed, without issue, on the first try.

While it is clear my Server upgrade involved less packages (~1000) than my Workstation (~3000), I'd like to emphasize that 'dnf system-upgrade' can indeed work as I am sure the developers intended.

Perhaps, the list of causative 'gotchas' leading to initial upgrade difficulties, or failures, will be better understood, and published, in the coming days.
Comment 5 clockskew 2015-11-05 00:32:33 EST
Zbigniew,

Thank you for providing more insight into the upgrade challenges, and objectives.

Noted, and I agree with your input re: 1277965.

In the interim since my last post, another VBoxxed F23-Beta (Workstation) to F23 upgrade proceeded and completed, as expected, on my end.

I am now convinced dnf system-upgrade has enormous potential and even in its current early release state, dnf system-upgrade is already a superior replacement for FedUp, which had its own (lengthy) set of issues.

Best regards - and keep improving on the excellent work already done.
Comment 6 Kamil Páral 2015-11-09 08:20:33 EST
I believe the core issue is different than discussed - the downloaded packages in /var/lib/dnf/system-update are deleted after first **successful transaction**. In theory that should be system-upgrade transaction itself, so the packages don't occupy space on a newly upgraded system. In reality, though, people tend to perform transactions between "system-upgrade download" and "system-upgrade reboot". The most common cases:

1. A depedency issue was reported not during depsolve, but during transaction check (after download). People need to to correct that, i.e. uninstall something. That wipes all downloaded packages. After they run "system-upgrade download" again, the download starts from scratch. Very annoying. This is the case of comment 0.

2. Insufficient disk space. You might encounter an error (after download) that there's no enough space for performing the upgrade. The easiest solution - uninstall something large you don't use much, e.g. a big game. Again, that wipes all downloaded packages. Grrrr.

3. Limited bandwidth in some area. You might bring your laptop to e.g. your university to download all the packages, because you have slow or pay-by-data internet connection at home. Then you want to proceed with the upgrade at home. But in the meantime, you perform some transaction, like installing or removing an application. Downloaded packages gone.


I don't know what is the easy solution here, because we don't want to make people manually clean packages after upgrade - most would not. Things like tpmfiles.d are not practical here I believe, because they can introduce race conditions.

What I can think of as practical solutions:
a) Implement a change in dnf that would hardcode a longer timeout for /var/lib/dnf/system-upgrade (let's say "first successful transaction after a week" - measured from the last-modified timestamp of that directory)

b) Allow dnf plugins to configure this timeout programatically and then honor that (e.g. a status file placed inside that directory saying how long to not wipe this dir)

c) Make a special directory for system-upgrade (do not share it with offline updates) and manage the files in the plugin - wipe that directory after a successful upgrade, but do not touch it otherwise
Comment 7 Will Woods 2015-11-09 15:02:37 EST
*** Bug 1279071 has been marked as a duplicate of this bug. ***
Comment 8 Will Woods 2015-11-12 09:58:09 EST
*** Bug 1281305 has been marked as a duplicate of this bug. ***
Comment 9 "FeRD" (Frank Dana) 2015-11-15 00:30:28 EST
(In reply to Kamil Páral from comment #6)
> I believe the core issue is different than discussed - the downloaded
> packages in /var/lib/dnf/system-update are deleted after first **successful
> transaction**. In theory that should be system-upgrade transaction itself,
> so the packages don't occupy space on a newly upgraded system. In reality,
> though, people tend to perform transactions between "system-upgrade
> download" and "system-upgrade reboot". The most common cases:
> 
> 1. A depedency issue was reported not during depsolve, but during
> transaction check (after download). People need to to correct that, i.e.
> uninstall something. That wipes all downloaded packages. After they run
> "system-upgrade download" again, the download starts from scratch. Very
> annoying. This is the case of comment 0.

Thanks for this explanation, Kamil. At the risk of "me too"-ing, I just hit the same issue running my first F22->F23 update, thanks to Google's broken google-earth-stable RPM. (It somehow owns /usr/bin, so unless it's temporarily moved out of the way every time filesystem.rpm is updated I'm treated to "file /usr/bin from install of filesystem-3.2-35.fc23.x86_64 conflicts with file from package google-earth-stable-6.0.3.2197-0.x86_64".)

What you say makes sense, though I guess I'm surprised to learn that the cache-management operates that way (which is to say, "blindly") since it seems fairly fragile.

I realize that bugzilla isn't the proper forum to hash out issues of software design, nor does anyone want a designed-by-committe dnf, so I'll try not to stray too far down that path.

> I don't know what is the easy solution here, because we don't want to make
> people manually clean packages after upgrade - most would not. Things like
> tpmfiles.d are not practical here I believe, because they can introduce race
> conditions.

It seems to me that correcting MOST 'system-upgrade download' transaction failures is going to involve one or more additional dnf transactions, which (obviously) have to succeed before the interrupted one can be reattempted. So "next successful transaction" definitely seems premature for blowing away the entire cache.

Deleting the files that were just processed in the successful transaction, sure! But nuking the system-update queue, without knowing the status of its contents? That's potentially a lot of data to download, and I'd hope dnf would err on the side of keeping it around until successfully consumed or manually cleaned. Almost every instance of system-upgrade's use should eventually reach one of those two outcomes, and that one-week "blind" expiration you mentioned, perhaps, could also to cover the remaining few. 

> c) Make a special directory for system-upgrade (do not share it with offline
> updates) and manage the files in the plugin - wipe that directory after a
> successful upgrade, but do not touch it otherwise

Especially since (as you say here) it's stored in /var/lib/dnf, not under /var/cache? So the download is already explicitly not being treated as a cache.

I don't even know enough about dnf's offline updates feature to do anything more than embarrass myself trying to discuss it intelligently, but... I can't imagine there's any advantage to the directory being shared with system-upgrade. They'll never download any of the same RPMs, since they'll be pulling from different releasevers, correct? So, there's no efficiency gained in sharing space, and if separating them allows for more appropriate management and disposition of the two filesets, then that seems like a win.

Anyway, I for one appreciate the update on this issue, and hope that we'll see it sorted out for Fedora 24. In the meantime, I'm just going to slap a Post-It on my monitor reminding me that if I hit any 'dnf system-upgrade' transaction failures while upgrading my other machines, I should only use raw /usr/bin/rpm to deal with them.

(I also sort of feel like a warning to similar effect should be included in "3.2.4 System Upgrades with DNF" at):
https://docs.fedoraproject.org/en-US/Fedora/23/html/Release_Notes/sect-Release_Notes-Changes_for_Sysadmin.html

Right now, the procedure just reads:

=[...]==============================
4. Prepare the upgrade enviroment.
# dnf system-upgrade download --releasever 23

5. Reboot to perform the upgrade.
# dnf system-upgrade reboot

6. If an upgrade fails, clean up the download, resolve any issues, and try again.
# dnf system-upgrade clean
====================================

No mention of download step transaction failures or issues related to solving them, nor is the issue discussed on the system-upgrade wiki page at:
https://fedoraproject.org/wiki/DNF_system_upgrade#How_do_I_use_it.3F
Comment 10 Arnaud Lacombe 2015-11-15 02:07:01 EST
This might be naive, but how about a stop-gap fix involving a --keepcache command line flag, which would act as the dnf.conf keepcache option setting, but non-intrusively.

The idea would be use this flag on dnf transaction used to resolve the system-upgrade, without resorting to bare rpm(1) or modifying /etc/dnf/dnf.conf ?
Comment 11 David H. Gutteridge 2015-11-15 03:29:42 EST
Further to comment 9, I think this should really be listed on the "Common F23 bugs" page (https://fedoraproject.org/wiki/Common_F23_bugs). The unnecessary bandwidth consumption is a major issue for some people.

Separately, I don't follow why "dnf system-upgrade reboot" doesn't explicitly note it has no upgrade files to work with when that's the case. One would assume the user is expecting the function to actually do something, and it would be abnormal to have no upgrade files present. Instead we get the message "Upgrade complete! Cleaning up and rebooting..." The upgrade couldn't possibly be complete, because no action was taken.
Comment 12 Kamil Páral 2015-11-18 04:22:32 EST
(In reply to Arnaud Lacombe from comment #10)
> This might be naive, but how about a stop-gap fix involving a --keepcache
> command line flag, which would act as the dnf.conf keepcache option setting,
> but non-intrusively.

Thanks, that's a good idea. The correct working option is --setopt=keepcache=True.

(In reply to David H. Gutteridge from comment #11)
> Further to comment 9, I think this should really be listed on the "Common
> F23 bugs" page (https://fedoraproject.org/wiki/Common_F23_bugs). The
> unnecessary bandwidth consumption is a major issue for some people.

I'll put it there.

> Separately, I don't follow why "dnf system-upgrade reboot" doesn't
> explicitly note it has no upgrade files to work with when that's the case.

That would be https://github.com/rpm-software-management/dnf-plugin-system-upgrade/issues/33 .
Comment 13 Zbigniew Jędrzejewski-Szmek 2015-11-19 14:59:13 EST
*** Bug 1283601 has been marked as a duplicate of this bug. ***
Comment 14 Zbigniew Jędrzejewski-Szmek 2015-11-19 21:49:32 EST
*** Bug 1283818 has been marked as a duplicate of this bug. ***
Comment 15 Zbigniew Jędrzejewski-Szmek 2015-12-02 08:32:46 EST
*** Bug 1287570 has been marked as a duplicate of this bug. ***
Comment 16 David H. Gutteridge 2016-06-30 17:26:36 EDT
I don't know if this is fully resolved, but I had a much better experience in this context upgrading to Fedora 24. I repeatedly hit issues where the upgrade couldn't continue, but it kept all the existing packages in cache between (successful) dnf operations, so nothing had to be downloaded again.
Comment 17 Zbigniew Jędrzejewski-Szmek 2016-07-01 11:16:01 EDT
dnf changed to purge the cache only on successful installation transactions [1] in version 1.0.2. So in the common case where you uninstall a few packages by hand to allow the upgrade to proceed the expected thing happens and already downloaded packages are not deleted. We can close this, since the behaviour is significantly changed from when this report was first opened. If we need to do further fine-tuning, let's do it in a separate report.

[1] http://dnf.readthedocs.io/en/latest/release_notes.html#id11

Note You need to log in before you can comment on or make changes to this bug.