Bug 1276886

Confirm - same release of dnf and dnf-plugin-system-upgrade. In my case it downloaded 8GB and restarted downloading everything again after a conflict.

We're working on it ;)

I can also confirm this failure while using dnf system-upgrade from a fully updated Fed 22, and attempting a Fed 23 upgrade.

In my case, I was forced to re-download 4.1 Gb following what I thought was a minor package conflict.

This behavior is frustrating (to be unbelievably kind), specifically when the package error throws no useful information concerning a resolution.

My larger questions are why was this new upgrade tool released without warning or discussion on Fedora's website - and why was it released without sufficient testing?

I also note Fedora's use of this unintelligible directive, theoretically in 'English':

"3.  Download the updated packages. There are two options which it is usually a good idea to set:

    sudo dnf system-upgrade download --releasever=23 [--distro-sync] [--best] [--allowerasing] "

That was a direct quote from:

https://fedoraproject.org/wiki/DNF_system_upgrade

as displayed immediately following the Fed 23.iso release avaliability on:

https://getfedora.org/

System-Upgrade may turn out to an excellent tool following several more iterations, but it does not currently appear ready for the user base, at large.

Adding balance, completeness, and perspective:

The forced double-download issue I mentioned above occurred while trying to upgrade a Fed 22 Workstation.  It is also important to note that despite the considerable wasted time, that Workstation upgrade did ultimately succeed.

I have now also had the the chance to upgrade a different Fed 22 system using:

dnf system-upgrade download --releasever=23

When I upgraded my fully updated Fed 22 Server to Fed 23 Server, the entire upgrade initiated, proceeded, and successfully completed, without issue, on the first try.

While it is clear my Server upgrade involved less packages (~1000) than my Workstation (~3000), I'd like to emphasize that 'dnf system-upgrade' can indeed work as I am sure the developers intended.

Perhaps, the list of causative 'gotchas' leading to initial upgrade difficulties, or failures, will be better understood, and published, in the coming days.

Zbigniew,

Thank you for providing more insight into the upgrade challenges, and objectives.

Noted, and I agree with your input re: 1277965.

In the interim since my last post, another VBoxxed F23-Beta (Workstation) to F23 upgrade proceeded and completed, as expected, on my end.

I am now convinced dnf system-upgrade has enormous potential and even in its current early release state, dnf system-upgrade is already a superior replacement for FedUp, which had its own (lengthy) set of issues.

Best regards - and keep improving on the excellent work already done.

I believe the core issue is different than discussed - the downloaded packages in /var/lib/dnf/system-update are deleted after first **successful transaction**. In theory that should be system-upgrade transaction itself, so the packages don't occupy space on a newly upgraded system. In reality, though, people tend to perform transactions between "system-upgrade download" and "system-upgrade reboot". The most common cases:

1. A depedency issue was reported not during depsolve, but during transaction check (after download). People need to to correct that, i.e. uninstall something. That wipes all downloaded packages. After they run "system-upgrade download" again, the download starts from scratch. Very annoying. This is the case of comment 0.

2. Insufficient disk space. You might encounter an error (after download) that there's no enough space for performing the upgrade. The easiest solution - uninstall something large you don't use much, e.g. a big game. Again, that wipes all downloaded packages. Grrrr.

3. Limited bandwidth in some area. You might bring your laptop to e.g. your university to download all the packages, because you have slow or pay-by-data internet connection at home. Then you want to proceed with the upgrade at home. But in the meantime, you perform some transaction, like installing or removing an application. Downloaded packages gone.


I don't know what is the easy solution here, because we don't want to make people manually clean packages after upgrade - most would not. Things like tpmfiles.d are not practical here I believe, because they can introduce race conditions.

What I can think of as practical solutions:
a) Implement a change in dnf that would hardcode a longer timeout for /var/lib/dnf/system-upgrade (let's say "first successful transaction after a week" - measured from the last-modified timestamp of that directory)

b) Allow dnf plugins to configure this timeout programatically and then honor that (e.g. a status file placed inside that directory saying how long to not wipe this dir)

c) Make a special directory for system-upgrade (do not share it with offline updates) and manage the files in the plugin - wipe that directory after a successful upgrade, but do not touch it otherwise

*** Bug 1279071 has been marked as a duplicate of this bug. ***

*** Bug 1281305 has been marked as a duplicate of this bug. ***

(In reply to Kamil Páral from comment #6)
> I believe the core issue is different than discussed - the downloaded
> packages in /var/lib/dnf/system-update are deleted after first **successful
> transaction**. In theory that should be system-upgrade transaction itself,
> so the packages don't occupy space on a newly upgraded system. In reality,
> though, people tend to perform transactions between "system-upgrade
> download" and "system-upgrade reboot". The most common cases:
> 
> 1. A depedency issue was reported not during depsolve, but during
> transaction check (after download). People need to to correct that, i.e.
> uninstall something. That wipes all downloaded packages. After they run
> "system-upgrade download" again, the download starts from scratch. Very
> annoying. This is the case of comment 0.

Thanks for this explanation, Kamil. At the risk of "me too"-ing, I just hit the same issue running my first F22->F23 update, thanks to Google's broken google-earth-stable RPM. (It somehow owns /usr/bin, so unless it's temporarily moved out of the way every time filesystem.rpm is updated I'm treated to "file /usr/bin from install of filesystem-3.2-35.fc23.x86_64 conflicts with file from package google-earth-stable-6.0.3.2197-0.x86_64".)

What you say makes sense, though I guess I'm surprised to learn that the cache-management operates that way (which is to say, "blindly") since it seems fairly fragile.

I realize that bugzilla isn't the proper forum to hash out issues of software design, nor does anyone want a designed-by-committe dnf, so I'll try not to stray too far down that path.

> I don't know what is the easy solution here, because we don't want to make
> people manually clean packages after upgrade - most would not. Things like
> tpmfiles.d are not practical here I believe, because they can introduce race
> conditions.

It seems to me that correcting MOST 'system-upgrade download' transaction failures is going to involve one or more additional dnf transactions, which (obviously) have to succeed before the interrupted one can be reattempted. So "next successful transaction" definitely seems premature for blowing away the entire cache.

Deleting the files that were just processed in the successful transaction, sure! But nuking the system-update queue, without knowing the status of its contents? That's potentially a lot of data to download, and I'd hope dnf would err on the side of keeping it around until successfully consumed or manually cleaned. Almost every instance of system-upgrade's use should eventually reach one of those two outcomes, and that one-week "blind" expiration you mentioned, perhaps, could also to cover the remaining few. 

> c) Make a special directory for system-upgrade (do not share it with offline
> updates) and manage the files in the plugin - wipe that directory after a
> successful upgrade, but do not touch it otherwise

Especially since (as you say here) it's stored in /var/lib/dnf, not under /var/cache? So the download is already explicitly not being treated as a cache.

I don't even know enough about dnf's offline updates feature to do anything more than embarrass myself trying to discuss it intelligently, but... I can't imagine there's any advantage to the directory being shared with system-upgrade. They'll never download any of the same RPMs, since they'll be pulling from different releasevers, correct? So, there's no efficiency gained in sharing space, and if separating them allows for more appropriate management and disposition of the two filesets, then that seems like a win.

Anyway, I for one appreciate the update on this issue, and hope that we'll see it sorted out for Fedora 24. In the meantime, I'm just going to slap a Post-It on my monitor reminding me that if I hit any 'dnf system-upgrade' transaction failures while upgrading my other machines, I should only use raw /usr/bin/rpm to deal with them.

(I also sort of feel like a warning to similar effect should be included in "3.2.4 System Upgrades with DNF" at):
https://docs.fedoraproject.org/en-US/Fedora/23/html/Release_Notes/sect-Release_Notes-Changes_for_Sysadmin.html

Right now, the procedure just reads:

=[...]==============================
4. Prepare the upgrade enviroment.
# dnf system-upgrade download --releasever 23

5. Reboot to perform the upgrade.
# dnf system-upgrade reboot

6. If an upgrade fails, clean up the download, resolve any issues, and try again.
# dnf system-upgrade clean
====================================

No mention of download step transaction failures or issues related to solving them, nor is the issue discussed on the system-upgrade wiki page at:
https://fedoraproject.org/wiki/DNF_system_upgrade#How_do_I_use_it.3F

This might be naive, but how about a stop-gap fix involving a --keepcache command line flag, which would act as the dnf.conf keepcache option setting, but non-intrusively.

The idea would be use this flag on dnf transaction used to resolve the system-upgrade, without resorting to bare rpm(1) or modifying /etc/dnf/dnf.conf ?

Further to comment 9, I think this should really be listed on the "Common F23 bugs" page (https://fedoraproject.org/wiki/Common_F23_bugs). The unnecessary bandwidth consumption is a major issue for some people.

Separately, I don't follow why "dnf system-upgrade reboot" doesn't explicitly note it has no upgrade files to work with when that's the case. One would assume the user is expecting the function to actually do something, and it would be abnormal to have no upgrade files present. Instead we get the message "Upgrade complete! Cleaning up and rebooting..." The upgrade couldn't possibly be complete, because no action was taken.

(In reply to Arnaud Lacombe from comment #10)
> This might be naive, but how about a stop-gap fix involving a --keepcache
> command line flag, which would act as the dnf.conf keepcache option setting,
> but non-intrusively.

Thanks, that's a good idea. The correct working option is --setopt=keepcache=True.

(In reply to David H. Gutteridge from comment #11)
> Further to comment 9, I think this should really be listed on the "Common
> F23 bugs" page (https://fedoraproject.org/wiki/Common_F23_bugs). The
> unnecessary bandwidth consumption is a major issue for some people.

I'll put it there.

> Separately, I don't follow why "dnf system-upgrade reboot" doesn't
> explicitly note it has no upgrade files to work with when that's the case.

That would be https://github.com/rpm-software-management/dnf-plugin-system-upgrade/issues/33 .

*** Bug 1283601 has been marked as a duplicate of this bug. ***

*** Bug 1283818 has been marked as a duplicate of this bug. ***

*** Bug 1287570 has been marked as a duplicate of this bug. ***

I don't know if this is fully resolved, but I had a much better experience in this context upgrading to Fedora 24. I repeatedly hit issues where the upgrade couldn't continue, but it kept all the existing packages in cache between (successful) dnf operations, so nothing had to be downloaded again.

dnf changed to purge the cache only on successful installation transactions [1] in version 1.0.2. So in the common case where you uninstall a few packages by hand to allow the upgrade to proceed the expected thing happens and already downloaded packages are not deleted. We can close this, since the behaviour is significantly changed from when this report was first opened. If we need to do further fine-tuning, let's do it in a separate report.

[1] http://dnf.readthedocs.io/en/latest/release_notes.html#id11