Bug 1263677 - it's very easy to end up with a partially-upgraded system
it's very easy to end up with a partially-upgraded system
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dnf-plugin-system-upgrade (Show other bugs)
22
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Will Woods
Fedora Extras Quality Assurance
AcceptedBlocker
:
Depends On:
Blocks: F23FinalBlocker
  Show dependency treegraph
 
Reported: 2015-09-16 08:08 EDT by Kamil Páral
Modified: 2015-11-03 00:52 EST (History)
13 users (show)

See Also:
Fixed In Version: dnf-plugin-system-upgrade-0.7.0-1.fc22
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-03 00:52:20 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dnf system-upgrade with skipped packages (174.35 KB, text/plain)
2015-09-16 08:14 EDT, Kamil Páral
no flags Details

  None (edit)
Description Kamil Páral 2015-09-16 08:08:10 EDT
Description of problem:
The new system-upgrade tool seems to lead to just a partially-upgraded system in my cases. This problem is discussed in length at
https://github.com/rpm-software-management/dnf-plugin-system-upgrade/issues/10
. This bugzilla ticket is created for the purpose of blocker bug discussion and tracking. I'll try to summarize the issue here, but don't forget to look at the github ticket if you want to learn the details.

There is a difference between how fedup worked and how system-upgrade currently works:

* fedup seemed to download all updated packages from FN+1 repo and force install them, even though that left some packages with broken dependencies. The downside was that if there was an upgradepath issue between packages, it could have left some important packages broken and non-functional (that brought QA quite a bit of work and worries). OTOH, the upside is that if the broken packages were third-party packages, they didn't negatively effect the rest of the system - the core system was updated and the third-party packages were non-functional in the worst case. The user's system was safe.

* system-upgrade now uses dnf (standard "dnf update" approach) and never ever breaks dependencies. If dependencies cannot be resolved for an updated package, it skips that package and excludes it from the upgrade. That in turn can cause other dependent packages to be excluded from the upgrade. In other words, dnf finds the largest set of packages that can be upgraded without breaking any dependencies, and skips the rest. The upside is that no packages will ever get broken dependencies, so they should theoretically keep working. The downside is that any single package (no matter how non-important) can prevent a large portion of core system from upgrading - it just needs to depend on an important system package or library.

We discussed system-upgrade behavior during our QA meeting and consider it quite worrisome. It's very easy to end up in an FN+1 "upgraded" system where arbitrary number of packages were left at the older FN version. In my real-world scenario (installing vlc from rpmfusion), I ended up with 10-15% of packages not upgraded, including many core system packages.

My worry is that:
a) partially upgraded system is quite likely to not work correctly, even though all RPM dependencies are satisfied. All it takes is for some important system package to forget to define some RPM dependency, and the system might not even boot into graphics. Or many apps might be broken. Nobody runs a mixture of FN/FN+1 packages, so it's hard to expect how badly the system might get broken.

b) this partially upgraded system will not recover itself (fully upgrade) until all issues holding the affected packages back from being upgraded are not resolved. That might happen soon enough for packages provided by Fedora (usually we just need to fix the upgrade path), but it might never happen for third-party packages (especially those without yum repos). It seems possible to fix the system manually by running "dnf distro-sync --allowerasing" (provided you booted fine), but the user needs to know about this.

c) the fact that a partial upgrade will be performed is very inconspicuous. You have to look very carefully at the system-upgrade output and you have to understand packaging to find out that a certain portion of your system will not be upgraded. Even then, many people will not realize the potential consequences. There is no warning about this.

d) this problem will change heavily in time. Any time we break upgrade path in our repos (and we do that unfortunately quite often), this can happen. Any time a third-party package is installed or updated, this can happen.

At the moment, the problem can be easily reproduced with vlc/mplayer/ffmpeg installed from rpmfusion. I tested other popular third-party apps like skype, dropbox, steam, google-chrome and atom and did not hit the issue. But that doesn't mean the situation can't change rapidly when one of those packages start depending on a particular version of a system package/library. Also, I tested just a small part of software that our users are likely to have installed.

There are some proposed solutions for this, which could significantly increase system-upgrade reliability (or at least seem to do that, though certainly not a panacea):
1. change "dnf update" mode during system upgrade to "dnf distro-sync --allowerasing". The distro sync bit will allow us to work around broken upgrade path issues and install a lower version of a package during upgrade. That mostly covers issues related to Fedora repos. The "--allowerasing" bit will allow us to remove packages which wouldn't have satisfied dependencies with FN+1 versions of packages, and would cause other packages to be held back. That mostly covers third-party packages with missing FN+1 repos or with no repos at all. This approach is currently blocked by dnf bug 1260989 and bug 1244175, hopefully to be resolved soon.

2. improve system-upgrade messaging towards the user
- in case we keep "dnf update" mode: either reject any upgrades that would result in skipping system packages due to unresolved dependencies, or very clearly alert about it ("this might result in a broken system") and have it confirmed by the user
- in case we go for "dnf distro-sync --allowerasing" mode: very clearly alert about any packages that are going to be removed and have it confirmed by the user


Version-Release number of selected component (if applicable):
dnf-plugin-system-upgrade-0.4.0-1.fc22.noarch

How reproducible:
always, at least until rpmfusion has f23 repos. after that, we can create a fake package to reproduce the issue, if needed.

Steps to Reproduce:
1. install vlc from rpmfusion on fully-updated default install of F22 Workstation
2. sudo dnf system-upgrade download --releasever=23
3. look at the output and search for "Skipping packages with broken dependencies:" section. All those packages would be skipped and not upgraded.
Comment 1 Kamil Páral 2015-09-16 08:14:41 EDT
Created attachment 1073991 [details]
dnf system-upgrade with skipped packages

This is an example of a system-upgrade output on a fairly default up-to-date F22 machine with vlc installed from rpmfusion. 190 packages are to be skipped from upgrade, including PackageKit, evince, evolution, gnome-shell, gnome-session, grub2, nautilus, perl, rpm, etc. Look for "Skipping packages with broken dependencies" section.

It's easy to showcase this with third-party packages, but it could happen all the same with a Fedora package as well (hard to judge whether it's higher or lower probability).
Comment 2 Kamil Páral 2015-09-16 09:20:20 EDT
I'm proposing this for a blocker bug discussion. Even though the following criteria are Beta, I'm proposing it for Final, because I think the current unsafe upgrade is still OK for Beta, but far from ideal for Final. The criteria say:
"For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed. "
" The upgraded system must meet all release criteria. "
"The upgraded system must include all packages that would be present on the system after a default installation from install media, plus any packages the user previously had (minus any obsolete content)."
https://fedoraproject.org/wiki/Fedora_23_Beta_Release_Criteria#Upgrade_requirements

This particular use case (conditional partial-upgrade) is not covered in them, but it can be argued that such an upgraded system is very different from a default installation and is likely to not meet all release criteria.

The major question will probably be how likely this partial upgrade is to occur. Also whether there are some downsides or other concerns with using distro-sync mode to mitigate this.
Comment 3 Will Woods 2015-09-16 12:42:48 EDT
The basic question is this: What should happen if there's a conflict between our base system and some third-party software? Do we:

a) break the third-party software and install the new base entirely, or
b) keep the system consistent and update components as they become ready?

The first one was the policy used by anaconda and fedup; the latter is the DNF policy, and a much more "rolling-release"-style approach.

I do think it's likely that most current users expect that a major-version system upgrade might break third-party apps until their vendors update the app for the new OS, so I'm not *opposed* to that policy.

In that case, what we'd really want is `dnf distro-sync --force`. You don't want to *erase* packages with broken deps - they usually still work, in fact. You just want DNF to ignore the fact that we're breaking dependencies, like anaconda and fedup used to do.

The problem is: there's not really a good way to do that in DNF, as far as I know. It doesn't have a --force flag. In fact (AFAICT) DNF plugins and extensions simply *can't* break system dependencies. I *tried* to make dnf system-upgrade behavior match fedup behavior, but DNF just.. wouldn't do it.

So: if it's the case that our new system package manager refuses to *ever* break system dependencies, we either need to change our policy, or work to make DNF/hawkey to let system upgrades break dependencies.

I think it's up to the DNF team (and/or FESCo) to decide how to best proceed here.
Comment 4 Kamil Páral 2015-09-17 03:51:37 EDT
Thanks, Will, that's a good thought. There's one assumption I don't fully agree with, though - you think it's better to keep the third-party apps installed even with broken dependencies, I think it's better to remove them. It's true they often keep working, but just as often they don't (e.g. in this particular example with vlc). I don't think that putting the system into a broken deps state just because the apps _might_ still work is worth it. As you say, most current users probably expect third party apps might get broken during upgrade until their vendors provide an updated version. If we clearly say "we can upgrade your system, but we will need to remove packages A, B and C, you will need to reinstall them with updated versions later", I think it's a fair approach, it's better than to leave the apps likely broken, and it's safer for the base system (those packages could be some core system plugins that no longer apply).

So if change that assumption (removal is better than breaking deps), the solution becomes quite elegant, I think:
Compute what we need to remove in order to fully upgrade the system, and then present the user with a choice - either confirm the upgrade including the removals, or wait until all software providers supply updated versions. If somebody wants to risk using software with broken deps, they can always force install it after the upgrade. Or there could be a command line option for it (with a big fat warning), but we should default to the safer way (removal).

One thing that we never ever want to do (I believe) is to change user's system to a "rolling upgrade" of sorts and just partially upgrade the packages (end up with a mixture of FN and FN+1 packages). Surely not as a default operation mode. That's a recipe for disaster. Instead if should work in all-or-nothing mode (upgrade the system with some possible removals, or cancel).

I'll ask DNF team to provide us with feedback regarding preferred upgrade behaviors (update, distro-sync, --allowerasing, etc) and even the possibility for plugins to break deps (even though I see it as a complementary mode as best, surely not the default behavior).

But the all-or-nothing mode is something that could be worked on inside system-upgrade in the meantime... ?
Comment 5 Honza Silhan 2015-09-17 06:33:46 EDT
(In reply to Will Woods from comment #3)
> In that case, what we'd really want is `dnf distro-sync --force`. You don't
> want to *erase* packages with broken deps - they usually still work, in
> fact. You just want DNF to ignore the fact that we're breaking dependencies,
> like anaconda and fedup used to do.

Nope, I agree with Kamil that for system maintainability it's better to know that your packages are gone, have the user informed of that fact instead of hoping that they will magically work with broken dependencies.

> The problem is: there's not really a good way to do that in DNF, as far as I
> know. It doesn't have a --force flag. In fact (AFAICT) DNF plugins and
> extensions simply *can't* break system dependencies. I *tried* to make dnf
> system-upgrade behavior match fedup behavior, but DNF just.. wouldn't do it.

call it a feature of DNF.

> I think it's up to the DNF team (and/or FESCo) to decide how to best proceed
> here.

In my (DNF) opinion all packages should be forced to be updated/downgraded by their next releasever alternatives. In case of package conflict, packages from fedora/updates repositories should be preferred and third-party packages removed. On the system should not remain any packages with broken dependencies but rather be uninstalled.

In order to do that we should:
1) use "dnf distro-sync --allowerasing" (system-upgrade)
2) fix bug 1244175 (DNF)
3) prefer fedora packages over third-party ones (DNF/system-upgrade/libsolv) [1]
4) write testcases with possible scenarios (QEs)

[1] https://github.com/openSUSE/libsolv/issues/103
Comment 6 Will Woods 2015-09-21 13:25:16 EDT
(In reply to Jan Silhan from comment #5)
> (In reply to Will Woods from comment #3)
> > In that case, what we'd really want is `dnf distro-sync --force`. You don't
> > want to *erase* packages with broken deps - they usually still work, in
> > fact. You just want DNF to ignore the fact that we're breaking dependencies,
> > like anaconda and fedup used to do.
> 
> Nope, I agree with Kamil that for system maintainability it's better to know
> that your packages are gone, have the user informed of that fact instead of
> hoping that they will magically work with broken dependencies.

I agree with the *spirit* of this argument - We Should Not Break The User's System. But I'm not sure that users will accept this as the *only* mode of operation.

During major-version upgrades it's not uncommon to see dependency problems that turn out to be harmless - e.g. changes in libraries that don't actually affect the parts of the API your app is using. (We *should* have better metadata so we don't get these kinds of false positives, but here we are.)

So if this happens with (say) VLC, the current options are:
a) remove it from the system
b) leave it on the system, possibly broken, until an update fixes it

The problem with a) is that when the update is released that fixes VLC, it *won't be installed on the user's system*.

So. If there's a way to mark erased packages so they'll get re-installed once the dependency chain is fixed, then I'm OK with erasing them.

Otherwise, we're messing up people's systems because of temporary dependency problems and not really giving them a good way to fix it once the problems are resolved.

> > The problem is: there's not really a good way to do that in DNF, as far as I
> > know. It doesn't have a --force flag. In fact (AFAICT) DNF plugins and
> > extensions simply *can't* break system dependencies. I *tried* to make dnf
> > system-upgrade behavior match fedup behavior, but DNF just.. wouldn't do it.
> 
> call it a feature of DNF.

I actually like this feature, but it *does* run counter to user expectations for upgrade behavior in this case.

> > I think it's up to the DNF team (and/or FESCo) to decide how to best proceed
> > here.
> 
> In my (DNF) opinion all packages should be forced to be updated/downgraded
> by their next releasever alternatives. In case of package conflict, packages
> from fedora/updates repositories should be preferred and third-party
> packages removed. On the system should not remain any packages with broken
> dependencies but rather be uninstalled.
> 
> In order to do that we should:
> 1) use "dnf distro-sync --allowerasing" (system-upgrade)

I can make this the default behavior, but I'd really like there to be
a way to retain a list of userinstalled packages that get removed during the upgrade, so they'll get reinstalled when the dependency problems are resolved.

(Or, failing that, a --force flag to ignore dependency problems, like anaconda and fedup used to do. Which is kind of a bad idea, but that's what everyone expects..)

> 2) fix bug 1244175 (DNF)
> 3) prefer fedora packages over third-party ones (DNF/system-upgrade/libsolv)

How do we determine which repos are "fedora" (or, more broadly, "base-system") and which are "third-party"?

> 4) write testcases with possible scenarios (QEs)

This seems like a solid plan - but how long do we have to get this done for F23?
Comment 7 Adam Williamson 2015-09-22 12:10:32 EDT
I'd just like to emphasize that for me there's something just as important as what the tool ultimately actually does, and that's *what it tells the user*.

Whatever the solution is I'd want it to have a fairly strong warning for any case where packages are going to be removed. I'd also like the case where a bunch of packages are not going to be updated to be clear. The main thing is not to let the user think a perfectly clean upgrade is going to happen when in fact there are problems.
Comment 8 Adam Williamson 2015-09-22 12:13:35 EDT
Also I think we shouldn't just consider this to be exclusively a 'third party software' issue. There certainly have been cases where Fedora's own repositories have issues that would cause us to encounter these behaviours.
Comment 9 Stephen Gallagher 2015-09-23 09:14:43 EDT
I've opened https://fedorahosted.org/fesco/ticket/1481 to get a decision from the Fedora Engineering Steering Committee. I will report the results back once we have them.
Comment 10 Adam Williamson 2015-09-23 11:07:38 EDT
Discussed at 2015-09-23 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-22/f23-blocker-review.2015-09-22-16.00.html . Accepted as a blocker: we agreed that the potential for inadvertently arriving at a half-upgraded system is sufficient to constitute a violation of "The upgraded system must meet all release criteria." (sub-clause of the upgrade criterion). The requirement here is pretty much that whatever FESCo decides on needs to get done (assuming it's reasonable).
Comment 11 Michael Catanzaro 2015-09-23 15:54:53 EDT
A bit of input with my Workstation hat on: our current designs for graphical upgrades are to remove third-party software that conflicts with the upgraded system. The user would be presented with a list of applications that will be removed during the upgrade before confirming the upgrade.
Comment 12 Will Woods 2015-09-23 16:04:24 EDT
...are you planning for a "re-install this application once a compatible version is available" checkbox/mode?
Comment 13 Chris Murphy 2015-09-23 16:08:21 EDT
(In reply to Michael Catanzaro from comment #11)
I mentioned this just now in the FESCO ticket. I can think of no example anywhere, where applications are removed during upgrades, not even on mobile where apps get abandoned with increasing regularity as the platform matures. I would be irritated if they only way I can upgrade is to agree to (possibly/likely) arbitrary app removal. App breakage is annoying, but there's decades of history conditioning people that apps can break upon major OS version upgrades.

Hopefully I'm just confused, but at the moment to me this sounds like a very different UX than anyone is used to.
Comment 14 Michael Catanzaro 2015-09-23 17:02:08 EDT
(In reply to Will Woods from comment #12)
> ...are you planning for a "re-install this application once a compatible
> version is available" checkbox/mode?

I am not sure. I don't think so, but this problem might not have been considered yet.

(In reply to Chris Murphy from comment #13)
> (In reply to Michael Catanzaro from comment #11)
> I mentioned this just now in the FESCO ticket. I can think of no example
> anywhere, where applications are removed during upgrades, not even on mobile
> where apps get abandoned with increasing regularity as the platform matures.

When we get third parties using xdg-app, this problem won't exist anymore. Until then, we've got to pick the least-bad option. Seems to me that's "remove" but it's debatable.
Comment 15 Chris Murphy 2015-09-23 18:19:33 EDT
(In reply to Michael Catanzaro from comment #14)
> When we get third parties using xdg-app, this problem won't exist anymore.
> Until then, we've got to pick the least-bad option. Seems to me that's
> "remove" but it's debatable.

Least bad is the one with the long standing history, which is "maybe breaks". The alternative "remove" is actually "definitely breaks" because the app isn't even there. I think that shift is unexpected and will result in perturbation.
Comment 16 Adam Williamson 2015-09-23 18:33:38 EDT
Just stating my personal opinion - this is nothing official on behalf of blocker process or QA or anything else - I agree with cmurf and (I think?) wwoods that out of 'erase packages with broken deps' and 'keep packages with broken deps' I prefer the latter, for the same reasons:

i) it's less drastic
ii) it provides a chance for the issue to be resolved later
iii) it's what we've been doing for years
Comment 17 Will Woods 2015-09-24 16:24:48 EDT
(In reply to awilliam@redhat.com from comment #16)
> Just stating my personal opinion - this is nothing official on behalf of
> blocker process or QA or anything else - I agree with cmurf and (I think?)
> wwoods that out of 'erase packages with broken deps' and 'keep packages with
> broken deps' I prefer the latter, for the same reasons:
> 
> i) it's less drastic
> ii) it provides a chance for the issue to be resolved later
> iii) it's what we've been doing for years

Okay, but this is still literally not possible with current DNF. So unless/until the DNF team commits to making that possible, further discussion of this particular option is moot.

Let's keep further discussion of personal opinions and *possible* solutions to places better suited for that (e.g. mailing lists and/or FESCo meetings).

The next comments on this bug should be FESCo and/or DNF-team approved plans for what to do for F23 (and beyond, if possible).
Comment 18 Honza Silhan 2015-09-25 11:22:50 EDT
(In reply to Will Woods from comment #6)
> (In reply to Jan Silhan from comment #5)
> > In my (DNF) opinion all packages should be forced to be updated/downgraded
> > by their next releasever alternatives. In case of package conflict, packages
> > from fedora/updates repositories should be preferred and third-party
> > packages removed. On the system should not remain any packages with broken
> > dependencies but rather be uninstalled.
> > 
> > In order to do that we should:
> > 1) use "dnf distro-sync --allowerasing" (system-upgrade)
> 
> I can make this the default behavior, but I'd really like there to be
> a way to retain a list of userinstalled packages that get removed during the
> upgrade, so they'll get reinstalled when the dependency problems are
> resolved.
>
> So if this happens with (say) VLC, the current options are:
> a) remove it from the system
> b) leave it on the system, possibly broken, until an update fixes it
> 
> The problem with a) is that when the update is released that fixes VLC, it
> *won't be installed on the user's system*.
> 
> So. If there's a way to mark erased packages so they'll get re-installed
> once the dependency chain is fixed, then I'm OK with erasing them.

You can store the package names which is about to be removed by system-upgrade. And anytime DNF action including transaction is triggered next time (dnf install/update/...) it will try to mark them for installation in current transaction (with some informative message from system-upgrade). Will, let me know directly for technical details.

> > 2) fix bug 1244175 (DNF)
> > 3) prefer fedora packages over third-party ones (DNF/system-upgrade/libsolv)
> 
> How do we determine which repos are "fedora" (or, more broadly,
> "base-system") and which are "third-party"?

Good question. I don't know at this moment how to do it generally without hard coding repository names in fedup for each distro.

update: The feature is implemented in libsolv and could be propagated into hawkey and DNF.

> > 4) write testcases with possible scenarios (QEs)
> 
> This seems like a solid plan - but how long do we have to get this done for
> F23?

I am not sure if we can make it all till F23 Final freeze with proper testing.

How I see the system-upgrade should look like:
1) distro-sync for all packages (delete the ones from third-party repos in case of conflicts), user confirms whether to proceed to 4) or advise him to do 2) with consequences explained
2) distro-sync that allows erasing of any packages, user confirms whether to proceed to 4) or advise him to do 3) with consequences explained
3) update packages that can be updated, user confirms whether to proceed to 4)
4) transaction done and reboot(In reply to Will Woods from comment #17)

(In reply to awilliam@redhat.com from comment #16)
> I agree with cmurf and (I think?)
> wwoods that out of 'erase packages with broken deps' and 'keep packages with
> broken deps' I prefer the latter, for the same reasons:
>
> i) it's less drastic

Update is less drastic but the consequences are brutal. This can cause a lot pain when user encounter unexpected unidentified behavior. Maintainers of applications would waste their time during reproducing/debugging the issue too.

> ii) it provides a chance for the issue to be resolved later

first approach too, see the solution above

> iii) it's what we've been doing for years

time for change ;)
Comment 19 Adam Williamson 2015-09-25 11:25:58 EDT
and can the user query this setup so they know that they have some packages in this weird 'not installed, but ready to show up at a moment's notice' state?
Comment 20 Kamil Páral 2015-09-29 08:33:36 EDT
Folks, let's not overengineer this? Is it really so hard to tell users "system upgrade will require VLC and Skype to be removed, install it after the upgrade again or contact the package supplier if there are any issues with that"? People are not stupid, they managed to install it once, they will manage to install it again (or they will complain to the repo maintainers and a fixed version will appear). General users will be very glad for an increased upgrade reliability (finally!) and power users will find their way around if absolutely needed (force install the outdated package version). We don't need complicated and error-prone systems that try to install these packages again at a seemingly random time in the future. You also seem to forget about packages not coming from a repo - if they are removed, it will point them into downloading a newer version; if they are not removed and kept broken instead, many people will not realize a newer version is necessary, they will just see it as broken by the upgrade. Again, let's not overengineer this.


Any other upgrade mode (e.g. "break my system" mode) can be an addition for power users, but it will require substantial DNF changes and we don't need to deal with it right here and right now. It's not release critical. We can deal with it in a separate bug report.
Comment 21 John Ellson 2015-09-29 12:39:39 EDT
Do you mind votes from a Lurkers?    I vote for reliability.   Just give me a list of packages that need to be removed before upgrading, and I'll take care of it manually, and of reinstalling afterwards when updates are available.

And a gentle request?   Could we develop the upgrade process *before* the beta release next time?    I think this is about the third release in a row where the upgrade process has been a last minute afterthought.
Comment 22 Adam Williamson 2015-09-29 13:29:30 EDT
https://github.com/rpm-software-management/dnf-plugin-system-upgrade/commits/master?page=3

development began in May. That's long before Alpha.
Comment 23 Will Woods 2015-09-30 16:28:36 EDT
(In reply to John Ellson from comment #21)
> Do you mind votes from a Lurkers?   

Pretty sure I asked for comments from FESCo members and/or DNF team members, but a well-considered opinion never really hurts!

> I vote for reliability.   

...well, sure. Who doesn't? But what that means to you might be different from what other users want, and without some good data or evidence I really don't know how to discern who's right.

> Just give me
> a list of packages that need to be removed before upgrading, and I'll take
> care of it manually, and of reinstalling afterwards when updates are
> available.

Lucky you, this is now basically the plan! It's almost exactly equivalent to:

   dnf distro-sync --allow-erasing --releasever=<NEWVERSION>

See https://fedorahosted.org/fesco/ticket/1481 for details on that.

> And a gentle request?   Could we develop the upgrade process *before* the
> beta release next time?    I think this is about the third release in a row
> where the upgrade process has been a last minute afterthought.

Well. Development actually began in April, during F22 Beta, immediately after we discovered that fedup-style upgrades were no longer supportable:

  http://lists.freedesktop.org/archives/systemd-devel/2015-April/031013.html

I wrote a fully-functional prototype replacement, and announced the plan to drop fedup literally the day after F22 was released:

  https://lists.fedoraproject.org/pipermail/devel/2015-May/210905.html

So yeah, not sure what to tell you. I'm just one guy, and I did my best.



Anyway: as mentioned earlier in this comment, we now have a plan for the default behavior for F23, and so I'll be implementing that in the plugin shortly.
Comment 24 Kamil Páral 2015-10-15 08:40:00 EDT
I have tried testing system-upgrade with https://bodhi.fedoraproject.org/updates/FEDORA-2015-02efefba0e (which should fix bug 1260989). Due to some unresolved dependencies (that update is still in updates-testing), it requires using --allowerasing (we will want to use it by default anyway, so nothing new). If I use it on the "system-upgrade download" line, the download is performed correctly. But the actual upgrade fails with:

Oct 15 14:31:37 localhost.localdomain dnf[641]: Error: package python-dnf-1.1.3-1.fc22.noarch requires dnf-conf = 1.1.3-1.fc22, but none of the providers can be installed
Oct 15 14:31:37 localhost.localdomain dnf[641]: (try to add '--allowerasing' to command line to replace conflicting packages)

So it seems that even though I used it for download, that flag was not used during the actual upgrade. Maybe it's obvious, but I just want to point out that we should make sure we use the same flags during both download and upgrade. If there are any such supported command line options, we need to apply them in both cases.
Comment 25 Kamil Páral 2015-10-19 10:14:04 EDT
Will, what is the status of the necessary system-upgrade adjustments? All the blocking dnf bugs have hopefully been resolved, so we're just waiting for new system-upgrade.
Comment 26 Will Woods 2015-10-19 12:15:29 EDT
distro-sync mode is default as of commit 98ceee2:

  https://github.com/rpm-software-management/dnf-plugin-system-upgrade/commit/98ceee2

Upgrade correctly uses --allowerasing and/or --best as of commit 49a9318:

  https://github.com/rpm-software-management/dnf-plugin-system-upgrade/commit/49a9318

At this point I'm waiting for more info from the DNF team on bug 1266589, since that seems to be the cause of many random upgrade failures and I'd like to get that fixed before the next build if possible.
Comment 27 Adam Williamson 2015-10-19 20:30:10 EDT
Well if we're waiting for that, we probably should send it through some lovely bureaucracy...I'll mark it as a proposed blocker.
Comment 28 Kamil Páral 2015-10-20 03:52:12 EDT
I'd rather have a new system-upgrade release right now, so that we can test it ASAP. I believe bug 1266589 is likely just a dupe of 1260989 and is in fact already resolved.
Comment 29 Will Woods 2015-10-20 12:54:22 EDT
(In reply to Kamil Páral from comment #28)
> I'd rather have a new system-upgrade release right now, so that we can test
> it ASAP. I believe bug 1266589 is likely just a dupe of 1260989 and is in
> fact already resolved.

The evidence in bug 1266589 seems to indicate that it's a different problem than bug 1260989, but okay - I'll ignore that and see about getting a new build done ASAP.
Comment 30 Kamil Páral 2015-10-28 14:08:58 EDT
Here's the update:
https://bodhi.fedoraproject.org/updates/dnf-plugin-system-upgrade-0.7.0-1.fc22
Comment 31 Fedora Update System 2015-10-28 15:45:13 EDT
dnf-plugin-system-upgrade-0.7.0-1.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-e00b75e39f
Comment 32 Kamil Páral 2015-10-29 10:07:14 EDT
I have tried simulating the original problem with vlc. I created erase_me package which conflicts with fedora-release-23. On default system-upgrade invocation, this is what I see:

$ sudo dnf system-upgrade download --releasever=23
Last metadata expiration check performed 0:00:00 ago on Thu Oct 29 14:54:19 2015.
Error: package erase_me-1-1.noarch requires fedora-release < 23, but none of the providers can be installed
(try to add '--allowerasing' to command line to replace conflicting packages)

I hoped for something more interactive, but if the user reads carefully and runs it again with --allowerasing, the erase_me package is marked for removal and the upgrade proceeds correctly. I'm a bit unhappy that the list of to-be-removed packages is shown somewhere in the standard dnf output (but often not at the end), so it's not very visible and obvious which packages are going to get removed. But even though the UI is not best, it gets the work done. Hopefully the UI will be more polished in the future.
Comment 33 Stephen Gallagher 2015-10-29 10:22:45 EDT
The FESCo decision was made requiring that some UI for the decision must be present, but did not specify that it must meet any particular guidelines. It was considered acceptable for F23 that the standard DNF output could be used, but it's recommended that this be improved in the future.
Comment 34 Fedora Update System 2015-11-01 19:27:33 EST
dnf-plugin-system-upgrade-0.7.0-1.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update dnf-plugin-system-upgrade'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-e00b75e39f
Comment 35 Fedora Update System 2015-11-03 00:51:55 EST
dnf-plugin-system-upgrade-0.7.0-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.