Bug 892061 (fedup-distro-sync)

I had the same problem. Let us see what will happen on second system

Upgraded second system. KDE started OK.

Had to "upgrade" qt-* by:
   yum downgrade qt-x11 qt-config qt-qt-assistant qt qt-mysql qt-examples qt-demos qt-devel qt-doc

The same for linux-firmware, media-player-info, libfprint, kdiff3, clamav*
(Had to upgrade by 'yum downgrade')

It's a packaging thing. 

In this case, the new KDE packages hadn't been pushed yet, so the F17 packages were newer than the F18 packages, so the F18 ones didn't get installed.

But (as you noticed) the F17 KDE packages require F17 libraries, which get upgraded to incompatible F18 versions, so then KDE doesn't work anymore.

The obvious fix is: push F18 KDE updates! And they've been pushed now, so this specific problem is fixed. See bug 888085 for details.

---

The larger problem is: what *should* happen in these cases?

Should we really install older packages if one of them says ".fc18"?
  If "yes, sometimes" - when, exactly? How do we know?
  If "yes, always" - why doesn't yum enforce this?
  In either case - why aren't we using RPM tags to control this?
  (We could abuse Epoch, or Distribution, or OS, or use unused tags like DistTag or Collections...)

More information about the possible problems and possible solutions is necessary before we can come up with a good policy / implementation for this.

*** Bug 820351 has been marked as a duplicate of this bug. ***

And it is not the first time I see similar packaging problem.

I have had to upgrade some packages using "yum downgrade ..." already for several Fedora version upgrades like to Fedora 17. Each time it takes unnecessary time to hunt down previous Fedora version packages and upgrade them manually using "yum downgrade ..."

I guess some permanent solution for this problem would be nice.

I could imagine adding command line option to add extra version sorting key using REGEX which takes precedence to the usual version number. Such regular expression could then be used to extract Fedora distribution version code and as result to avoid the problems being discussed here. Also it would be generic enough and not for Fedora distribution only

*** Bug 895862 has been marked as a duplicate of this bug. ***

The problem is still present with Fedora 19. Upgraded to it with fedup.

Had to upgrade manually emacs and emacs-common using 'yum downgrade'.
I do not have time to check for full list now, but I'll do it sometimes later.

So I guess the version (in which seen) should be updated from F17 to F19

there's no point associating it with a specific version as it's essentially an RFE for fedup rather than a specific bug.

To be clear, this is (currently) not a fedup bug, but a packaging problem. The Fedora Package Naming Guidelines say:

   In Fedora we want to ensure that there is always an upgrade path from
   Fedora release to Fedora release and from Fedora release to the packages
   in updates. To do that we need to make sure the packages in the newer
   Fedora releases have an equal or higher Epoch:Version-Release (EVR) than
   the ones in older releases.[1]

In other words: it's up to package maintainers to ensure that there is a viable upgrade path for their packages.

So fedup's behavior isn't going to change unless the Fedora Project decides on some packaging policies for upgrades that answer the questions in comment 3.

If that happens, I'll happily make whatever changes are necessary, but it's not my place to invent new policies without oversight.

[1] https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Package_Versioning

(In reply to Will Woods from comment #9)
> To be clear, this is (currently) not a fedup bug, but a packaging problem.

Agreed, but still, there are things that can be done in fedup to partially mitigate these problems. Probably the easiest way would be:

1. Check in advance whether the upgrade path requirement is satisfied for all packages (if there is a package in F-N and the same package in F-N+1, then its version must be same or higher in F-N+1).
2. If the upgrade path is broken for some packages, print a warning ("the upgrade might not be safe"), list all these packages, and tell people to report bugs against them.

This in-advance warning could:
a) save some trouble for a lot of users, because some of them would decide to wait until the problems are resolved, instead of potentially breaking their systems
b) force the package maintainers to update their packages, because the issues would more visible and there would be more bug reports

In last several upgrades I have always met these problems. In worst case they can lead to unusable system due to correct versions of shared libraries not available for some critical package. Ensuring correct upgrade path at release time is not sufficient as user may choose to upgrade later.

If there would be a possibility of Fedora distribution version to take precedence like I suggested in comment 5 this problem would be solved even if user decides to use for example DVD for upgrade.

(In reply to Kamil Páral from comment #10)
> 1. Check in advance whether the upgrade path requirement is satisfied for
> all packages (if there is a package in F-N and the same package in F-N+1,
> then its version must be same or higher in F-N+1).
> 2. If the upgrade path is broken for some packages, print a warning ("the
> upgrade might not be safe"), list all these packages, and tell people to
> report bugs against them.

I agree somewhat - a simple "The following packages won't be updated because no newer version was found: [...]" listing might be useful to people.

But it can't really be a big "WARNING YOUR SYSTEM WILL PROBABLY BREAK", because it's not always a problem when the upgrade path is "broken". There are packages that don't have a new version because they don't need to be upgraded; the Packaging/Naming Guidelines allow for this, so we have to allow for it.

(If we *did* want that to be true, we could easily just enforce this via Epoch (or some other tag like DISTTAG or DISTRIBUTION or whatever) and this problem would go away. But I'm not going to change that policy without oversight!)

(In reply to Andris Pavenis from comment #11)
> In last several upgrades I have always met these problems. In worst case
> they can lead to unusable system due to correct versions of shared libraries
> not available for some critical package.

fedup already runs a test RPM transaction to warn the user about dependency problems; fatal transaction errors will cause fedup to exit with an error.

The solution is simple. Fedup should perform a "yum distro-sync". It simply makes no sense running "yum distro-sync" and having packages to be either downgraded or fetching the same version with the correct "fcN" tag, after a successful fedup transition to the newer version.

I mean think about it. What are the actions that we run after an upgrade?

1. yum distro-sync
2. yum list extras
3. yum remove <obsolete packages from previous releases>

Personally, I would also love fedup to include up to step 3 to deliver a ready system, but I would be happy if it includes at least step 1. Step 1 is a must for the 99% of the cases. If the user for some weird reason would like to keep previous release package, it should be fixed manually.

I disagree with Mr. Woods. The packaging problem you are describing might be true, but it is simply a workaround and not the solution. Fedup fails to deliver a system with the correct latest release packages, so it's all fedup's fault. Fedup should be aware of all package versioning exceptions and handle them efficiently. A failure to do so is considered as an application bug and only that. Any other interpretation sounds like a cheap excuse.

(In reply to Will Woods from comment #12)
> But it can't really be a big "WARNING YOUR SYSTEM WILL PROBABLY BREAK",

Let's say "might".

> because it's not always a problem when the upgrade path is "broken". There
> are packages that don't have a new version because they don't need to be
> upgraded; the Packaging/Naming Guidelines allow for this, so we have to
> allow for it.

Since F19 we rebuild _all_ packages on branch date. So this is no longer the case. Every F18 package now has its F19 equivalent. If F18 version is higher than F19 version, then it's a violation of our packaging policies ("upgrade path"), and notifying our users that the not everything is in order might help the users substantially.

Will: I understand your POV but I _do_ think it's a tad obtuse. On a simple practical level I really don't see why fedup shouldn't simply do a distro-sync. Yes, we have a policy saying the upgrade path should always be respected. But come on: in practical terms, history indicates that this does not happen. That's why yum *has* a distro-sync command in the first place (partly). It's why the 'upgrading with yum' page recommends using yum distro-sync, not yum upgrade. Practically speaking, the distro-sync behaviour is going to produce the desired result more frequently than the update behaviour. Can we please just use it?

(In reply to Adam Williamson from comment #15)
> Will: I understand your POV but I _do_ think it's a tad obtuse. On a simple
> practical level I really don't see why fedup shouldn't simply do a
> distro-sync.

1) That's not how we've ever done upgrades, so it's a substantial behavior change
2) It doesn't *solve* the underlying problems, it just papers over them,
3) It violates the principle of Least Surprise by causing the upgrade to modify *more* packages than necessary,
4) It would forcibly remove any locally-customized packages, which would certainly result in new bug reports,
5) There's already a simple solution to the problem - run 'yum distro-sync' after the upgrade.

But you're right about the practicalities of it. So: one could probably add a "--distro-sync" flag to fedup that would make fedup do what you want - replace newer packages with older ones from the new release.

But it's probably not going to be the default, for the reasons above, and the same reason that 'yum distro-sync' isn't yum's default behavior for updates, and roughly the same as why RPM writes .rpmnew files rather than replacing your config files when you've modified them: when we can't be sure that something is always the Right Thing To Do, we generally defer to the user.

1) is perfectly true, and people have been complaining about this problem *for as long as we've been doing upgrades the other way* :)
2) yes. Indeed it is. As 'solving the underlying problems' hasn't happened in the last decade or so, papering over them seems not an unreasonable thing to do.
3) Well, 'necessary' is the question, isn't it? If the packages that didn't get 'updated' cause significant borkage on the system - as, in several cases of this bug, they seem to - surely it's reasonable to suggest that updating them turns out to have been 'necessary'.
4) Probably true.
5) Well, assuming your system is at least bootable, and that you find this bug that tells you what's going on and why you have to run a command to complete your upgrade after having done your upgrade. But it doesn't exactly read as great.

Adding a parameter would be fine, but even if you don't make it the default, the practical upshot is going to be that it's going to get written into the Wiki pages and the forum guide and the howto articles and everywhere anyone tells anyone else how to use fedup, and we'll waste just as many man-hours as it ultimately takes people to type '--distro-sync' on all fedup upgrades forever more (I suspect). But meh, there are worse things in the world. =)

I vote for "--no-distro-sync" instead of "--distro-sync" parameter too. Default should be distro-sync. 

Experienced users that would like to take full control on the upgrade procedure would probably be those that they have heavily customised their system and they would like to preserve their local modifications. For them the "--no-distro-sync" should be suffice. The majority resides in the distro-sync side in my opinion.

(In reply to Will Woods from comment #16)
> But it's probably not going to be the default, for the reasons above, and
> the same reason that 'yum distro-sync' isn't yum's default behavior for
> updates, and roughly the same as why RPM writes .rpmnew files rather than
> replacing your config files when you've modified them: when we can't be sure
> that something is always the Right Thing To Do, we generally defer to the
> user.

Upgrades and other difficult and risky operations should be easier for the general public and more demanding for power users. If you have locally modified packages, no problem, but the upgrade procedure requires you to do a few extra steps. Not the other way around - if you are a general user, then suffer for your ignorance.

I would be the first one to put --distro-sync into all our wiki guides and recommend it as the default command to run. Because it makes sense. distro-sync is also present in all upgrade guides over the Internet. So if there is such an option, it should be --no-distro-sync.

Let me attempt to clarify my point from comment 9:

There is existing policy on this matter already.
I'm not going to override that policy without approval from FESCo.
This is not my decision to make.

The citation in comment 9 is a policy for package maintainers, but doesn't really speak about they way how system upgrades should be performed, in my view. But I understand that you don't want to decide this on your own.

Will, would you mind creating a FESCo ticket then (and linking it here), summarizing the current questions/problems and asking for the best approach recommendation? It would be great to have this cleared up, so that we might be able to move forward here (or stop these discussions, depending on FESCo's resolution). Thanks a lot.

I think the current policy is fine. It leaves package maintainers free to decide whether or not their package *needs* to be replaced during the system upgrade. This also makes them responsible for building new versions of packages if they *do* want their package to be upgraded.

The fact that package maintainers sometimes fail to fulfill this responsibility is what causes the problems you see - not any defect in fedup.

fedup performs upgrades the exact same way that anaconda did for the past decade, using the default package upgrade behavior provided by RPM and yum, in accordance with all existing policy. It's working as designed. There's no bug here.

I suggest you work on the root of the problem: preventing upgrade path problems from happening in the first place. Maybe maintainers need earlier notification of upgrade problems, or maybe you could get packaging policies changed so newer releases' packages *are* considered newer.

You're welcome to fork fedup and make it work however *you* think is right, but unless I hear otherwise from FESCo and/or the Packaging Committee I have no plan to spend any more of my time on this.

"I suggest you work on the root of the problem: preventing upgrade path problems from happening in the first place. Maybe maintainers need earlier notification of upgrade problems"

Come on, they get a notification at build time. It's part of autoqa. You probably wrote it.

"You're welcome to fork fedup and make it work however *you* think is right"

Well, that's clearly absurd. fedup is a crucial part of Fedora: it is our officially recommended upgrade method. Forking it would serve no good purpose, only confusion.

So, let's recap:

1. Request for distrosync by default -- declined
2. Request for distrosync by option -- unclear, bug closed
3. Request to print a warning for upgradepath problems - unclear, bug closed
4. Request to continue discussion with FESCo -- unanswered, bug closed

I don't want to sound too harsh, but do you really need FESCo to bless every single detail, even printing warnings?

The bug is closed and there is no information whether something of those ideas will get implemented, or won't. It seems they won't, at all. We can fork it, great (Mirek Suchy already did with fedora-upgrade, kinda).

I agree it would be great to _prevent_ the upgrade path violations and all of this would go away. But we all know it's not going to happen very soon and the changes in fedup could save many common users from rendering their desktop unbootable in the mean time.

I can definitely create a ticket for FESCo to discuss the best upgrade approach. But I'm not even sure whether you want to discuss it with them, Will. Should I create the ticket?