Bug 1176403

Summary: FedUp does not upgrade wget
Product: [Fedora] Fedora Reporter: bob mckay <urilabob>
Component: fedupAssignee: Will Woods <wwoods>
Status: CLOSED CANTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: micah, tflink, thozza, urilabob, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-08 21:48:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
fedup log (installed wrong wget among other things) none

Description bob mckay 2014-12-21 12:33:48 UTC
Description of problem:
FedUp does not upgrade wget

Version-Release number of selected component (if applicable):
0.9.1-1.fc20

How reproducible:
One-off (because downgrade is necessary to reproduce)

Steps to Reproduce:
1. yum update, package-cleanup for dupes, problems, orphans; rpmconf -a
2. run fedup

Actual results:
wget (along with samba and sqlite) is not upgraded to F21

Expected results:
wget is upgraded to F21

Additional info:
Readily worked around (use yum to remove wget and dependencies; manually reinstall wget and dependencies).

Comment 1 bob mckay 2014-12-22 08:58:57 UTC
The cause is now clearer: inconsistencty between F20 and F21 repos for wget:

wget.x86_64                        1.16.1-1.fc21                        @updates

wget.x86_64                        1.16-3.fc20                          @updates

As I understand it, fedup can't handle this kind of inconsistency. 

Since the problem is with wget repos, reassigning to F21 wget.

Comment 2 Tomáš Hozza 2014-12-22 12:16:34 UTC
I don't see any issue withe the versions.

$ rpmdev-vercmp 1.16-3.fc20 1.16.1-1.fc21
1.16-3.fc20 < 1.16.1-1.fc21

The upgrade path is OK.

Comment 3 bob mckay 2014-12-23 03:14:53 UTC
Thanks for the feedback, Tomas.

Perhaps fedup isn't using rpmdev-vercmp for the comparison? All I can say for sure is that, reliably, at the moment fedup isn't upgrading wget (nor samba nor sqlite, which have the same issue of an fc21 version number that is lower than fc20). If you wish to verify, just create a vm, install fc20 and update to latest, then upgrade to fc21. You'll get warnings that wget etc. won't be upgraded (at that stage, leaving you with a system that can't be rebooted because a reboot will trigger an upgrade), and if you proceed then you will get a non-upgraded wget.

From Tomas' comments, it seems likely that this is an issue of differing interpretation of the spec for version fcn to fcn+1 relationships between at least some package maintainers and fedup maintainers (I'm sorry, I'm not sure where to find the documentation of this, so I can't check it). I have no idea whose interpretation is correct. If you are sure that the current wget version setups are correct according to the spec, then it must be fedup that is misinterpreting it. If that's the case, feel free to switch this bug report to fedup for fc20. But please accept that there is a real bug - either in fedup or in the interpretation of relative version requirements.

Comment 4 Tomáš Hozza 2015-01-05 07:32:10 UTC
(In reply to bob mckay from comment #3)
> Thanks for the feedback, Tomas.
> 
> Perhaps fedup isn't using rpmdev-vercmp for the comparison? All I can say
> for sure is that, reliably, at the moment fedup isn't upgrading wget (nor
> samba nor sqlite, which have the same issue of an fc21 version number that
> is lower than fc20). If you wish to verify, just create a vm, install fc20
> and update to latest, then upgrade to fc21. You'll get warnings that wget
> etc. won't be upgraded (at that stage, leaving you with a system that can't
> be rebooted because a reboot will trigger an upgrade), and if you proceed
> then you will get a non-upgraded wget.

You can boot the regular system by choosing the proper entry in the GRUB. FedUp just adds a new one, which is default, that's why after reboot your system will be upgraded.

Did you try to run 'yum update' after upgrade? Was wget updated using yum after upgrade?

> From Tomas' comments, it seems likely that this is an issue of differing
> interpretation of the spec for version fcn to fcn+1 relationships between at
> least some package maintainers and fedup maintainers (I'm sorry, I'm not
> sure where to find the documentation of this, so I can't check it). I have
> no idea whose interpretation is correct. If you are sure that the current
> wget version setups are correct according to the spec, then it must be fedup
> that is misinterpreting it. If that's the case, feel free to switch this bug
> report to fedup for fc20. But please accept that there is a real bug -
> either in fedup or in the interpretation of relative version requirements.

Moving to FedUp...

Comment 5 bob mckay 2015-01-05 09:39:23 UTC
(In reply to Tomas Hozza from comment #4)

Thanks again Tomas,
 
> You can boot the regular system by choosing the proper entry in the GRUB.
> FedUp just adds a new one, which is default, that's why after reboot your
> system will be upgraded.

OK, but it means that all reboots then have to be supervised from the console (which in my situation is not generally feasible). For others reading this who may be similarly stuck, is there some way to undo the default to FedUp kernel (not that it's a problem for me - I think I mentioned earlier that this was a virtual machine, so I have just rolled it back to the earlier F20 state). 
> 
> Did you try to run 'yum update' after upgrade? 

Yes, I always do.

> Was wget updated using yum after upgrade?

No. I fixed this for wget by using downgrade.

Comment 6 Will Woods 2015-01-05 22:26:34 UTC
(In reply to bob mckay from comment #5)
> (In reply to Tomas Hozza from comment #4)
> > You can boot the regular system by choosing the proper entry in the GRUB.
> > FedUp just adds a new one, which is default, that's why after reboot your
> > system will be upgraded.
> 
> OK, but it means that all reboots then have to be supervised from the
> console (which in my situation is not generally feasible). For others
> reading this who may be similarly stuck, is there some way to undo the
> default to FedUp kernel

`fedup --resetbootloader` will remove the System Upgrade entry.

(In reply to bob mckay from comment #3)
> Perhaps fedup isn't using rpmdev-vercmp for the comparison? All I can say
> for sure is that, reliably, at the moment fedup isn't upgrading wget (nor
> samba nor sqlite, which have the same issue of an fc21 version number that
> is lower than fc20). 

fedup is just a wrapper around yum; yum does all the actual version comparisons.

It's possible there are some slow/stuck mirrors that don't yet have the newer F21 wget, or that some other package(s) are messing with the dependencies, but fedup doesn't mess with dependency resolution at all.

Can you attach your /var/log/fedup.log? It will have extra detail that should clarify what the source of the problem is.

(If possible, move that file and re-run fedup to get a fresh log.)

Comment 7 bob mckay 2015-01-07 03:04:49 UTC
Created attachment 977067 [details]
fedup log (installed wrong wget among other things)

fedup.log as requested (includes both the upgrade F20->F21 (starting line 5699), and the preceding upgrade F19->F20).

Comment 8 bob mckay 2015-01-07 03:41:38 UTC
(In reply to Will Woods from comment #6) 
> `fedup --resetbootloader` will remove the System Upgrade entry.

Thanks Will, that is very helpful (I'm sorry, I didn't think to look in fedup, as I assumed that it was a grub issue I should be looking at the grub documentation - my bad).
> 
> (In reply to bob mckay from comment #3)
> > Perhaps fedup isn't using rpmdev-vercmp for the comparison? All I can say
> > for sure is that, reliably, at the moment fedup isn't upgrading wget (nor
> > samba nor sqlite, which have the same issue of an fc21 version number that
> > is lower than fc20). 
> 
> fedup is just a wrapper around yum; yum does all the actual version
> comparisons.

OK, but the issue remains that fedup retained 1.16-3.fc20 despite availability of 1.16.1-1.fc21. Is this the intended behaviour? (It doesn't seem to be what Tomas was expecting.)
> 
> It's possible there are some slow/stuck mirrors that don't yet have the
> newer F21 wget, or that some other package(s) are messing with the
> dependencies, but fedup doesn't mess with dependency resolution at all.

That may well have been part of the issue (I'm in Korea so that my mirrors are all in Asia - and it was over Christmas). However I'm still not seeing anything later than 1.16.1-1.fc21 in 'yum list'. Are you sure there is a 1.16-3.fc21 or later wget in the primary repo (I'm not sure how to check what is in the primary repo as distinct from the mirrors)? 
 
> Can you attach your /var/log/fedup.log? It will have extra detail that
> should clarify what the source of the problem is.

Done. I should explain that there were two different systems (both VMs) that had this problem. One had a separate issue that meant I had to revert to F20 by rolling back the VM. The other was successfully upgraded to F21 (but retained the F20 wget, which I fixed by downgrading wget). The log is from the one that was successfully upgraded to F21. 

> (If possible, move that file and re-run fedup to get a fresh log.)
Sorry, I misunderstood your request at first. Do you need me to re-run fedup on the one that is currently on F20?

Comment 9 Will Woods 2015-01-08 00:56:24 UTC
(In reply to bob mckay from comment #8)
> (In reply to Will Woods from comment #6) 
> > `fedup --resetbootloader` will remove the System Upgrade entry.
> 
> Thanks Will, that is very helpful (I'm sorry, I didn't think to look in
> fedup, as I assumed that it was a grub issue I should be looking at the grub
> documentation - my bad).
> > 
> > (In reply to bob mckay from comment #3)
> > > Perhaps fedup isn't using rpmdev-vercmp for the comparison? All I can say
> > > for sure is that, reliably, at the moment fedup isn't upgrading wget (nor
> > > samba nor sqlite, which have the same issue of an fc21 version number that
> > > is lower than fc20). 
> > 
> > fedup is just a wrapper around yum; yum does all the actual version
> > comparisons.
> 
> OK, but the issue remains that fedup retained 1.16-3.fc20 despite
> availability of 1.16.1-1.fc21. Is this the intended behaviour? (It doesn't
> seem to be what Tomas was expecting.)

That doesn't match the logs, though.

The logs show that you had wget-1.16.1-2.fc20.x86_64 already installed when you ran fedup, but wget-1.16.1-2.fc21 wasn't yet available:

[   446.555] (II) fedup:message() Packages without updates:
[   446.555] (II) fedup:message()   aic94xx-firmware-30-6.fc20.noarch
[   446.555] (II) fedup:message()   celt-0.11.3-1.fc20.x86_64
[   446.556] (II) fedup:message()   fedup-0.9.1-1.fc20.noarch
[   446.556] (II) fedup:message()   gnomebaker-0.6.4-18.fc20.x86_64
[   446.556] (II) fedup:message()   prelink-0.5.0-1.fc20.x86_64
[   446.556] (II) fedup:message()   sqlite-3.8.7.4-1.fc20.x86_64
[   446.556] (II) fedup:message()   wget-1.16.1-2.fc20.x86_64
[   446.557] (II) fedup:message()   1:anaconda-yum-plugins-1.0-10.fc20.noarch
[   446.557] (II) fedup:message()   2:libsmbclient-4.1.14-1.fc20.x86_64
[   446.557] (II) fedup:message()   2:libwbclient-4.1.14-1.fc20.x86_64
[   446.557] (II) fedup:message()   2:samba-client-4.1.14-1.fc20.x86_64
[   446.557] (II) fedup:message()   2:samba-common-4.1.14-1.fc20.x86_64
[   446.558] (II) fedup:message()   2:samba-libs-4.1.14-1.fc20.x86_64
[   446.561] (II) fedup:<module>() /bin/fedup exiting cleanly at Sun Dec 21 05:35:28 2014

Sure enough, if you check the timestamps from the update system:

  https://admin.fedoraproject.org/updates/FEDORA-2014-17194/wget-1.16.1-2.fc20
  https://admin.fedoraproject.org/updates/FEDORA-2014-17134/wget-1.16.1-2.fc21

wget-1.16.1-2.fc20 was pushed to stable on Dec 20.
wget-1.16.1-2.fc21 didn't get pushed until Dec 23.

Which means there was a broken upgrade path for wget (and probably other packages) between Dec 20 and Dec 23, because the F20 updates got pushed out before the F21 updates did.

So when you ran fedup on Dec 21, your system had wget-1.16.1-2, and F21 still had the older version. Fedup doesn't do downgrades, so it skipped wget. Which is the intended behavior.

(There have been requests for a 'distro-sync' mode that would downgrade wget in this case, but I'd much rather we just didn't break upgrade paths in the first place..)

Anyway, if you ran fedup after Dec 23 or so, it should have picked up the new wget. So it should work fine now, barring weird mirror problems.

All you need to do is run the same fedup command as before - it'll keep everything it already downloaded, but also fetch any new updates that have appeared in the meantime.

So: does fedup see the wget update now?

Comment 10 bob mckay 2015-01-08 06:50:46 UTC
Hi Will; yes, fedup sees the wget update now. So the specific problem is now solved. The general problem is not.

There are problems are now with:
Packages without updates:
  celt-0.11.3-1.fc20.x86_64
  ctags-5.8-16.fc20.x86_64
  eclipse-birt-chart-4.3.1-2.fc20.noarch
  fedup-0.9.1-1.fc20.noarch
  firstboot-19.2-1.fc19.x86_64
  libbeagle-0.3.9-11.fc20.x86_64
  par2cmdline-0.4.tbb.20100203-17.fc20.x86_64
  perl-PlRPC-0.2020-15.fc20.noarch
  piccolo2d-1.3.1-3.fc20.noarch
  prelink-0.5.0-1.fc20.x86_64
  python-django14-1.4.16-1.fc20.noarch
  rpmfusion-free-release-20-1.noarch
  rpmfusion-nonfree-release-20-1.noarch
  system-config-lvm-1.1.18-1.fc18.noarch
  xmlgraphics-commons-1.5-2.fc20.noarch
  xorg-x11-drv-r128-6.9.2-1.fc20.x86_64
  1:anaconda-yum-plugins-1.0-10.fc20.noarch

WARNING: problems were encountered during transaction test:
  broken dependencies
    xorg-x11-drv-r128-6.9.2-1.fc20.x86_64 requires xorg-x11-server-Xorg-1.14.4-13.fc20.x86_64

Leaving aside those where there are obvious reasons for the failed upgrades (orphans that I have deliberately chosen to retain, rpmfusion, fedup, firstboot that I wasn't sure whether I should remove), there are still 12 packages with real synchronisation problems. It's clear that a lot of package maintainers are continuing to run their F20 updates ahead of F21. From previous related comments, that appears to be because they are under the misapprehension that fedup will still push the package to the fc21 version, even if it's a downgrade, so that it doesn't matter. Since every package listed here may need checking by most users to make sure it isn't going to break their system, that's a lot of wasted work. If fedup is going to retain a policy to never downgrade (which by the way, I agree with), surely there needs to be a corresponding general policy that the latest release of Fedora should have the latest package releases (even if that means holding back an F20 package release until the F21 release is ready) except in unavoidable cases - and those cases should require special approval.

Comment 11 Tomáš Hozza 2015-01-08 08:42:39 UTC
(In reply to bob mckay from comment #10)
> Leaving aside those where there are obvious reasons for the failed upgrades
> (orphans that I have deliberately chosen to retain, rpmfusion, fedup,
> firstboot that I wasn't sure whether I should remove), there are still 12
> packages with real synchronisation problems. It's clear that a lot of
> package maintainers are continuing to run their F20 updates ahead of F21.
> From previous related comments, that appears to be because they are under
> the misapprehension that fedup will still push the package to the fc21
> version, even if it's a downgrade, so that it doesn't matter. Since every
> package listed here may need checking by most users to make sure it isn't
> going to break their system, that's a lot of wasted work. If fedup is going
> to retain a policy to never downgrade (which by the way, I agree with),
> surely there needs to be a corresponding general policy that the latest
> release of Fedora should have the latest package releases (even if that
> means holding back an F20 package release until the F21 release is ready)
> except in unavoidable cases - and those cases should require special
> approval.

I think the special case should be e.g. security update of a component.

Anyway I agree with you that the F20 update should be waiting for F21 to update first, however I think this should be automatically handled by bodhi or rel-eng tooling.

If I as a maintainer push an update in bodhi with the same version-release (e.g. 1.16.1-1 for F20, F21, rawhide) and use the stable karma feature I have no control of how updates are pushed to stable. This should be handles automatically by bodhi and e.g. F20 update should not be pushed to stable until F21 is, if it would break the update path.

I think we should move this Bug to bodhi or release-engineering.

Comment 12 Will Woods 2015-01-08 21:48:31 UTC
Bugzilla isn't meant for policy discussions. Please try the mailing lists instead.

fedup's code is working as intended, and the temporary problem with the wget update is fixed. So there's nothing for me to fix here.