|Summary:||Upgrade to F23 crashes early in upgrade boot|
|Product:||[Fedora] Fedora||Reporter:||Adam Williamson <awilliam>|
|Component:||fedup||Assignee:||Will Woods <wwoods>|
|Status:||CLOSED ERRATA||QA Contact:||Fedora Extras Quality Assurance <extras-qa>|
|Version:||23||CC:||a9016009, aruizrui, balay, jsedlak, kparal, robatino, robert.roth.off, satellitgo, tflink, vondruch, wwoods, zbyszek|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2015-10-16 15:19:48 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
|Bug Depends On:|
Description Adam Williamson 2015-07-22 22:23:19 UTC
Testing upgrade from F21 to F23 (I don't think the source release matters in this case), after downloading the updates and rebooting, the system crashes early in boot of the upgrade environment. A couple of suspicious errors are visible: systemd: [/etc/systemd/system/system-upgrade-shell.service:8] Couldn't unescape assignment, ignoring: TERM=linux PS1=system-update:\w\$\x20 systemd: Caught <ABRT>, dumped core as pid 132. systemd:: Freezing execution. and that's about it. Per https://fedorahosted.org/fesco/ticket/1463 there seem to be moves afoot to replace fedup with a dnf plugin, but this seems terribly late for F23. Per all current policies / procedures / etc, fedup is the official upgrade mechanism and still blocks the release, so nominating this as a Beta blocker: "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed. ... This criterion applies to the recommended upgrade mechanisms only." - https://fedoraproject.org/wiki/Fedora_23_Beta_Release_Criteria#Upgrade_requirements 'recommended upgrade mechanisms' is a link to https://fedoraproject.org/wiki/Upgrading , which as I write this, says: "Upgrading with FedUp Recommended Upgrade Method This is the recommended method to upgrade your Fedora system. For instructions on upgrading, refer to the FedUp page."
Comment 2 Adam Williamson 2015-07-22 22:47:29 UTC
Confirmed that this also occurs with F21.
Comment 3 Will Woods 2015-07-23 16:11:54 UTC
Short answer: fedup is abandoned. Docs/criteria/tests/etc will need to be adapted to the New Upgrade Method.. once we figure out exactly what that is. (dnf-plugin-fedup is the current prototype but it might get integrated into upstream DNF; we should have a clearer plan for it in a couple of days.) (In reply to Adam Williamson from comment #0) > Testing upgrade from F21 to F23 (I don't think the source release matters in > this case), after downloading the updates and rebooting, the system crashes > early in boot of the upgrade environment. A couple of suspicious errors are > visible: > > systemd: [/etc/systemd/system/system-upgrade-shell.service:8] Couldn't > unescape assignment, ignoring: TERM=linux PS1=system-update:\w\$\x20 > systemd: Caught <ABRT>, dumped core as pid 132. > systemd:: Freezing execution. > > and that's about it. Hey look, a systemd crash. So here's the deal: fedup's boot mechanism is using systemd in an unsupported (and, honestly, inherently broken) way, which inevitably causes these kind of problems. The systemd maintainers have explicitly rejected supporting it, and I completely agree with their reasoning. It's just not fixable. This is the main reason fedup is deprecated and abandoned for F23 and onward.  For reference, here's the discussion in systemd-devel: http://lists.freedesktop.org/archives/systemd-devel/2015-March/029030.html http://lists.freedesktop.org/archives/systemd-devel/2015-April/031013.html > Per https://fedorahosted.org/fesco/ticket/1463 there seem to be moves afoot > to replace fedup with a dnf plugin, but this seems terribly late for F23. There's really no discussion to be had, here. fedup is unfixably broken; upstream can't (and won't) support it. There's no choice but to abandon and replace it. I *have* been telling people this since May: https://lists.fedoraproject.org/pipermail/devel/2015-May/210905.html I also wrote a prototype replacement, and I've been working to get it integrated into upstream DNF. systemd upstream is also helping ensure this works: http://lists.freedesktop.org/archives/systemd-devel/2015-July/033605.html As far as I'm concerned, that's the way forward. > Per all current policies / procedures / etc, fedup is the official upgrade > mechanism and still blocks the release, so nominating this as a Beta blocker: Yeah, that's going to need to be changed. Fedup should not be the official upgrade mechanism. I'm sorry, but it's just not supportable. Consider it abandoned by upstream.
Comment 4 Adam Williamson 2015-07-23 18:00:57 UTC
"telling people this since May" is fine, but there's a reason we have a Change process. QA looks at the Changes to know what to test. docs looks at the Changes to know what to document. Users look at the Changes to know what's coming. The reason we have a process for that is that experience tells us it doesn't work very well to try and plan our product releases by remembering everything anyone sent to any mailing list we have. We can deal with testing a new mechanism, but it has to *be* there in some sense. A proof of concept in a github repo is a long way from a thing that's integrated into our product and properly communicated to our users, and we need to get from point A to point B in three months, and I'd be more confident about us achieving that if we were using the process that was put in place to do it.
Comment 5 Will Woods 2015-07-23 23:01:36 UTC
Comment 6 Adam Williamson 2015-07-27 16:54:52 UTC
Discussed at 2015-07-27 blocker review meeting: http://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-07-27/f23-blocker-review.2015-07-27-16.00.log.txt . Accepted as a Beta blocker. As things stand, fedup is still the 'recommended upgrade mechanism' for upgrades to Fedora 23. If it stops being that, this particular bug will no longer merit blocker status (but whatever replaces it as the 'recommended upgrade mechanism' will have to work).
Comment 7 Adam Williamson 2015-09-04 02:23:00 UTC
As the Change to DNF upgrades was accepted by FESCo and is well underway, I'm re-proposing this for discussion, with the expectation it will be rendered not-a-blocker. I've been updating wiki pages and stuff, so the Upgrading page now lists DNF as the 'supported' method for upgrades to F23.
Comment 8 Will Woods 2015-09-04 20:54:17 UTC
FWIW, the code to build upgrade.img has been removed from lorax upstream, so after the next F23 build of lorax it won't even be possible to use fedup for upgrades to F23. (See https://github.com/rhinstaller/lorax/commit/d9c23d1)
Comment 9 Will Woods 2015-09-08 15:07:07 UTC
*** Bug 1260319 has been marked as a duplicate of this bug. ***
Comment 10 Robert Roth 2015-09-08 15:36:42 UTC
As fedup does not work from F22 to F23, and fedup's only task is to upgrade the system (and it fails at that), shouldn't it be removed from the F22 repository?
Comment 11 Adam Williamson 2015-09-08 15:37:25 UTC
The new tool obsoletes it. Things are not removed from the frozen release repository, ever, by policy (except maybe in case of some sort of terrible legal issue).
Comment 12 Will Woods 2015-09-08 16:44:25 UTC
Right now the packages are in updates-testing, so: [wwoods@metroid ~]$ sudo dnf --enablerepo=updates-testing update Last metadata expiration check performed 1:32:45 ago on Tue Sep 8 11:08:47 2015. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: dnf-plugin-system-upgrade noarch 0.4.0-1.fc22 updates-testing 41 k replacing fedup.noarch 0.9.2-1.fc22 [etc] So, yes, fedup will be removed *from your system* once dnf-plugin-system-upgrade is installed. I plan to officially retire the fedup and fedup-dracut packages from the distribution once dnf-plugin-system-upgrade makes it to the main repos.
Comment 13 Vít Ondruch 2015-09-09 15:02:03 UTC
Why there is not some fedup compatibility layer? IOW fedup was greatly marketed, so why not keep using this command whatever runs in background?
Comment 14 Adam Williamson 2015-09-09 15:04:41 UTC
[adamw@adam openqa_fedora_tools (dnf-update)]$ rpm -qf /usr/bin/fedup dnf-plugin-system-upgrade-0.4.0-1.fc23.noarch
Comment 15 Vít Ondruch 2015-09-09 15:28:50 UTC
(In reply to firstname.lastname@example.org from comment #14) > [adamw@adam openqa_fedora_tools (dnf-update)]$ rpm -qf /usr/bin/fedup > dnf-plugin-system-upgrade-0.4.0-1.fc23.noarch Ah, ok ... no worries then ;)
Comment 16 Adam Williamson 2015-09-10 19:18:03 UTC
Discussed at 2015-09-10 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-10/f23-blocker-review.2015-09-10-16.00.log.txt . This is now demoted from blocker status, as fedup is no longer the 'supported' method for upgrades to Fedora 23, per https://fedoraproject.org/wiki/Changes/DNF_System_Upgrades and https://fedoraproject.org/wiki/Upgrading (which I've updated to reflect the accepted Change). wwoods may wish to close the bug, we'd have no objection to him doing so.
Comment 17 Alberto Ruiz 2015-09-18 16:03:20 UTC
So last night I tried to use fedup to upgrade from F22-F23 and ended up stumbling upon this very bug. If we are going to present something new to our users for F22->F23, shouldn't fedup help the user upgrade even if it doesn't work anymore? It could just ask you to install the dnf plugin and run dnf system-upgrade (I understand that rework fedup as a wrap of the new dnf functionality could be a bit of a challenge as the semantics of the command line are different). At the very least it should show you a warning that fedup is deprecated and where to go from there so that the user can still upgrade which is why the run fedup in the first place. As is, users unaware of fedup being deprecated will end up in users wasting quite a bit of time (namely hours) downloading all the upgrades and not realizing that the thing will not work until they reboot, with no obvious way to know how to go ahead.
Comment 18 Will Woods 2015-09-18 19:38:18 UTC
(In reply to Alberto Ruiz from comment #17) > So last night I tried to use fedup to upgrade from F22-F23 and ended up > stumbling upon this very bug. > > If we are going to present something new to our users for F22->F23, > shouldn't fedup help the user upgrade even if it doesn't work anymore? The replacement package was just pushed to the stable repos today. So if you update your system after today, you get the new package: ======================================================================== Package Arch Version Repository Size ======================================================================== Installing: dnf-plugin-system-upgrade noarch 0.4.1-1.fc22 updates 44 k replacing fedup.noarch 0.9.2-1.fc22 python2-dnf-plugin-system-upgrade noarch 0.4.1-1.fc22 updates 25 k > It could just ask you to install the dnf plugin and run dnf system-upgrade > (I understand that rework fedup as a wrap of the new dnf functionality could > be a bit of a challenge as the semantics of the command line are different). > > At the very least it should show you a warning that fedup is deprecated and > where to go from there so that the user can still upgrade which is why the > run fedup in the first place. The new package provides a new /usr/bin/fedup which does exactly that: [wwoods@metroid ~]$ sudo fedup --network 23 fedup has been replaced by 'dnf system-upgrade'. Use that instead. Redirecting to 'dnf system-upgrade download --releasever 23': [etc...] [wwoods@metroid ~]$ sudo fedup --network 23 --product workstation fedup has been replaced by 'dnf system-upgrade'. Use that instead. fedup: warning: '--product' is not used anymore. ignoring. Redirecting to 'dnf system-upgrade download --releasever 23': [etc...] > As is, users unaware of fedup being deprecated will end up in users wasting > quite a bit of time (namely hours) downloading all the upgrades and not > realizing that the thing will not work until they reboot, with no obvious > way to know how to go ahead. Well. The replacement package is now live, so anyone who updates won't be using the old fedup anymore. For anyone who tries using the old fedup, post-Beta builds will not have upgrade.img anymore, so it won't even be possible to hit this bug after that. (That was *supposed* to happen for Beta, but since this bug was closed nobody noticed that upgrade.img was still getting built. Alas; I'm reopening this bug until upgrade.img is dropped from the repo that fedup tries to use by default.)
Comment 19 Fedora Update System 2015-09-18 19:45:04 UTC
lorax-23.18-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-15675
Comment 20 Will Woods 2015-09-21 18:25:05 UTC
*** Bug 1264942 has been marked as a duplicate of this bug. ***
Comment 21 Adam Williamson 2015-09-23 15:10:37 UTC
Discussed at 2015-09-22 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-22/f23-blocker-review.2015-09-22-16.00.html . Rejected as a blocker: we agreed that so long as fedup has been obsoleted in all stable releases, the presence of the upgrade.img's in the trees would be unfortunate but not severe enough to block the release. Sorry, Will, I just realized we forgot to discuss this as an FE. I'll throw an FE nomination on here now just in case it doesn't get pushed before freeze.
Comment 22 Adam Williamson 2015-09-28 22:16:31 UTC
Discussed at 2015-09-28 freeze exception review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-28/f23-blocker-review.2015-09-28-16.01.html . Accepted as a freeze exception issue: dropping upgrade.img images from the tree would avoid people with old fedup installs trying upgrades and wasting time and bandwidth, cannot cleanly be done post-release, and should be a safe fix.
Comment 23 Fedora Update System 2015-10-16 15:19:42 UTC
lorax-23.18-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.