Bug 1245838 - Upgrade to F23 crashes early in upgrade boot
Summary: Upgrade to F23 crashes early in upgrade boot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: fedup
Version: 23
Hardware: x86_64
OS: All
unspecified
high
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeException
: 1260319 1264942 (view as bug list)
Depends On:
Blocks: F23FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2015-07-22 22:23 UTC by Adam Williamson
Modified: 2015-10-16 15:19 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-16 15:19:48 UTC


Attachments (Terms of Use)
fedup.log (175.14 KB, text/plain)
2015-07-22 22:27 UTC, Adam Williamson
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1266901 None None None Never

Internal Links: 1266901

Description Adam Williamson 2015-07-22 22:23:19 UTC
Testing upgrade from F21 to F23 (I don't think the source release matters in this case), after downloading the updates and rebooting, the system crashes early in boot of the upgrade environment. A couple of suspicious errors are visible:

systemd[1]: [/etc/systemd/system/system-upgrade-shell.service:8] Couldn't unescape assignment, ignoring: TERM=linux PS1=system-update:\w\$\x20
systemd[1]: Caught <ABRT>, dumped core as pid 132.
systemd:[1]: Freezing execution.

and that's about it.

Per https://fedorahosted.org/fesco/ticket/1463 there seem to be moves afoot to replace fedup with a dnf plugin, but this seems terribly late for F23. Per all current policies / procedures / etc, fedup is the official upgrade mechanism and still blocks the release, so nominating this as a Beta blocker:

"For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed. 
... 
This criterion applies to the recommended upgrade mechanisms only." - https://fedoraproject.org/wiki/Fedora_23_Beta_Release_Criteria#Upgrade_requirements

'recommended upgrade mechanisms' is a link to https://fedoraproject.org/wiki/Upgrading , which as I write this, says:

"Upgrading with FedUp

Recommended Upgrade Method
This is the recommended method to upgrade your Fedora system. For instructions on upgrading, refer to the FedUp page."

Comment 1 Adam Williamson 2015-07-22 22:27:08 UTC
Created attachment 1055095 [details]
fedup.log

Comment 2 Adam Williamson 2015-07-22 22:47:29 UTC
Confirmed that this also occurs with F21.

Comment 3 Will Woods 2015-07-23 16:11:54 UTC
Short answer: fedup is abandoned. Docs/criteria/tests/etc will need to be adapted to the New Upgrade Method.. once we figure out exactly what that is.

(dnf-plugin-fedup is the current prototype but it might get integrated into upstream DNF; we should have a clearer plan for it in a couple of days.)

(In reply to Adam Williamson from comment #0)
> Testing upgrade from F21 to F23 (I don't think the source release matters in
> this case), after downloading the updates and rebooting, the system crashes
> early in boot of the upgrade environment. A couple of suspicious errors are
> visible:
> 
> systemd[1]: [/etc/systemd/system/system-upgrade-shell.service:8] Couldn't
> unescape assignment, ignoring: TERM=linux PS1=system-update:\w\$\x20
> systemd[1]: Caught <ABRT>, dumped core as pid 132.
> systemd:[1]: Freezing execution.
> 
> and that's about it.

Hey look, a systemd crash.

So here's the deal: fedup's boot mechanism is using systemd in an unsupported (and, honestly, inherently broken) way, which inevitably causes these kind of problems.

The systemd maintainers have explicitly rejected supporting it[1], and I completely agree with their reasoning. It's just not fixable.

This is the main reason fedup is deprecated and abandoned for F23 and onward.

[1] For reference, here's the discussion in systemd-devel:
  http://lists.freedesktop.org/archives/systemd-devel/2015-March/029030.html
  http://lists.freedesktop.org/archives/systemd-devel/2015-April/031013.html

> Per https://fedorahosted.org/fesco/ticket/1463 there seem to be moves afoot
> to replace fedup with a dnf plugin, but this seems terribly late for F23.

There's really no discussion to be had, here. fedup is unfixably broken; upstream can't (and won't) support it. There's no choice but to abandon and replace it.

I *have* been telling people this since May:

  ​https://lists.fedoraproject.org/pipermail/devel/2015-May/210905.html

I also wrote a prototype replacement, and I've been working to get it integrated into upstream DNF. systemd upstream is also helping ensure this works:

  http://lists.freedesktop.org/archives/systemd-devel/2015-July/033605.html

As far as I'm concerned, that's the way forward.

> Per all current policies / procedures / etc, fedup is the official upgrade
> mechanism and still blocks the release, so nominating this as a Beta blocker:

Yeah, that's going to need to be changed. Fedup should not be the official upgrade mechanism. I'm sorry, but it's just not supportable. Consider it abandoned by upstream.

Comment 4 Adam Williamson 2015-07-23 18:00:57 UTC
"telling people this since May" is fine, but there's a reason we have a Change process. QA looks at the Changes to know what to test. docs looks at the Changes to know what to document. Users look at the Changes to know what's coming. The reason we have a process for that is that experience tells us it doesn't work very well to try and plan our product releases by remembering everything anyone sent to any mailing list we have.

We can deal with testing a new mechanism, but it has to *be* there in some sense. A proof of concept in a github repo is a long way from a thing that's integrated into our product and properly communicated to our users, and we need to get from point A to point B in three months, and I'd be more confident about us achieving that if we were using the process that was put in place to do it.

Comment 6 Adam Williamson 2015-07-27 16:54:52 UTC
Discussed at 2015-07-27 blocker review meeting: http://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-07-27/f23-blocker-review.2015-07-27-16.00.log.txt . Accepted as a Beta blocker. As things stand, fedup is still the 'recommended upgrade mechanism' for upgrades to Fedora 23. If it stops being that, this particular bug will no longer merit blocker status (but whatever replaces it as the 'recommended upgrade mechanism' will have to work).

Comment 7 Adam Williamson 2015-09-04 02:23:00 UTC
As the Change to DNF upgrades was accepted by FESCo and is well underway, I'm re-proposing this for discussion, with the expectation it will be rendered not-a-blocker. I've been updating wiki pages and stuff, so the Upgrading page now lists DNF as the 'supported' method for upgrades to F23.

Comment 8 Will Woods 2015-09-04 20:54:17 UTC
FWIW, the code to build upgrade.img has been removed from lorax upstream, so after the next F23 build of lorax it won't even be possible to use fedup for upgrades to F23.

(See https://github.com/rhinstaller/lorax/commit/d9c23d1)

Comment 9 Will Woods 2015-09-08 15:07:07 UTC
*** Bug 1260319 has been marked as a duplicate of this bug. ***

Comment 10 Robert Roth 2015-09-08 15:36:42 UTC
As fedup does not work from F22 to F23, and fedup's only task is to upgrade the system (and it fails at that), shouldn't it be removed from the F22 repository?

Comment 11 Adam Williamson 2015-09-08 15:37:25 UTC
The new tool obsoletes it. Things are not removed from the frozen release repository, ever, by policy (except maybe in case of some sort of terrible legal issue).

Comment 12 Will Woods 2015-09-08 16:44:25 UTC
Right now the packages are in updates-testing, so:

[wwoods@metroid ~]$ sudo dnf --enablerepo=updates-testing update
Last metadata expiration check performed 1:32:45 ago on Tue Sep  8 11:08:47 2015.
Dependencies resolved.
================================================================================
 Package                      Arch   Version              Repository       Size
================================================================================
Installing:
 dnf-plugin-system-upgrade    noarch 0.4.0-1.fc22         updates-testing  41 k
     replacing  fedup.noarch 0.9.2-1.fc22
[etc]

So, yes, fedup will be removed *from your system* once dnf-plugin-system-upgrade is installed.

I plan to officially retire the fedup and fedup-dracut packages from the distribution once dnf-plugin-system-upgrade makes it to the main repos.

Comment 13 Vít Ondruch 2015-09-09 15:02:03 UTC
Why there is not some fedup compatibility layer? IOW fedup was greatly marketed, so why not keep using this command whatever runs in background?

Comment 14 Adam Williamson 2015-09-09 15:04:41 UTC
[adamw@adam openqa_fedora_tools (dnf-update)]$ rpm -qf /usr/bin/fedup
dnf-plugin-system-upgrade-0.4.0-1.fc23.noarch

Comment 15 Vít Ondruch 2015-09-09 15:28:50 UTC
(In reply to awilliam@redhat.com from comment #14)
> [adamw@adam openqa_fedora_tools (dnf-update)]$ rpm -qf /usr/bin/fedup
> dnf-plugin-system-upgrade-0.4.0-1.fc23.noarch

Ah, ok ... no worries then ;)

Comment 16 Adam Williamson 2015-09-10 19:18:03 UTC
Discussed at 2015-09-10 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-10/f23-blocker-review.2015-09-10-16.00.log.txt . This is now demoted from blocker status, as fedup is no longer the 'supported' method for upgrades to Fedora 23, per https://fedoraproject.org/wiki/Changes/DNF_System_Upgrades and https://fedoraproject.org/wiki/Upgrading (which I've updated to reflect the accepted Change).

wwoods may wish to close the bug, we'd have no objection to him doing so.

Comment 17 Alberto Ruiz 2015-09-18 16:03:20 UTC
So last night I tried to use fedup to upgrade from F22-F23 and ended up stumbling upon this very bug.

If we are going to present something new to our users for F22->F23, shouldn't fedup help the user upgrade even if it doesn't work anymore?

It could just ask you to install the dnf plugin and run dnf system-upgrade (I understand that rework fedup as a wrap of the new dnf functionality could be a bit of a challenge as the semantics of the command line are different).

At the very least it should show you a warning that fedup is deprecated and where to go from there so that the user can still upgrade which is why the run fedup in the first place.

As is, users unaware of fedup being deprecated will end up in users wasting quite a bit of time (namely hours) downloading all the upgrades and not realizing that the thing will not work until they reboot, with no obvious way to know how to go ahead.

Comment 18 Will Woods 2015-09-18 19:38:18 UTC
(In reply to Alberto Ruiz from comment #17)
> So last night I tried to use fedup to upgrade from F22-F23 and ended up
> stumbling upon this very bug.
> 
> If we are going to present something new to our users for F22->F23,
> shouldn't fedup help the user upgrade even if it doesn't work anymore?

The replacement package was just pushed to the stable repos today.
So if you update your system after today, you get the new package:

  ========================================================================
   Package                           Arch   Version        Repository Size
  ========================================================================
  Installing:
   dnf-plugin-system-upgrade         noarch 0.4.1-1.fc22   updates    44 k
       replacing  fedup.noarch 0.9.2-1.fc22
   python2-dnf-plugin-system-upgrade noarch 0.4.1-1.fc22   updates    25 k

> It could just ask you to install the dnf plugin and run dnf system-upgrade
> (I understand that rework fedup as a wrap of the new dnf functionality could
> be a bit of a challenge as the semantics of the command line are different).
> 
> At the very least it should show you a warning that fedup is deprecated and
> where to go from there so that the user can still upgrade which is why the
> run fedup in the first place.

The new package provides a new /usr/bin/fedup which does exactly that:

  [wwoods@metroid ~]$ sudo fedup --network 23
  fedup has been replaced by 'dnf system-upgrade'. Use that instead.
  Redirecting to 'dnf system-upgrade download --releasever 23':
  [etc...]


  [wwoods@metroid ~]$ sudo fedup --network 23 --product workstation
  fedup has been replaced by 'dnf system-upgrade'. Use that instead.
  fedup: warning: '--product' is not used anymore. ignoring.
  Redirecting to 'dnf system-upgrade download --releasever 23':
  [etc...]

> As is, users unaware of fedup being deprecated will end up in users wasting
> quite a bit of time (namely hours) downloading all the upgrades and not
> realizing that the thing will not work until they reboot, with no obvious
> way to know how to go ahead.

Well. The replacement package is now live, so anyone who updates won't be using the old fedup anymore. For anyone who tries using the old fedup, post-Beta builds will not have upgrade.img anymore, so it won't even be possible to hit this bug after that.

(That was *supposed* to happen for Beta, but since this bug was closed nobody noticed that upgrade.img was still getting built. Alas; I'm reopening this bug until upgrade.img is dropped from the repo that fedup tries to use by default.)

Comment 19 Fedora Update System 2015-09-18 19:45:04 UTC
lorax-23.18-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-15675

Comment 20 Will Woods 2015-09-21 18:25:05 UTC
*** Bug 1264942 has been marked as a duplicate of this bug. ***

Comment 21 Adam Williamson 2015-09-23 15:10:37 UTC
Discussed at 2015-09-22 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-22/f23-blocker-review.2015-09-22-16.00.html . Rejected as a blocker: we agreed that so long as fedup has been obsoleted in all stable releases, the presence of the upgrade.img's in the trees would be unfortunate but not severe enough to block the release.

Sorry, Will, I just realized we forgot to discuss this as an FE. I'll throw an FE nomination on here now just in case it doesn't get pushed before freeze.

Comment 22 Adam Williamson 2015-09-28 22:16:31 UTC
Discussed at 2015-09-28 freeze exception review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-28/f23-blocker-review.2015-09-28-16.01.html . Accepted as a freeze exception issue: dropping upgrade.img images from the tree would avoid people with old fedup installs trying upgrades and wasting time and bandwidth, cannot cleanly be done post-release, and should be a safe fix.

Comment 23 Fedora Update System 2015-10-16 15:19:42 UTC
lorax-23.18-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.