Bug 896023

Summary: The upgrade doesn't continue after reboot, possibly because systemd doesn't see system-update.target
Product: [Fedora] Fedora Reporter: David Howells <dhowells>
Component: fedupAssignee: Will Woods <wwoods>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 17CC: rstrode, tflink, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: fedup-0.7.3-5.fc17 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-05-15 03:29:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
grub2 config
none
lsinitrd on /boot/initramfs-fedup.img
none
Output of "journalctl -a" with "systemd.log_level=debug"
none
dmesg showing "plymouth.debug=stream:/dev/kmsg" none

Description David Howells 2013-01-16 13:46:14 UTC
Description of problem:

I ran fedup to set up an upgrade from F17 to F18.  It downloaded all the packages okay and said I needed to reboot.  I rebooted and selected the upgrade option in the grub menu.  So far, so good.  However, the boot proceeded directly to the usual KDM login screen.

Investigating shows that the expected F18 kernel is running (3.6.10-4.fc18.x86_64), though the new kernel doesn't manage to load the module for my network interfaces (which shouldn't be a problem).  /proc/cmdline shows:

BOOT_IMAGE=/vmlinuz-fedup root=UUID=42892800-af7a-46a2-a5dd-6f2920d54955 ro rd.lvm=0 rd.dm=0 rd.md.uuid=d7919697:a7814731:82cdd3e4:60041633 SYSFONT=latarcyrheb-sun16 KEYTABLE=uk rd.luks=0 LANG=en_US.UTF-8 rd.md.uuid=f15b9681:b8f6126e:3108f1db:5daef8ec rhgb quiet upgrade systemd.unit=system-upgrade.target plymouth.splash=fedup

which looks reasonable.  I can see the system-upgrade.target file in the expected place:

    /usr/lib/systemd/system/system-upgrade.target

However, doing systemctl --all and grepping for system-upgrade only shows this:

    system\x...\x2droot.mount loaded active   mounted       /system-upgrade-root

so it looks like systemd isn't finding the target it's supposed to head towards.  Note that the pair of \x2d in this seem really odd as they ought to be just minus signs.

/system-upgrade and /system-upgrade-root seem to have reasonable stuff contained therein.

Version-Release number of selected component (if applicable):

    fedup-0.7.2-1.fc17.noarch
    systemd-44-23.fc17.x86_64
    systemd-44-23.fc17.i686

Comment 1 David Howells 2013-01-16 17:24:38 UTC
Or maybe it does see it:

[root@warthog system]# systemctl status system-upgrade.target
system-upgrade.target - System Upgrade
          Loaded: loaded (/usr/lib/systemd/system/system-upgrade.target; static)
          Active: inactive (dead)
            Docs: http://freedesktop.org/wiki/Software/systemd/SystemUpdates
                  man:systemd.special(7)

Comment 2 David Howells 2013-01-16 17:30:41 UTC
According to:

   http://freedesktop.org/wiki/Software/systemd/SystemUpdates

the system upgrade should be automatically triggered by a symlink:

    /system-update

with no need to put anything on the kernel command line to persuade systemd to do stuff.  However, fedup seems to have made this:

    /system-upgrade

Comment 3 David Howells 2013-01-16 17:39:08 UTC
Created attachment 679725 [details]
grub2 config

Comment 4 David Howells 2013-01-16 17:49:16 UTC
Created attachment 679732 [details]
lsinitrd on /boot/initramfs-fedup.img

initramfs-fedup.img contains:

-rw-r--r--   1 root     root          162 Jan  9 19:58 etc/systemd/system/upgrade.target

rather than etc/systemd/system-upgrade.target is that correct, given the option on the kernel command line:

    systemd.unit=system-upgrade.target

Comment 5 David Howells 2013-01-16 18:42:01 UTC
Created attachment 679746 [details]
Output of "journalctl -a" with "systemd.log_level=debug"

Comment 6 David Howells 2013-01-16 19:20:04 UTC
Created attachment 679754 [details]
dmesg showing "plymouth.debug=stream:/dev/kmsg"

Comment 7 Will Woods 2013-01-16 20:33:55 UTC
(In reply to comment #2)
Don't mix up SystemUpdates and fedup - they're totally separate.

Also don't mix up the initramfs 'upgrade.target' and the host's 'system-upgrade.target'. They're two different things - and they're both needed.

The fedup boot process is a bit weird: it starts in the initramfs, then starts the host system to mount disks, then goes *back* to initramfs to run the upgrade.

But I'm not sure that's necessarily important here...

What's more interesting is that you have an encrpyted /home, which should probably get unlocked, but there's apparently no prompt for it?

Comment 8 David Howells 2013-01-17 11:45:07 UTC
Indeed, I see no prompt.  Looking through the journal, the password entry service tried to run and failed, and after a bit, plymouthd segfaulted.

Comment 9 David Howells 2013-01-17 11:45:34 UTC
Running without rhgb on the kernel command line worked.

Comment 10 Ray Strode [halfline] 2013-01-18 15:07:54 UTC
So i spent some time looking at this, and believe I know the problem.

In order for plymouth to be able to display text in its graphical mode, it needs access to font rendering libraries (and fonts, etc).  These libraries currently include, pango, fontconfig, freetype, and cairo.  We don't want to ship all of that stuff in the initrd, so we instead put all font rendering functionality in a "label" plugin that gets loaded after the root filesystem is mounted.

plymouth currently makes no ABI gaurantees from release to release.  That normally doesn't matter, because plymouthd and the label plugin are normally built together.

In this situation, we're using an f18 plymouth with an f17 label plugin.  There is an abi incompatibility, because the newer version of the label plugin has additional virtual functions. The newer label plugin is backward compatible with the older plymouthd, but the older label plugin isn't forward compatible with the newer plymouthd.

Comment 11 Ray Strode [halfline] 2013-01-18 15:13:20 UTC
The easiest, though not really so elegant, fix is probably to ship the newer label plugin with fedup in its initrd and copy the label plugin to the root fs before doing the switch root.

alternatively we could do a plymouth build for f17 that fixed the abi incompatibility and then we could force users to upgrade their plymouth before starting the fedup process.

Comment 12 Fedora Update System 2013-03-15 23:33:56 UTC
fedup-0.7.3-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-1.fc17

Comment 13 Fedora Update System 2013-03-17 00:53:38 UTC
Package fedup-0.7.3-1.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.7.3-1.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-3956/fedup-0.7.3-1.fc17
then log in and leave karma (feedback).

Comment 14 Will Woods 2013-04-22 16:04:02 UTC
Can anyone confirm that this update fixes upgrades on systems with /home (and only /home) encrypted?

Comment 15 Fedora Update System 2013-04-30 14:50:29 UTC
fedup-0.7.3-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.3-3.fc18

Comment 16 Fedora Update System 2013-04-30 14:54:30 UTC
fedup-0.7.3-3.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-3.fc17

Comment 17 Fedora Update System 2013-04-30 15:36:48 UTC
fedup-0.7.3-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc17

Comment 18 Fedora Update System 2013-04-30 15:37:34 UTC
fedup-0.7.3-4.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc18

Comment 19 Fedora Update System 2013-04-30 15:38:25 UTC
fedup-0.7.3-4.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc19

Comment 20 Fedora Update System 2013-05-15 03:29:17 UTC
fedup-0.7.3-4.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2013-06-07 16:42:28 UTC
fedup-0.7.3-5.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-5.fc17

Comment 22 Fedora Update System 2013-06-08 03:33:57 UTC
fedup-0.7.3-4.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 23 Fedora Update System 2013-06-23 05:55:39 UTC
fedup-0.7.3-5.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.