Bug 957783

Created attachment 741516 [details]
complete console output from fedup run

Tested fedup-0.7.3-3.fc18.noarch from koji build today with Minimal and Gnome Installs of i686 and x86_64 using both qemu/kvm and virtualbox.  The need to force power off still exists in fedup-0.7.3-3.fc18.noarch.  Otherwise a successful F18 --> f19 upgrade was achieved for both arches with fedup.

Created attachment 741785 [details]
fedup-0.7.3-3.fc18.noarch screenshot at "freeze"

Adding 'rd.upgrade.debugshell' (enables something like systemd-debug-shell.service) lets me debug this further.

shutdown.target, umount.target, final.target are all 'active'.
reboot.target and systemd-reboot.service are dead but 'start'-ing.

'systemctl dump' causes systemd to segfault and dump core:
[   70.625258] systemd[1]: segfault at 8 ip 00007f4f99e9e205 sp 00007fff4a230b40 error 4 in systemd[7f4f99e66000+f3000]
Caught <SEGV>, dumped core as pid 652.
Freezing execution.

Created attachment 741966 [details]
`systemctl --full --all` output

Created attachment 741967 [details]
systemctl --full output

This is just 'systemctl --full', which is much easier to read.

Here's the short version:
UNIT                          LOAD   ACTIVE       SUB          JOB  
-.mount                       loaded active       mounted      stop 
mnt.mount                     loaded active       mounted           
sysroot-proc.mount            loaded active       mounted           
sysroot.mount                 loaded active       mounted           
initrd-switch-root.service    loaded deactivating stop-sigterm stop 
plymouth-reboot.service       loaded inactive     dead         start
plymouth-start.service        loaded active       running           
systemd-modules-load.service  loaded failed       failed            
systemd-reboot.service        loaded inactive     dead         start
systemd-udev-trigger.service  loaded active       exited            
systemd-udevd.service         loaded active       running           
upgrade-debug-shell.service   loaded active       running           
upgrade-switch-root.service   error  deactivating stop-sigkill stop
systemd-journald.socket       loaded active       listening    stop 
systemd-udevd-kernel.socket   loaded active       listening         
final.target                  loaded active       active            
reboot.target                 loaded inactive     dead         start
shutdown.target               loaded active       active            
umount.target                 loaded active       active            

Note that 'systemctl --force --no-block reboot' does cause the system to reboot as expected.

Any chance you could post the bt from the coredump?

Created attachment 742001 [details]
systemd core dump

Created attachment 742026 [details]
'bt full' output

Core was generated by `/usr/lib/systemd/systemd --switched-root --system --deserialize 25'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fabe4966e7b in raise () from /lib64/libpthread.so.0
#0  0x00007fabe4966e7b in raise () from /lib64/libpthread.so.0
#1  0x00007fabe64a42f4 in crash (sig=11) at src/core/main.c:140
#2  <signal handler called>
#3  mount_dump (u=0x7fabe7edc920, f=0x7fabe7ee8f40, prefix=0x7fabe7ca3d00 "\t") at src/core/mount.c:828
#4  0x00007fabe650f840 in unit_dump (u=0x7fabe7edc920, f=f@entry=0x7fabe7ee8f40, prefix=0x7fabe654d12d "", prefix@entry=0x0) at src/core/unit.c:763
#5  0x00007fabe64a6c06 in manager_dump_units (s=s@entry=0x7fabe7e47650, f=f@entry=0x7fabe7ee8f40, prefix=prefix@entry=0x0) at src/core/manager.c:1108
#6  0x00007fabe64d8337 in bus_manager_message_handler (connection=0x7fabe7f5de40, message=0x7fabe7f5f620, data=0x7fabe7e47650) at src/core/dbus-manager.c:1231
#7  0x00007fabe4faee66 in _dbus_object_tree_dispatch_and_unlock () from /lib64/libdbus-1.so.3
#8  0x00007fabe4fa1a31 in dbus_connection_dispatch () from /lib64/libdbus-1.so.3
#9  0x00007fabe64d549a in bus_dispatch (m=m@entry=0x7fabe7e47650) at src/core/dbus.c:528
#10 0x00007fabe64a9c30 in manager_loop (m=0x7fabe7e47650) at src/core/manager.c:1743
#11 0x00007fabe64a1c55 in main (argc=5, argv=0x7fff5cbbe798) at src/core/main.c:1764

'bt full' is attached.

I'm not 100% sure that the crash and the hang are the same problem, but:

The crash occurs in mount_dump() for /sys/kernel/config.

m->parameters_proc_self_mountinfo is filled in, but m->from_proc_self_mountinfo is false (as is m->from_fragment). So get_mount_parameters() returns p=NULL,
and the subsequent 'p->what' is a NULL dereference.

Not sure how this happens, but the debugging continues...

Seeing same behavior consistently on ppc64 between f18 and f19.  I think this is a beta blocker right?

Proposing as a Beta blocker per https://fedoraproject.org/wiki/QA:Testcase_upgrade_fedup_cli_previous_desktop , referenced from https://fedoraproject.org/wiki/Fedora_19_Beta_Release_Criteria#Upgrade_requirements

if the upgrade basically works and the upgraded system is okay, then I'd say not a blocker.

I agree this is inconvenient, but doesn't have to block Beta release. If this is rejected for Beta, please re-propose as Final blocker.

-1 blocker +1 FE and Final blocker

Karsten: note that the key determining factor in whether a bug is a blocker is the criteria, not the test cases. A failure of a validation test case isn't automatically and invariably a blocker; you can get a 'fail' result for a test case but the problem can not violate the criteria. You can call these 'non-blocking failures' or something. We try to make the criteria and test cases line up as closely as possible, but it's never perfect.

Discussed at 2013-05-23 Fedora 19 Beta Go/No-Go meeting, acting as a blocker review meeting: http://meetbot.fedoraproject.org/fedora-meeting-2/2013-05-23/f19_beta_gono-go_meeting.2013-05-23-17.01.log.txt . As this bug does not appear to make the actual upgrade process fail - the upgrade succeeds, and the upgraded system works, you just get an ugly crash in the middle - this does not violate the criterion "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed." and hence is not a blocker.

clearing rejectedblocker as this may be proposed as blocker for final.

Is this still the case with Final? I don't recall seeing it in my Final upgrade tests.

I've done 2 fedup tests using RC2 on ppc64 (Gnome & minimal install F18) and both succeeded without a problem and rebooted automatically.

So i'm not sure if this got automagically fixed by some updates between the report and RC2.

Just my $0.02.

Thanks & regards, Phil

OK. Since, as I said, others tested and no-one seems to see this any more, let's figure that it's fixed unless other reports come in.

I have done a few upgrades and I haven't seen this either.

I just experienced it by upgrading to F19 with fedup.

Same as https://bugzilla.redhat.com/show_bug.cgi?id=957783#c3

The upgrade was stuck, pressed escape and I got the message: "Reached target shutdown". I pressed the reset button of my computer and it booted correctly.

People may wait long unless they know about the "escape" key to see the logs.

Thomas, how long have you waited?

For several minutes there was 0 activity for a while (no disk spinning) and the message was displayed, time to search the web on a different device, bump into this issue and pressed reset. I would say at least 4-5 minutes in total

Reopening then. Thomas, can you provide your logs? They should be at /var/log/fedup.log or somewhere around (maybe also upgrade.log?).

Created attachment 767729 [details]
theute fedup.log

Created attachment 767730 [details]
theute upgrade.log

I'm seeing this problem too.

I'm upgrading F18->F19 on 32-bit on an HP Mini.

Initial symptoms are a hang on first re-boot into upgrade boot with the progress bar about 20% across.

After disabling plymouth, and turning off 'rhgb' and 'quiet', I can see that the boot process is stalling at the point:

  systemd-journald[56]: Received SIGTERM

  Welcome to Fedora 18 (Spherical Cow)!

  systemd-cgroups-agent[394]: Failed to get D-Bus connection: Failed to connect to socket /org/freedesktop/systemd1/private: connection refused

That is not this bug. Please file it separately. This bug is "At the end of an upgrade from F18 to F19", i.e., not a failure to boot half way through.

Please, if your symptom does not precisely match that in the first comment, file your bug separately.

OK thanks Adam,  I've opened this as 980663.

I am in the process to upgrade another machine with similar environment, anything you would like me to do/check if that happens again that would help you ?

Will, Harald? Running fedup with --debuglog couldn't hurt, I guess.

See comment #1 through comment #4 for debugging info. If 'systemctl dump' in the debug shell makes systemd segfault and/or systemd is mysteriously stuck at 'final.target' then this is the same bug.

If systemd isn't stuck / crashing then you have a different bug; please close this and file a new bug.

If it is stuck, then this bug is still in the same place it was before. I lack the time and expertise to debug systemd any deeper than I already have, but the backtrace and core dump should help.

It didn't fail on the other machine

Some debug commands to issue before rebooting.

# systemctl status upgrade-switch-root.service
# systemctl show upgrade-switch-root.service
# systemctl show sys-kernel-config.mount

(In reply to Will Woods from comment #9)
> Created attachment 742026 [details]
> 'bt full' output
> 
> Core was generated by `/usr/lib/systemd/systemd --switched-root --system
> --deserialize 25'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007fabe4966e7b in raise () from /lib64/libpthread.so.0
> #0  0x00007fabe4966e7b in raise () from /lib64/libpthread.so.0
> #1  0x00007fabe64a42f4 in crash (sig=11) at src/core/main.c:140
> #2  <signal handler called>
> #3  mount_dump (u=0x7fabe7edc920, f=0x7fabe7ee8f40, prefix=0x7fabe7ca3d00
> "\t") at src/core/mount.c:828
> #4  0x00007fabe650f840 in unit_dump (u=0x7fabe7edc920,
> f=f@entry=0x7fabe7ee8f40, prefix=0x7fabe654d12d "", prefix@entry=0x0) at
> src/core/unit.c:763
> #5  0x00007fabe64a6c06 in manager_dump_units (s=s@entry=0x7fabe7e47650,
> f=f@entry=0x7fabe7ee8f40, prefix=prefix@entry=0x0) at src/core/manager.c:1108
> #6  0x00007fabe64d8337 in bus_manager_message_handler
> (connection=0x7fabe7f5de40, message=0x7fabe7f5f620, data=0x7fabe7e47650) at
> src/core/dbus-manager.c:1231
> #7  0x00007fabe4faee66 in _dbus_object_tree_dispatch_and_unlock () from
> /lib64/libdbus-1.so.3
> #8  0x00007fabe4fa1a31 in dbus_connection_dispatch () from
> /lib64/libdbus-1.so.3
> #9  0x00007fabe64d549a in bus_dispatch (m=m@entry=0x7fabe7e47650) at
> src/core/dbus.c:528
> #10 0x00007fabe64a9c30 in manager_loop (m=0x7fabe7e47650) at
> src/core/manager.c:1743
> #11 0x00007fabe64a1c55 in main (argc=5, argv=0x7fff5cbbe798) at
> src/core/main.c:1764
> 
> 'bt full' is attached.

commit 1e4fc9b

I found my way to this page after having this same problem upon upgrading from f17 to f19 using FedUp.  I was still running f16, so upgraded to f17 first with Preupgrade before using FedUp.

The machine froze with the Fedora logo and barely full progress bar.  I hit the escape key (would not have known that without coming to this page) and it dropped me to the text screen, which ended with the "Reached target shutdown" message.

Ctrl-Alt-Del had no effect, neither did Ctrl-Alt-F# to try and switch shells.

After powering down the laptop with the power button and rebooting, everything seemed to be fine.

I'll be happy to provide the upgrade logs or any other info if it would help.

Jeff

Yep, same here.

Upgrading from F17 to F19 with FedUp, upgrading ends (monitored in console) at:

[OK] Reached target Shutdown.

Hitting [esc] key gives Fedora logo with progress bar -> changing in Beefy Miracle picture.
Hitting it again gives fast scrolling screen of previous upgrading messages ending in  [OK] Reached target Shutdown.

Reboot after 10 minutes by turning off/on machine:
Starts till GDM
Loggin in: "Something went wrong, please log out"

Emptying one of the users home dir gave a perfect login for that user.
Must be a config thingie of F17, working on it..

Log in problem was the same as bug 979946

I've used fedup (F18 -> F19) three times now (on different computers) and every time it froze when trying to reboot during the upgrade.
Much like in this attached screenshot, the last line on the screen was "Reached target Shutdown.":
https://bugzilla.redhat.com/attachment.cgi?id=741785

It seems like a power off / power on can't do any harm (since the filesystems are already unmounted at this point), but it's still a bug.

I do have a snapshot of the old state (F18), so I could theoretically rerun the upgrade if there's something to try.

systemd-201-2.fc18.9 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/systemd-201-2.fc18.9

Package systemd-201-2.fc18.9:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-201-2.fc18.9'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-20267/systemd-201-2.fc18.9
then log in and leave karma (feedback).

systemd-201-2.fc18.9 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.