Bug 831634 - kernel panic during shutdown of fresh minimal installed F17
kernel panic during shutdown of fresh minimal installed F17
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: dracut-maint
Fedora Extras Quality Assurance
:
: 811403 813262 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-13 09:36 EDT by Dennis Jacobfeuerborn
Modified: 2012-07-03 11:52 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-07-03 11:52:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dennis Jacobfeuerborn 2012-06-13 09:36:53 EDT
I just installed a minimal F17 VM on an F17 based host and about 80% of the time when I shut down the VM I get these messages:

...
Detaching DM devices.
Not all DM devices detached, 1 left.
...
Cannot finalize remaining file systems and devices, trying to kill remaining processes.
...
Detaching DM devices.
Not all DM devices detached, 1 left.
Cannot finalize remaining file systems and devices, giving up.
Successfully changed into root pivot.
/shutdown: line 12: /lib/darcut-lib.sh: No such file or directory

followed by an "Attempted to kill init!" panic.
This is a minimal install from the netinst iso that is fully updated and has nothing else installed so far (except for a few low-level packages like vim-enhanced, less, acpid, wget)
Comment 1 Harald Hoyer 2012-06-13 09:51:09 EDT
Which dracut version?

Does a
# dracut -f
fix the issue?
Comment 2 Dennis Jacobfeuerborn 2012-06-13 11:47:50 EDT
(In reply to comment #1)
> Which dracut version?

dracut-018-40.git20120522

> Does a
> # dracut -f
> fix the issue?

Unfortunatly no. Did a "dracut -f" which ran without error, rebooted, then shut down and got the same panic.
Comment 3 Jim Wilson 2012-06-13 14:48:35 EDT
I have an eerily similar problem on bare metal after network upgrade then 'yum update' from F16.  Uname -r gives me 3.4.0-1.fc17.x86_64.

Both Gnome and XFCE hang after clicky-clicky shutdown with the Fedora balloon displayed and keyboard input seemingly ignored.  If I'm fast enough to press ESC after the balloon first appears, screen switches to text(ish) mode, and I see:

#### BEGIN BEST EFFORT AT COPYING SCREEN; I COULD DEVISE NO WAY TO SAVE IT ####

Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /bgoot.
Unmounted /proc/fs/nsfd.
Unmounted /dev/mqueue.
Unmounted /sys/kernel/config.
Unmounted /dev/homepages.
Unmounted /sys/kernel/debug.
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Not all DM devices detached, 1 left.
Detaching DM devies.
Not all DM devices detached, 1 left.
Cannot finalize remainign file systems and devices, trying to kill remaining processes.
Not all DM devices detached, 1 left.
Unmounted /oldroot/proc.
Unmounted /oldroot/dev/pts.
Unmounted /oldroot/run.
Unmounted /oldroot/sys/fs/selinux.
Unmounted /oldroot/sys/fs/cgrop/systemd.
Unmounted /oldroot/sys/fs/cgroup/memory.
Unmounted /oldroot/sys/fs/cgroup/freezer.
Unmounted /oldroot/sys/fs/cgroup/blkio.
Unmounted /oldroot/dev/shm.
Unmounted /oldroot/sys/kernel/security.
Unmounted /oldroot/sys/fs/cgroup/cpu,cpuacct.
Unmounted /oldroot/dev.
Unmounted /oldroot/sys/fs/cgroup/devices.
Unmounted /oldroot/sys/fs/cgroup/cpuset.
Unmounted /oldroot/sys/fs/cgroup/perf_event.
Unmounted /oldroot/sys/fs/csgroup.
Unmounted /oldroot/sys.
Unmounted /oldroot.
/shutdown line 60: poweroff: command not found


dracut Warning: Signal caught!

dropping to debug shell.
sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell.
shutdown:/#

####  END BEST EFFORT AT COPYING SCREEN ####

The last line is a shell prompt. Typing (in the blind, because terminal
doesn't echo) reveals I have a small repertoire of commands (I presume mostly
bash built-ins) available.

If I log out of 'debug shell' a couple of times, I eventually do get a kernel panic.
Comment 4 Harald Hoyer 2012-06-18 07:42:08 EDT
(In reply to comment #2)
> (In reply to comment #1)
> > Which dracut version?
> 
> dracut-018-40.git20120522
> 
> > Does a
> > # dracut -f
> > fix the issue?
> 
> Unfortunatly no. Did a "dracut -f" which ran without error, rebooted, then
> shut down and got the same panic.

Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-018-65.git20120612.fc17'
# dracut -f

Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9390/dracut-018-65.git20120612.fc17
then log in and leave karma (feedback).
Comment 5 Roman Kagan 2012-06-18 09:15:52 EDT
We're having the same issue with unattended installations of F17 in virtual machines in Parallels Server.

Snippet of the VM's serial port log

localhost login: [   41.582232] type=1305 audit(1340022955.174:86): audit_pid=0 old=482 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   41.595898] type=1131 audit(1340022955.188:87): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="auditd" exe="/usr/lib/systemd/systemd" hostnam
e=? addr=? terminal=? res=success'
[   41.637776] type=1130 audit(1340022955.230:88): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="plymouth-poweroff" exe="/usr/lib/systemd/syste
md" hostname=? addr=? terminal=? res=success'
[   41.651740] type=1131 audit(1340022955.243:89): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="NetworkManager" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.691906] type=1131 audit(1340022955.283:90): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="mdmonitor-takeover" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.694296] type=1131 audit(1340022955.286:91): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-sysctl" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.756948] type=1131 audit(1340022955.348:92): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-vconsole-setup" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.764075] type=1128 audit(1340022955.356:93): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='init exe="/usr/lib/systemd/systemd-update-utmp" hostname=? addr=? terminal=? res=success'
[   41.777223] type=1130 audit(1340022955.369:94): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-update-utmp-shutdown" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.779733] type=1131 audit(1340022955.371:95): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-update-utmp-shutdown" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /boot.
Unmounted /sys/kernel/config.
Unmounted /sys/kernel/debug.
Unmounted /dev/mqueue.
Unmounted /dev/hugepages.
[   43.099400] EXT4-fs (dm-1): re-mounted. Opts: (null)
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Not all DM devices detached, 1 left.
Detaching DM devices.
Not all DM devices detached, 1 left.
Cannot finalize remaining file systems and devices, trying to kill remaining processes.
Detaching DM devices.
Not all DM devices detached, 1 left.

Cannot finalize remaining file systems and devices, giving up.
Successfully changed into root pivot.
/shutdown: line 12: /lib/dracut-lib.sh: No such file or directory
[   43.154407] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   43.154407] 
[   43.155842] Pid: 1, comm: shutdown Not tainted 3.4.2-4.fc17.x86_64 #1
[   43.156575] Call Trace:
[   43.156922]  [<ffffffff815e6649>] panic+0xba/0x1c6
[   43.157494]  [<ffffffff8105b48b>] do_exit+0x86b/0x8a0
[   43.158063]  [<ffffffff81068647>] ? __set_task_blocked+0x37/0x80
[   43.158652]  [<ffffffff8105b80f>] do_group_exit+0x3f/0xa0
[   43.159234]  [<ffffffff8105b887>] sys_exit_group+0x17/0x20
[   43.159775]  [<ffffffff815f88e9>] system_call_fastpath+0x16/0x1b



complete command line: 

BOOT_IMAGE=/vmlinuz-3.4.2-4.fc17.x86_64 root=/dev/mapper/vg-lv_root ro rd.md=0 rd.dm=0 console=tty0 KEYTABLE=us console=ttyS0,115200 SYSFONT=True ignore_loglevel rd.lvm.lv=vg/lv_root rd.luks=0 debug rd.lvm.lv=vg/lv_swap LANG=en_US.UTF-8


Updating dracut and regenerating initramfs didn't help.
Comment 6 Roman Kagan 2012-06-18 17:49:53 EDT
I tracked it down to a race between unpacking the initramfs and actual shutdown.

That is, initramfs is populated from the compressed cpio image in /usr/lib/dracut/dracut-initramfs-restore called from dracut-shutdown.service unit; however systemd proceeds to systemd-shutdown before unpacking is complete.  As a result, /shutdown is present in the initramfs after pivot_root, but /lib is not.

My systemd-foo is too weak but the following patch fixes things for me:

--- ./usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service	2012-06-19 01:44:42.503000001 +0400
+++ /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service	2012-06-19 01:34:16.207999972 +0400
@@ -8,7 +8,7 @@
 [Unit]
 Description=Restore /run/initramfs
 After=getty@tty1.service prefdm.service
-Before=reboot.service
+Before=reboot.service shutdown.target
 DefaultDependencies=no
 ConditionPathExists=/run/initramfs/.need_shutdown
 ConditionPathExists=!/run/initramfs/bin/sh

I've no idea why symlink /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service doesn't have that effect.
Comment 7 Harald Hoyer 2012-06-19 06:31:22 EDT
(In reply to comment #6)
> I tracked it down to a race between unpacking the initramfs and actual
> shutdown.
> 
> That is, initramfs is populated from the compressed cpio image in
> /usr/lib/dracut/dracut-initramfs-restore called from dracut-shutdown.service
> unit; however systemd proceeds to systemd-shutdown before unpacking is
> complete.  As a result, /shutdown is present in the initramfs after
> pivot_root, but /lib is not.
> 
> My systemd-foo is too weak but the following patch fixes things for me:
> 
> --- ./usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service
> 2012-06-19 01:44:42.503000001 +0400
> +++ /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service
> 2012-06-19 01:34:16.207999972 +0400
> @@ -8,7 +8,7 @@
>  [Unit]
>  Description=Restore /run/initramfs
>  After=getty@tty1.service prefdm.service
> -Before=reboot.service
> +Before=reboot.service shutdown.target
>  DefaultDependencies=no
>  ConditionPathExists=/run/initramfs/.need_shutdown
>  ConditionPathExists=!/run/initramfs/bin/sh
> 
> I've no idea why symlink
> /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service
> doesn't have that effect.

Thank you very much!
Comment 8 Fedora Update System 2012-06-22 10:17:47 EDT
dracut-018-78.git20120622.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-018-78.git20120622.fc17
Comment 9 Chris Travers 2012-06-23 11:11:12 EDT
This has fixed similar problems for me.
Comment 10 Fedora Update System 2012-06-25 20:33:55 EDT
Package dracut-018-78.git20120622.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-018-78.git20120622.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9847/dracut-018-78.git20120622.fc17
then log in and leave karma (feedback).
Comment 11 Cole Robinson 2012-06-29 10:09:02 EDT
*** Bug 811403 has been marked as a duplicate of this bug. ***
Comment 12 Cole Robinson 2012-06-29 10:10:32 EDT
*** Bug 813262 has been marked as a duplicate of this bug. ***
Comment 13 Fedora Update System 2012-07-03 11:52:27 EDT
dracut-018-78.git20120622.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.