Bug 831634

Summary: kernel panic during shutdown of fresh minimal installed F17
Product: [Fedora] Fedora Reporter: Dennis Jacobfeuerborn <dennisml>
Component: dracutAssignee: dracut-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 17CC: chris, dracut-maint, harald, jonathan, lsof, mishu, rvkagan, tflink, wilson
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-03 15:52:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dennis Jacobfeuerborn 2012-06-13 13:36:53 UTC
I just installed a minimal F17 VM on an F17 based host and about 80% of the time when I shut down the VM I get these messages:

...
Detaching DM devices.
Not all DM devices detached, 1 left.
...
Cannot finalize remaining file systems and devices, trying to kill remaining processes.
...
Detaching DM devices.
Not all DM devices detached, 1 left.
Cannot finalize remaining file systems and devices, giving up.
Successfully changed into root pivot.
/shutdown: line 12: /lib/darcut-lib.sh: No such file or directory

followed by an "Attempted to kill init!" panic.
This is a minimal install from the netinst iso that is fully updated and has nothing else installed so far (except for a few low-level packages like vim-enhanced, less, acpid, wget)

Comment 1 Harald Hoyer 2012-06-13 13:51:09 UTC
Which dracut version?

Does a
# dracut -f
fix the issue?

Comment 2 Dennis Jacobfeuerborn 2012-06-13 15:47:50 UTC
(In reply to comment #1)
> Which dracut version?

dracut-018-40.git20120522

> Does a
> # dracut -f
> fix the issue?

Unfortunatly no. Did a "dracut -f" which ran without error, rebooted, then shut down and got the same panic.

Comment 3 Jim Wilson 2012-06-13 18:48:35 UTC
I have an eerily similar problem on bare metal after network upgrade then 'yum update' from F16.  Uname -r gives me 3.4.0-1.fc17.x86_64.

Both Gnome and XFCE hang after clicky-clicky shutdown with the Fedora balloon displayed and keyboard input seemingly ignored.  If I'm fast enough to press ESC after the balloon first appears, screen switches to text(ish) mode, and I see:

#### BEGIN BEST EFFORT AT COPYING SCREEN; I COULD DEVISE NO WAY TO SAVE IT ####

Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /bgoot.
Unmounted /proc/fs/nsfd.
Unmounted /dev/mqueue.
Unmounted /sys/kernel/config.
Unmounted /dev/homepages.
Unmounted /sys/kernel/debug.
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Not all DM devices detached, 1 left.
Detaching DM devies.
Not all DM devices detached, 1 left.
Cannot finalize remainign file systems and devices, trying to kill remaining processes.
Not all DM devices detached, 1 left.
Unmounted /oldroot/proc.
Unmounted /oldroot/dev/pts.
Unmounted /oldroot/run.
Unmounted /oldroot/sys/fs/selinux.
Unmounted /oldroot/sys/fs/cgrop/systemd.
Unmounted /oldroot/sys/fs/cgroup/memory.
Unmounted /oldroot/sys/fs/cgroup/freezer.
Unmounted /oldroot/sys/fs/cgroup/blkio.
Unmounted /oldroot/dev/shm.
Unmounted /oldroot/sys/kernel/security.
Unmounted /oldroot/sys/fs/cgroup/cpu,cpuacct.
Unmounted /oldroot/dev.
Unmounted /oldroot/sys/fs/cgroup/devices.
Unmounted /oldroot/sys/fs/cgroup/cpuset.
Unmounted /oldroot/sys/fs/cgroup/perf_event.
Unmounted /oldroot/sys/fs/csgroup.
Unmounted /oldroot/sys.
Unmounted /oldroot.
/shutdown line 60: poweroff: command not found


dracut Warning: Signal caught!

dropping to debug shell.
sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell.
shutdown:/#

####  END BEST EFFORT AT COPYING SCREEN ####

The last line is a shell prompt. Typing (in the blind, because terminal
doesn't echo) reveals I have a small repertoire of commands (I presume mostly
bash built-ins) available.

If I log out of 'debug shell' a couple of times, I eventually do get a kernel panic.

Comment 4 Harald Hoyer 2012-06-18 11:42:08 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > Which dracut version?
> 
> dracut-018-40.git20120522
> 
> > Does a
> > # dracut -f
> > fix the issue?
> 
> Unfortunatly no. Did a "dracut -f" which ran without error, rebooted, then
> shut down and got the same panic.

Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-018-65.git20120612.fc17'
# dracut -f

Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9390/dracut-018-65.git20120612.fc17
then log in and leave karma (feedback).

Comment 5 Roman Kagan 2012-06-18 13:15:52 UTC
We're having the same issue with unattended installations of F17 in virtual machines in Parallels Server.

Snippet of the VM's serial port log

localhost login: [   41.582232] type=1305 audit(1340022955.174:86): audit_pid=0 old=482 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   41.595898] type=1131 audit(1340022955.188:87): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="auditd" exe="/usr/lib/systemd/systemd" hostnam
e=? addr=? terminal=? res=success'
[   41.637776] type=1130 audit(1340022955.230:88): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="plymouth-poweroff" exe="/usr/lib/systemd/syste
md" hostname=? addr=? terminal=? res=success'
[   41.651740] type=1131 audit(1340022955.243:89): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="NetworkManager" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.691906] type=1131 audit(1340022955.283:90): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="mdmonitor-takeover" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.694296] type=1131 audit(1340022955.286:91): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-sysctl" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.756948] type=1131 audit(1340022955.348:92): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-vconsole-setup" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.764075] type=1128 audit(1340022955.356:93): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='init exe="/usr/lib/systemd/systemd-update-utmp" hostname=? addr=? terminal=? res=success'
[   41.777223] type=1130 audit(1340022955.369:94): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-update-utmp-shutdown" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   41.779733] type=1131 audit(1340022955.371:95): pid=0 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="systemd-update-utmp-shutdown" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /boot.
Unmounted /sys/kernel/config.
Unmounted /sys/kernel/debug.
Unmounted /dev/mqueue.
Unmounted /dev/hugepages.
[   43.099400] EXT4-fs (dm-1): re-mounted. Opts: (null)
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Not all DM devices detached, 1 left.
Detaching DM devices.
Not all DM devices detached, 1 left.
Cannot finalize remaining file systems and devices, trying to kill remaining processes.
Detaching DM devices.
Not all DM devices detached, 1 left.

Cannot finalize remaining file systems and devices, giving up.
Successfully changed into root pivot.
/shutdown: line 12: /lib/dracut-lib.sh: No such file or directory
[   43.154407] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   43.154407] 
[   43.155842] Pid: 1, comm: shutdown Not tainted 3.4.2-4.fc17.x86_64 #1
[   43.156575] Call Trace:
[   43.156922]  [<ffffffff815e6649>] panic+0xba/0x1c6
[   43.157494]  [<ffffffff8105b48b>] do_exit+0x86b/0x8a0
[   43.158063]  [<ffffffff81068647>] ? __set_task_blocked+0x37/0x80
[   43.158652]  [<ffffffff8105b80f>] do_group_exit+0x3f/0xa0
[   43.159234]  [<ffffffff8105b887>] sys_exit_group+0x17/0x20
[   43.159775]  [<ffffffff815f88e9>] system_call_fastpath+0x16/0x1b



complete command line: 

BOOT_IMAGE=/vmlinuz-3.4.2-4.fc17.x86_64 root=/dev/mapper/vg-lv_root ro rd.md=0 rd.dm=0 console=tty0 KEYTABLE=us console=ttyS0,115200 SYSFONT=True ignore_loglevel rd.lvm.lv=vg/lv_root rd.luks=0 debug rd.lvm.lv=vg/lv_swap LANG=en_US.UTF-8


Updating dracut and regenerating initramfs didn't help.

Comment 6 Roman Kagan 2012-06-18 21:49:53 UTC
I tracked it down to a race between unpacking the initramfs and actual shutdown.

That is, initramfs is populated from the compressed cpio image in /usr/lib/dracut/dracut-initramfs-restore called from dracut-shutdown.service unit; however systemd proceeds to systemd-shutdown before unpacking is complete.  As a result, /shutdown is present in the initramfs after pivot_root, but /lib is not.

My systemd-foo is too weak but the following patch fixes things for me:

--- ./usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service	2012-06-19 01:44:42.503000001 +0400
+++ /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service	2012-06-19 01:34:16.207999972 +0400
@@ -8,7 +8,7 @@
 [Unit]
 Description=Restore /run/initramfs
 After=getty prefdm.service
-Before=reboot.service
+Before=reboot.service shutdown.target
 DefaultDependencies=no
 ConditionPathExists=/run/initramfs/.need_shutdown
 ConditionPathExists=!/run/initramfs/bin/sh

I've no idea why symlink /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service doesn't have that effect.

Comment 7 Harald Hoyer 2012-06-19 10:31:22 UTC
(In reply to comment #6)
> I tracked it down to a race between unpacking the initramfs and actual
> shutdown.
> 
> That is, initramfs is populated from the compressed cpio image in
> /usr/lib/dracut/dracut-initramfs-restore called from dracut-shutdown.service
> unit; however systemd proceeds to systemd-shutdown before unpacking is
> complete.  As a result, /shutdown is present in the initramfs after
> pivot_root, but /lib is not.
> 
> My systemd-foo is too weak but the following patch fixes things for me:
> 
> --- ./usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service
> 2012-06-19 01:44:42.503000001 +0400
> +++ /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service
> 2012-06-19 01:34:16.207999972 +0400
> @@ -8,7 +8,7 @@
>  [Unit]
>  Description=Restore /run/initramfs
>  After=getty prefdm.service
> -Before=reboot.service
> +Before=reboot.service shutdown.target
>  DefaultDependencies=no
>  ConditionPathExists=/run/initramfs/.need_shutdown
>  ConditionPathExists=!/run/initramfs/bin/sh
> 
> I've no idea why symlink
> /usr/lib/systemd/system/shutdown.target.wants/dracut-shutdown.service
> doesn't have that effect.

Thank you very much!

Comment 8 Fedora Update System 2012-06-22 14:17:47 UTC
dracut-018-78.git20120622.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-018-78.git20120622.fc17

Comment 9 Chris Travers 2012-06-23 15:11:12 UTC
This has fixed similar problems for me.

Comment 10 Fedora Update System 2012-06-26 00:33:55 UTC
Package dracut-018-78.git20120622.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-018-78.git20120622.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9847/dracut-018-78.git20120622.fc17
then log in and leave karma (feedback).

Comment 11 Cole Robinson 2012-06-29 14:09:02 UTC
*** Bug 811403 has been marked as a duplicate of this bug. ***

Comment 12 Cole Robinson 2012-06-29 14:10:32 UTC
*** Bug 813262 has been marked as a duplicate of this bug. ***

Comment 13 Fedora Update System 2012-07-03 15:52:27 UTC
dracut-018-78.git20120622.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.