Bug 1178497

yum reinstall dracut didn't help.

uninstalling dracut made the trick and i can turn off the system, but obviously this is not the desired way.

i am going to re-install dracut now, to see if the problem is showing again

yes, re-installing dracut with the removed dependencies (dracut abrt-addon-vmcore abrt-cli dracut-config-rescue dracut-network kexec-tools plymouth-scripts)

causes the problem to re-appear

Hi Adrea,
systemctl reboot -f -f

helps for three shutdowns/reboot/poweroffs. Interesting. A Colin Walters who is on the fedora-cloud-atomic team gave a patch for an alleged lvm bug which is referenced in the link you provided.

But it is a no permanent solution.

+# Work around anaconda/dracut/lvm bug
+sync
+sync
+sync
+systemctl reboot -f -f
+%end

LinforPros

Sadly that doesn't help me. 
Also, as stated, removing Dracut allows me turn off the machine and installing back restore the problem.

I hate to inform you that the following action has nullified the bug. I "hate" to do that because we will never know what had happened and why it occurred in the first place. In other words it is a lesson not learned. 



Packages Altered:
    Updated dracut-038-31.git20141204.fc21.x86_64               @updates
    Update         038-32.git20141216.fc21.x86_64               @updates
    Updated dracut-config-rescue-038-31.git20141204.fc21.x86_64 @updates
    Update                       038-32.git20141216.fc21.x86_64 @updates
    Updated dracut-network-038-31.git20141204.fc21.x86_64       @updates
    Update                 038-32.git20141216.fc21.x86_64       @updates


After the update I have successfully shutdown/reboot/poweroff my machine 7 times with no recurring problems.


LinforPros

My apologies,
the problem persists. this is insane. After 8th reboot the nineth one produces the same result. Server cannot be reliably rebooted/powered off.

LinforPros.

Can you please follow the instruction of the dracut man page about debugging shutdown?

 # mkdir -p /run/initramfs/etc/cmdline.d
 # echo "rd.debug rd.break=pre-shutdown rd.break=shutdown" > /run/initramfs/etc/cmdline.d/debug.conf
 # touch /run/initramfs/.need_shutdown

And maybe attach a photo of the screen. Some keyboards have a "Pause" or "ScrollLck" key to stop the output.


A workaround to prevent the loop at shutdown, is to remove "/run/initramfs/.need_shutdown" prior to shutdown, or to mask the dracut-shutdown.service


# systemctl mask dracut-shutdown.service


But please provide me with some debugging info first.

Oh, and please also do
http://freedesktop.org/wiki/Software/systemd/Debugging/#diagnosingshutdownproblems

add 
systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M enforcing=0
to the kernel command line and create 
 /usr/lib/systemd/system-shutdown/debug.sh
with the contents

#!/bin/sh
mount -o remount,rw /
dmesg > /shutdown-log.txt
mount -o remount,ro /

... then do:
# chmod a+x /usr/lib/systemd/system-shutdown/debug.sh

And attach /shutdown-log.txt to this bug report.

Many thanks!!!

Created attachment 983031 [details]
Photo of the error produced at Shutdown.target, dracut

As per the request to attach a photo, here it is.
LinforPros

Created attachment 983078 [details]
The log obtained via kernel command line

Hello, I am pleased to provide the log you suggested in you post. 
Just wanted to mention that the error showed up during two separate fedora installations albeit on the same machine. Always only after packstack --allinone installation. I regret that I failed to do an LVM snapshot so I could just start over. I will have to reinstall fedora again. Unless we find a solution to the problem faster :-)
thank you for looking at the log. If you can decipher anything from it that'd be great.

LinforPros

Hi Herald,

On the final note I would like to mention that:

sync && reboot -f
sync && poweroff -f

work perfectly well. Which suggests that it is not a kernel problem.

So does #systemctl mask dracut-shutdown.service help avoid the loop.

LinforPros

Hmm, I can only think of two things:
1. /run was not mounted with a tmpfs
2. /run was remounted read-only

Can you extend /usr/lib/systemd/system-shutdown/debug.sh
and add

cat /proc/mounts >> /shutdown-log.txt
cat /proc/self/mountinfo >> /shutdown-log.txt

after the dmesg line? So, I can confirm either of the two theories.

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Hi,
I've re-opened this bug as I am now seeing this on 3 different virtual machines on RHEL7.2 (still running with the latest 7.1 kernel) with several different releases of OpenStack (kilo/RDO, OSP7 and liberty/RDO).
I am gathering more information, and will update this bug accordingly.
Components:
# rpm -qa dracut\*
dracut-network-033-359.el7.x86_64
dracut-033-359.el7.x86_64
dracut-config-rescue-033-359.el7.x86_64

all el7.2 patches current as of today (20151125) except for the kernel:
[root@rh7x64 ~]# uname -a
Linux rh7x64 3.10.0-229.20.1.el7.x86_64 #1 SMP Thu Sep 24 12:23:56 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@rh7x64 ~]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-ia32:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.2 (Maipo)
Release:        7.2
Codename:       Maipo

Please note that to work around the issue, dracut can be invoked like this:

# dracut -f -o shutdown

(this excludes the entire 'shutdown' dracut module).

To disable this on a permanent basis and so subsequent kernel updates don't hang the systems either, I did this:

# cat /etc/dracut.conf.d/shut.conf 
omit_dracutmodules+="shutdown"

this makes dracut skip the 'shutdown' module in every case. (could have other side effects).

(In reply to Vincent S. Cojot from comment #14)
> Please note that to work around the issue, dracut can be invoked like this:
> 
> # dracut -f -o shutdown
> 
> (this excludes the entire 'shutdown' dracut module).
> 
> To disable this on a permanent basis and so subsequent kernel updates don't
> hang the systems either, I did this:
> 
> # cat /etc/dracut.conf.d/shut.conf 
> omit_dracutmodules+="shutdown"
> 
> this makes dracut skip the 'shutdown' module in every case. (could have
> other side effects).

# systemctl mask dracut-shutdown.service 

is a better short-term workaround.

please give me debug info as mentioned in comment #7 and comment #11

Hello Harald,
Just saw your reply. I will be providing the information shortly.
A few things to note:
All of the systems with the issue are Virtual Machines.
I just realized that all of them and on a VMWare Hypervisor with the VMWare tools installed.
Let me gather more debugging information.
Thank you for your patience,
Vincent

Currently following your instructions with:

[root@osp2 ~]# cat /usr/lib/systemd/system-shutdown/debug.sh
#!/bin/sh
mount -o remount,rw /
dmesg > /shutdown-log.txt
ps -faux >> /shutdown-log.txt
cat /proc/mounts >> /shutdown-log.txt
cat /proc/self/mountinfo >> /shutdown-log.txt
mount -o remount,ro /

Please note that 3/4 machines on which I have the issue have a PackStack of some kind installed (1 is RDO/kilo, 1 is OSP7, 1 is RDO/Liberty). The last (and 4th) VM doesn't have packstack installed.

Regards,

Vincent

Created attachment 1099432 [details]
screenshot of shutdown issue

Created attachment 1099433 [details]
shutdown-log.txt

Created attachment 1099434 [details]
small movie of console while shutting down..

tmpfs /run tmpfs ro,seclabel,nosuid,nodev,mode=755 0 0

23 59 0:18 / /run rw,nosuid,nodev shared:22 - tmpfs tmpfs ro,seclabel,mode=755

So, /run is mounted read-only suddenly.

Does any of your custom scripts mount /run read-only?

What are these, and why can't they be umounted???

[  237.377280] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.377406] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.377433] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.377522] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.377566] systemd-shutdown[1]: Unmounting /run/netns.
[  237.477388] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.477402] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.482242] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.482260] systemd-shutdown[1]: Could not unmount /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b: Invalid argument
[  237.482266] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.482271] systemd-shutdown[1]: Could not unmount /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1: Invalid argument
[  237.487535] systemd-shutdown[1]: Not all file systems unmounted, 2 left.

Seems like not all net namespaces are cleanup.

Hi Harald,
These things are OpenVswitch network namespaces..
And NO, I don't have any script that re-mounts /run read-only (that I know of).
I am not sure if we could do something about it but at any case it shouldn't prevent shutdown.
Let me do the tests again but this time with a non-OpenStack VM..
Regards,
Vincent

(In reply to Vincent S. Cojot from comment #25)
> Hi Harald,
> These things are OpenVswitch network namespaces..
> And NO, I don't have any script that re-mounts /run read-only (that I know
> of).
> I am not sure if we could do something about it but at any case it shouldn't
> prevent shutdown.
> Let me do the tests again but this time with a non-OpenStack VM..
> Regards,
> Vincent

Ok, please try the following steps:

edit /usr/lib/dracut/modules.d/99shutdown/shutdown.sh

after:
. /lib/dracut-lib.sh

add:
if [ "$(stat -c '%T' -f /)" = "tmpfs" ]; then
    mount -o remount,rw /
fi

edit /usr/lib/dracut/modules.d/99shutdown/module-setup.sh
change:
    inst_multiple umount poweroff reboot halt losetup
to:
    inst_multiple umount poweroff reboot halt losetup stat


recreate the initramfs:
# dracut --force

unmask the shutdown:
# systemctl unmask dracut-shutdown.service.

and reboot.

Hi Harald,

Prior to performing these changes, shall I undo the previous ones?

Should I remove /usr/lib/systemd/system-shutdown/debug.sh and /run/initramfs/etc/cmdline.d ? Same for the extra kernel args?

Thanks,

Vincent

I can confirm that your change fixes the issue for my OSP7 allinone VM.
The resulting patch is:
[root@osp2 ~]# less /tmp/dracut.patch 
*** /usr/lib/dracut/modules.d/99shutdown/shutdown.sh.orig       Mon Nov 30 10:21:11 2015
--- /usr/lib/dracut/modules.d/99shutdown/shutdown.sh    Mon Nov 30 10:21:22 2015
***************
*** 13,18 ****
--- 13,21 ----
  export TERM=linux
  export PATH=/usr/sbin:/usr/bin:/sbin:/bin
  . /lib/dracut-lib.sh
+ if [ "$(stat -c '%T' -f /)" = "tmpfs" ]; then
+     mount -o remount,rw /
+ fi
  
  mkdir /oldsys
  for i in sys proc run dev; do
*** /usr/lib/dracut/modules.d/99shutdown/module-setup.sh.orig   Mon Nov 30 10:21:55 2015
--- /usr/lib/dracut/modules.d/99shutdown/module-setup.sh        Mon Nov 30 10:22:22 2015
***************
*** 13,19 ****
  
  install() {
      local _d
!     inst_multiple umount poweroff reboot halt losetup
      inst_multiple -o kexec
      inst "$moddir/shutdown.sh" "$prefix/shutdown"
      [ -e "${initdir}/lib" ] || mkdir -m 0755 -p ${initdir}/lib
--- 13,19 ----
  
  install() {
      local _d
!     inst_multiple umount poweroff reboot halt losetup stat
      inst_multiple -o kexec
      inst "$moddir/shutdown.sh" "$prefix/shutdown"
      [ -e "${initdir}/lib" ] || mkdir -m 0755 -p ${initdir}/lib

(In reply to Vincent S. Cojot from comment #28)
> I can confirm that your change fixes the issue for my OSP7 allinone VM.

Thank you very much for your testing!

You are most welcomed, Harald!
Please feel free to send me updated test-cases/patches to test, etc...
This seemed pretty major.. do you think upstream dracut could be affected by the issue as well?
Regards,
Vincent

(In reply to Vincent S. Cojot from comment #30)
> You are most welcomed, Harald!
> Please feel free to send me updated test-cases/patches to test, etc...
> This seemed pretty major.. do you think upstream dracut could be affected by
> the issue as well?
> Regards,
> Vincent

already pushed to upstream:
https://github.com/haraldh/dracut/commit/54e09dfb72b557ac8ccd48f5d37089287d272ec7

Do you think we could expect an update on 7.2.z?
Thanks,
Vincent

I also hit this bug however the solution posted by Harald worked flawlessly. Any ideas on when this fix will be implemented in a RHEL release?

Thanks,

Dan

While implementing OSP7 with versionlock we hit with the problem too. Can this bug be updated when this will be included.

We just hit this problem in OSP 8 Director.  The shutdown hangs and the following message is dumped to the console in an infinite loop:

rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem

(In reply to Chris Dearborn from comment #40)
> We just hit this problem in OSP 8 Director.  The shutdown hangs and the
> following message is dumped to the console in an infinite loop:
> 
> rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only
> filesystem

@Chris,

What's your kernel?

Rgds,
Shinobu

We're running RHEL 7.2:

[stack@director-r13 etc]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[stack@director-r13 ~]$ uname -a
Linux director-r13.rcbd.lab 3.10.0-327.4.5.el7.x86_64 #1 SMP Thu Jan 21 04:10:29 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Oh, should have specified above that I'm running OSP 8 Beta 4.

i dont want to point out the obvious but i see several people making typo in spelling dracut leading me to believe it was not unintentional 

/lib/draUCt

Oops!  Sorry about that - the correct spelling is below.  See attached ConsoleError.gif.

rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system.

Created attachment 1120789 [details]
Screen shot of console error during shutdown

Steps to reproduce issue: 

 1. Create a root filesystem on LVM volume.
 2. Add a network namespace.
    # ip netns add namespace
 3. # systemctl reboot

Shutdown infinitely loops at: 
---
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system


Work-Around noted in https://bugzilla.redhat.com/show_bug.cgi?id=1178497#c26 did indeed resolve issue.  I did not need to unmask dracut-shutdown.service.  


--- /usr/lib/dracut/modules.d/99shutdown/shutdown.sh.BAK	2016-02-16 11:56:58.750828854 -0500
+++ /usr/lib/dracut/modules.d/99shutdown/shutdown.sh	2016-02-16 11:57:28.011290818 -0500
@@ -14,6 +14,10 @@ export TERM=linux
 export PATH=/usr/sbin:/usr/bin:/sbin:/bin
 . /lib/dracut-lib.sh
 
+if [ "$(stat -c '%T' -f /)" = "tmpfs" ]; then
+    mount -o remount,rw /
+fi
+
 mkdir /oldsys
 for i in sys proc run dev; do
     mkdir /oldsys/$i


--- /usr/lib/dracut/modules.d/99shutdown/module-setup.sh.BAK	2016-02-16 11:57:38.222445577 -0500
+++ /usr/lib/dracut/modules.d/99shutdown/module-setup.sh	2016-02-16 11:58:00.330736783 -0500
@@ -13,7 +13,7 @@ depends() {
 
 install() {
     local _d
-    inst_multiple umount poweroff reboot halt losetup
+    inst_multiple umount poweroff reboot halt losetup stat
     inst_multiple -o kexec
     inst "$moddir/shutdown.sh" "$prefix/shutdown"
     [ -e "${initdir}/lib" ] || mkdir -m 0755 -p ${initdir}/lib

hitting this issue on CentOS-7 -- trying the stated work around but this is definitely an annoyance!

Just to add myself to the list.
System is CentOS 7.2 fully updated where I'm testing Openstack Icehouse provided by rdo repo and the problem appears after running packstack.
At the moment applied the workaround of masking dracut-shutdown service.
I'm available to test potential complete resolutions.

I get this same error on shutdown

My system is setup as compute/network node connected to a controller on RHEL7.2 with OSP6

[root@node2 ~(keystone_admin)]# uname -a
Linux node2.jamesnet.ca 3.10.0-327.10.1.el7.x86_64 #1 SMP Sat Jan 23 04:54:55 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@node2 ~(keystone_admin)]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

[root@node2 ~(keystone_admin)]# openstack-service status
MainPID=2462 Id=neutron-dhcp-agent.service ActiveState=active
MainPID=2472 Id=neutron-l3-agent.service ActiveState=active
MainPID=1232 Id=neutron-metadata-agent.service ActiveState=active
MainPID=2471 Id=neutron-openvswitch-agent.service ActiveState=active
MainPID=1230 Id=openstack-ceilometer-compute.service ActiveState=active
MainPID=2461 Id=openstack-nova-compute.service ActiveState=active

[root@node2 ~(keystone_admin)]# rpm -qa  | grep -i openstack
openstack-nova-common-2014.2.3-65.el7ost.noarch
openstack-neutron-2014.2.3-33.el7ost.noarch
openstack-selinux-0.6.43-1.el7ost.noarch
openstack-neutron-common-2014.2.3-33.el7ost.noarch
openstack-neutron-openvswitch-2014.2.3-33.el7ost.noarch
openstack-ceilometer-common-2014.2.3-7.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-ceilometer-compute-2014.2.3-7.el7ost.noarch
openstack-nova-compute-2014.2.3-65.el7ost.noarch

[root@node2 ~(keystone_admin)]# rpm -qa | grep dracut
dracut-033-360.el7_2.x86_64
dracut-network-033-360.el7_2.x86_64
dracut-config-rescue-033-360.el7_2.x86_64

As a workaround i did as Herald said earlier

systemctl mask dracut-shutdown.service

Issue is still present, the systemctl mask dracut-shutdown.service resolved it as well.  Not sure what the difference is between the 2 solutions but this appears to be the easiest one to apply.  

Package info is below...

[root@mini-workstation ~]# rpm -qa | grep -i dracut
dracut-network-033-161.el7.x86_64
dracut-config-rescue-033-360.el7_2.x86_64
dracut-network-033-360.el7_2.x86_64
dracut-033-360.el7_2.x86_64
dracut-033-161.el7.x86_64

[root@mini-workstation ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

If you're hitting this on a Red Hat OpenStack Platform director system (the "undercloud"), the following procedure should enable you to shut down cleanly:

$ sudo openstack-service stop
$ sudo ip netns delete $(ip netns)
$ sudo poweroff

This does not fix it for me. Even after doing this I am running into

welcome to emergency mode.....

Bug reproduced after deployment of director on customer site.

System: RHEL 7.2
Kernel: 3.10.0-327.18.2.el7.x86_64
Host: VMware ESXi 5.5

After deploying director with "openstack undercloud install".

The server do not shutdown with the error:
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh' : Read-only file system

The server do not boot, only access to partition /boot during the boot but not other partitions.

*** Bug 1318935 has been marked as a duplicate of this bug. ***

We are experiencing exactly the same issue. After deploying director with "openstack undercloud install" and attempting to reboot will yield 'rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system.' which eventually ends up with repair prompt.

System: RHEL 7.2
Kernel: 3.10.0-327.18.2.el7.x86_64
Host: VMware ESXi 5.5

-----CUT HERE----

$ uname -a
Linux localhost 3.10.0-327.18.2.el7.x86_64 #1 SMP Fri Apr 8 05:09:53 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

$ rpm -qa | grep dracut
dracut-033-360.el7_2.x86_64
dracut-network-033-360.el7_2.x86_64
dracut-config-rescue-033-360.el7_2.x86_64

$ rpm -qa  | grep -i openstack
openstack-glance-11.0.1-4.el7ost.noarch
python-openstackclient-1.7.2-1.el7ost.noarch
openstack-neutron-openvswitch-7.0.4-2.el7ost.noarch
openstack-heat-api-5.0.1-5.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-2.el7ost.noarch
openstack-ceilometer-central-5.0.2-2.el7ost.noarch
openstack-swift-2.5.0-2.el7ost.noarch
openstack-ironic-inspector-2.2.5-2.el7ost.noarch
openstack-heat-engine-5.0.1-5.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-neutron-common-7.0.4-2.el7ost.noarch
openstack-neutron-7.0.4-2.el7ost.noarch
openstack-tripleo-common-0.3.1-1.el7ost.noarch
openstack-ironic-conductor-4.2.2-4.el7ost.noarch
openstack-neutron-ml2-7.0.4-2.el7ost.noarch
openstack-keystone-8.0.1-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-11.el7ost.noarch
openstack-nova-cert-12.0.3-1.el7ost.noarch
openstack-ceilometer-common-5.0.2-2.el7ost.noarch
openstack-ceilometer-collector-5.0.2-2.el7ost.noarch
openstack-ironic-common-4.2.2-4.el7ost.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-11.el7ost.noarch
openstack-heat-templates-0-0.1.20151019.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-ceilometer-api-5.0.2-2.el7ost.noarch
openstack-ceilometer-alarm-5.0.2-2.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.1-5.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
openstack-ceilometer-notification-5.0.2-2.el7ost.noarch
openstack-ceilometer-polling-5.0.2-2.el7ost.noarch
openstack-nova-conductor-12.0.3-1.el7ost.noarch
openstack-heat-common-5.0.1-5.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-nova-common-12.0.3-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch
openstack-selinux-0.6.58-1.el7ost.noarch
openstack-nova-compute-12.0.3-1.el7ost.noarch
openstack-nova-scheduler-12.0.3-1.el7ost.noarch
openstack-nova-api-12.0.3-1.el7ost.noarch
openstack-tripleo-0.0.7-1.el7ost.noarch
openstack-ironic-api-4.2.2-4.el7ost.noarch
openstack-heat-api-cfn-5.0.1-5.el7ost.noarch

$ openstack-service status
MainPID=21251 Id=neutron-dhcp-agent.service ActiveState=active
MainPID=22404 Id=neutron-openvswitch-agent.service ActiveState=active
MainPID=22349 Id=neutron-server.service ActiveState=active
MainPID=20905 Id=openstack-ceilometer-alarm-evaluator.service ActiveState=active
MainPID=20821 Id=openstack-ceilometer-alarm-notifier.service ActiveState=active
MainPID=20860 Id=openstack-ceilometer-api.service ActiveState=active
MainPID=20782 Id=openstack-ceilometer-central.service ActiveState=active
MainPID=20723 Id=openstack-ceilometer-collector.service ActiveState=active
MainPID=20683 Id=openstack-ceilometer-notification.service ActiveState=active
MainPID=22262 Id=openstack-glance-api.service ActiveState=active
MainPID=22217 Id=openstack-glance-registry.service ActiveState=active
MainPID=22556 Id=openstack-heat-api-cfn.service ActiveState=active
MainPID=22685 Id=openstack-heat-api-cloudwatch.service ActiveState=active
MainPID=22640 Id=openstack-heat-api.service ActiveState=active
MainPID=22602 Id=openstack-heat-engine.service ActiveState=active
MainPID=19092 Id=openstack-ironic-api.service ActiveState=active
MainPID=23608 Id=openstack-ironic-conductor.service ActiveState=active
MainPID=17951 Id=openstack-ironic-inspector-dnsmasq.service ActiveState=active
MainPID=17912 Id=openstack-ironic-inspector.service ActiveState=active
MainPID=20490 Id=openstack-nova-api.service ActiveState=active
MainPID=20439 Id=openstack-nova-cert.service ActiveState=active
MainPID=24187 Id=openstack-nova-compute.service ActiveState=active
MainPID=20595 Id=openstack-nova-conductor.service ActiveState=active
MainPID=20551 Id=openstack-nova-scheduler.service ActiveState=active
MainPID=19732 Id=openstack-swift-account-auditor.service ActiveState=active
MainPID=19439 Id=openstack-swift-account-reaper.service ActiveState=active
MainPID=19943 Id=openstack-swift-account-replicator.service ActiveState=active
MainPID=19982 Id=openstack-swift-account.service ActiveState=active
MainPID=19570 Id=openstack-swift-container-auditor.service ActiveState=active
MainPID=20050 Id=openstack-swift-container-replicator.service ActiveState=active
MainPID=20272 Id=openstack-swift-container-updater.service ActiveState=active
MainPID=20087 Id=openstack-swift-container.service ActiveState=active
MainPID=19695 Id=openstack-swift-object-auditor.service ActiveState=active
MainPID=20226 Id=openstack-swift-object-replicator.service ActiveState=active
MainPID=19775 Id=openstack-swift-object-updater.service ActiveState=active
MainPID=20189 Id=openstack-swift-object.service ActiveState=active
MainPID=19887 Id=openstack-swift-proxy.service ActiveState=active


------------

I've also seen this same issue many times. I'm currently running RDO Liberty. Here's the screen output from the SOL connection:

[425763.974877] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425763.980487] dracut: Disassembling device-mapper devices
[425763.983092] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425763.988309] dracut: Disassembling device-mapper devices
[425763.990867] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425763.996284] dracut: Disassembling device-mapper devices
[425764.000388] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.005928] dracut: Disassembling device-mapper devices
[425764.008458] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.013271] dracut: Disassembling device-mapper devices
[425764.015728] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.020493] dracut: Disassembling device-mapper devices
[425764.023061] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.027925] dracut: Disassembling device-mapper devices

I am having the exact same issue. This was a RDO packstack deployment with Mitaka

I encountered this bug and tried updating dracut to see if the fix from https://bugzilla.redhat.com/show_bug.cgi?id=1178497#c26 was included in the latest versions. The 033-360.el7_2.1 version does include the fix and I confirmed that I was successfully able to reboot.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2530.html