This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1178497 - unable to shutdown - dracut loop rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
unable to shutdown - dracut loop rm: cannot remove /lib/drauct/hooks/shutdown...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dracut (Show other bugs)
7.2
x86_64 Linux
urgent Severity urgent
: rc
: 7.3
Assigned To: Lukáš Nykrýn
Release Test Team
Maxim Svistunov
: Reopened, ZStream
: 1318935 (view as bug list)
Depends On: 1154739
Blocks: 1203710 1261979 1289485 1313485 1338759
  Show dependency treegraph
 
Reported: 2015-01-04 05:39 EST by Andrea Florio
Modified: 2016-11-04 03:56 EDT (History)
50 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: If a mount point resides in /run on shutdown and cannot be unmounted, systemd will mount this mountpoint read-only. This propagates to /run being read-only. Consequence: When the dracut shutdown procedure runs in a /run root to disassemble and unmount the old real root, its /run is read-only. dracut expects a writable /run, though, so it fails on shutdown. Fix: If dracut encounters a read-only /run on shutdown, it remounts it writeable. Result: The dracut shutdown procedure to disassemble and unmount the old real root succeeds.
Story Points: ---
Clone Of:
: 1253527 1338759 (view as bug list)
Environment:
Last Closed: 2016-11-04 03:56:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Photo of the error produced at Shutdown.target, dracut (4.71 MB, image/jpeg)
2015-01-22 15:01 EST, linforpros
no flags Details
The log obtained via kernel command line (1014.86 KB, text/plain)
2015-01-22 16:33 EST, linforpros
no flags Details
screenshot of shutdown issue (58.25 KB, image/png)
2015-11-26 15:37 EST, Vincent S. Cojot
no flags Details
shutdown-log.txt (1.04 MB, text/plain)
2015-11-26 15:44 EST, Vincent S. Cojot
no flags Details
small movie of console while shutting down.. (672.51 KB, application/octet-stream)
2015-11-26 15:45 EST, Vincent S. Cojot
no flags Details
Screen shot of console error during shutdown (183.51 KB, image/gif)
2016-02-03 08:43 EST, Chris Dearborn
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2160501 None None None 2016-06-30 00:26 EDT
Red Hat Knowledge Base (Solution) 2194521 None None None 2016-06-08 03:15 EDT

  None (edit)
Description Andrea Florio 2015-01-04 05:39:40 EST
Description of problem:

my system:

Linux localhost.localdomain 3.17.7-300.fc21.x86_64 #1 SMP Wed Dec 17 03:08:44 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Every time i try to shutdown/reboot/poweroff the system it doesn't work.

On the screen you onlt see this message popping up endless as in a loop:

rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem

this file doesn't even exist

Version-Release number of selected component (if applicable):

[root@localhost ~]# rpm -qi dracut
Name        : dracut
Version     : 038
Release     : 31.git20141204.fc21
Architecture: x86_64
Install Date: Wed 31 Dec 2014 06:57:40 PM CET
Group       : System Environment/Base
Size        : 922575
License     : GPLv2+ and LGPLv2+
Signature   : RSA/SHA256, Thu 04 Dec 2014 03:57:25 PM CET, Key ID 89ad4e8795a43f54
Source RPM  : dracut-038-31.git20141204.fc21.src.rpm
Build Date  : Thu 04 Dec 2014 12:10:53 PM CET
Build Host  : buildhw-01.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://dracut.wiki.kernel.org/
Summary     : Initramfs generator using udev
Description :
dracut contains tools to create a bootable initramfs for 2.6 Linux kernels.
Unlike existing implementations, dracut does hard-code as little as possible
into the initramfs. dracut contains various modules which are driven by the
event-based udev. Having root on MD, DM, LVM2, LUKS is supported as well as
NFS, iSCSI, NBD, FCoE with the dracut-network package.


How reproducible:

I'm not sure what triggered it. I have 2 Fedora 21 virtual machines, one is effected, the other one is not, but i am not the only one effected:

http://forums.fedoraforum.org/showthread.php?t=302352

http://web.archiveorange.com/archive/v/kMNMPYKVVgT1Cbn43Aes

Steps to Reproduce:
Just reboot the system

Actual results:

endless loop

Expected results:

system terminates the shutdown process properly
Additional info:
Comment 1 Andrea Florio 2015-01-04 05:46:47 EST
yum reinstall dracut didn't help.

uninstalling dracut made the trick and i can turn off the system, but obviously this is not the desired way.

i am going to re-install dracut now, to see if the problem is showing again
Comment 2 Andrea Florio 2015-01-04 05:48:15 EST
yes, re-installing dracut with the removed dependencies (dracut abrt-addon-vmcore abrt-cli dracut-config-rescue dracut-network kexec-tools plymouth-scripts)

causes the problem to re-appear
Comment 3 linforpros 2015-01-04 15:52:17 EST
Hi Adrea,
systemctl reboot -f -f

helps for three shutdowns/reboot/poweroffs. Interesting. A Colin Walters who is on the fedora-cloud-atomic team gave a patch for an alleged lvm bug which is referenced in the link you provided.

But it is a no permanent solution.

+# Work around anaconda/dracut/lvm bug
+sync
+sync
+sync
+systemctl reboot -f -f
+%end

LinforPros
Comment 4 Andrea Florio 2015-01-04 16:24:53 EST
Sadly that doesn't help me. 
Also, as stated, removing Dracut allows me turn off the machine and installing back restore the problem.
Comment 5 linforpros 2015-01-06 15:18:29 EST
I hate to inform you that the following action has nullified the bug. I "hate" to do that because we will never know what had happened and why it occurred in the first place. In other words it is a lesson not learned. 



Packages Altered:
    Updated dracut-038-31.git20141204.fc21.x86_64               @updates
    Update         038-32.git20141216.fc21.x86_64               @updates
    Updated dracut-config-rescue-038-31.git20141204.fc21.x86_64 @updates
    Update                       038-32.git20141216.fc21.x86_64 @updates
    Updated dracut-network-038-31.git20141204.fc21.x86_64       @updates
    Update                 038-32.git20141216.fc21.x86_64       @updates


After the update I have successfully shutdown/reboot/poweroff my machine 7 times with no recurring problems.


LinforPros
Comment 6 linforpros 2015-01-06 16:42:59 EST
My apologies,
the problem persists. this is insane. After 8th reboot the nineth one produces the same result. Server cannot be reliably rebooted/powered off.

LinforPros.
Comment 7 Harald Hoyer 2015-01-08 11:05:11 EST
Can you please follow the instruction of the dracut man page about debugging shutdown?

 # mkdir -p /run/initramfs/etc/cmdline.d
 # echo "rd.debug rd.break=pre-shutdown rd.break=shutdown" > /run/initramfs/etc/cmdline.d/debug.conf
 # touch /run/initramfs/.need_shutdown

And maybe attach a photo of the screen. Some keyboards have a "Pause" or "ScrollLck" key to stop the output.


A workaround to prevent the loop at shutdown, is to remove "/run/initramfs/.need_shutdown" prior to shutdown, or to mask the dracut-shutdown.service


# systemctl mask dracut-shutdown.service


But please provide me with some debugging info first.

Oh, and please also do
http://freedesktop.org/wiki/Software/systemd/Debugging/#diagnosingshutdownproblems

add 
systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M enforcing=0
to the kernel command line and create 
 /usr/lib/systemd/system-shutdown/debug.sh
with the contents

#!/bin/sh
mount -o remount,rw /
dmesg > /shutdown-log.txt
mount -o remount,ro /

... then do:
# chmod a+x /usr/lib/systemd/system-shutdown/debug.sh

And attach /shutdown-log.txt to this bug report.

Many thanks!!!
Comment 8 linforpros 2015-01-22 15:01:07 EST
Created attachment 983031 [details]
Photo of the error produced at Shutdown.target, dracut

As per the request to attach a photo, here it is.
LinforPros
Comment 9 linforpros 2015-01-22 16:33:13 EST
Created attachment 983078 [details]
The log obtained via kernel command line

Hello, I am pleased to provide the log you suggested in you post. 
Just wanted to mention that the error showed up during two separate fedora installations albeit on the same machine. Always only after packstack --allinone installation. I regret that I failed to do an LVM snapshot so I could just start over. I will have to reinstall fedora again. Unless we find a solution to the problem faster :-)
thank you for looking at the log. If you can decipher anything from it that'd be great.

LinforPros
Comment 10 linforpros 2015-01-22 16:52:30 EST
Hi Herald,

On the final note I would like to mention that:

sync && reboot -f
sync && poweroff -f

work perfectly well. Which suggests that it is not a kernel problem.

So does #systemctl mask dracut-shutdown.service help avoid the loop.

LinforPros
Comment 11 Harald Hoyer 2015-01-28 08:07:01 EST
Hmm, I can only think of two things:
1. /run was not mounted with a tmpfs
2. /run was remounted read-only

Can you extend /usr/lib/systemd/system-shutdown/debug.sh
and add

cat /proc/mounts >> /shutdown-log.txt
cat /proc/self/mountinfo >> /shutdown-log.txt

after the dmesg line? So, I can confirm either of the two theories.
Comment 12 Fedora End Of Life 2015-11-04 10:33:21 EST
This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 13 Vincent S. Cojot 2015-11-25 22:57:11 EST
Hi,
I've re-opened this bug as I am now seeing this on 3 different virtual machines on RHEL7.2 (still running with the latest 7.1 kernel) with several different releases of OpenStack (kilo/RDO, OSP7 and liberty/RDO).
I am gathering more information, and will update this bug accordingly.
Components:
# rpm -qa dracut\*
dracut-network-033-359.el7.x86_64
dracut-033-359.el7.x86_64
dracut-config-rescue-033-359.el7.x86_64

all el7.2 patches current as of today (20151125) except for the kernel:
[root@rh7x64 ~]# uname -a
Linux rh7x64 3.10.0-229.20.1.el7.x86_64 #1 SMP Thu Sep 24 12:23:56 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@rh7x64 ~]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-ia32:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.2 (Maipo)
Release:        7.2
Codename:       Maipo
Comment 14 Vincent S. Cojot 2015-11-26 00:10:16 EST
Please note that to work around the issue, dracut can be invoked like this:

# dracut -f -o shutdown

(this excludes the entire 'shutdown' dracut module).

To disable this on a permanent basis and so subsequent kernel updates don't hang the systems either, I did this:

# cat /etc/dracut.conf.d/shut.conf 
omit_dracutmodules+="shutdown"

this makes dracut skip the 'shutdown' module in every case. (could have other side effects).
Comment 15 Harald Hoyer 2015-11-26 10:13:28 EST
(In reply to Vincent S. Cojot from comment #14)
> Please note that to work around the issue, dracut can be invoked like this:
> 
> # dracut -f -o shutdown
> 
> (this excludes the entire 'shutdown' dracut module).
> 
> To disable this on a permanent basis and so subsequent kernel updates don't
> hang the systems either, I did this:
> 
> # cat /etc/dracut.conf.d/shut.conf 
> omit_dracutmodules+="shutdown"
> 
> this makes dracut skip the 'shutdown' module in every case. (could have
> other side effects).

# systemctl mask dracut-shutdown.service 

is a better short-term workaround.
Comment 16 Harald Hoyer 2015-11-26 10:15:05 EST
please give me debug info as mentioned in comment #7 and comment #11
Comment 17 Vincent S. Cojot 2015-11-26 15:23:04 EST
Hello Harald,
Just saw your reply. I will be providing the information shortly.
A few things to note:
All of the systems with the issue are Virtual Machines.
I just realized that all of them and on a VMWare Hypervisor with the VMWare tools installed.
Let me gather more debugging information.
Thank you for your patience,
Vincent
Comment 18 Vincent S. Cojot 2015-11-26 15:31:49 EST
Currently following your instructions with:

[root@osp2 ~]# cat /usr/lib/systemd/system-shutdown/debug.sh
#!/bin/sh
mount -o remount,rw /
dmesg > /shutdown-log.txt
ps -faux >> /shutdown-log.txt
cat /proc/mounts >> /shutdown-log.txt
cat /proc/self/mountinfo >> /shutdown-log.txt
mount -o remount,ro /

Please note that 3/4 machines on which I have the issue have a PackStack of some kind installed (1 is RDO/kilo, 1 is OSP7, 1 is RDO/Liberty). The last (and 4th) VM doesn't have packstack installed.

Regards,

Vincent
Comment 19 Vincent S. Cojot 2015-11-26 15:37 EST
Created attachment 1099432 [details]
screenshot of shutdown issue
Comment 20 Vincent S. Cojot 2015-11-26 15:44 EST
Created attachment 1099433 [details]
shutdown-log.txt
Comment 21 Vincent S. Cojot 2015-11-26 15:45 EST
Created attachment 1099434 [details]
small movie of console while shutting down..
Comment 22 Harald Hoyer 2015-11-27 06:03:11 EST
tmpfs /run tmpfs ro,seclabel,nosuid,nodev,mode=755 0 0

23 59 0:18 / /run rw,nosuid,nodev shared:22 - tmpfs tmpfs ro,seclabel,mode=755

So, /run is mounted read-only suddenly.

Does any of your custom scripts mount /run read-only?
Comment 23 Harald Hoyer 2015-11-27 06:07:37 EST
What are these, and why can't they be umounted???

[  237.377280] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.377406] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.377433] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.377522] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.377566] systemd-shutdown[1]: Unmounting /run/netns.
[  237.477388] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.477402] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.482242] systemd-shutdown[1]: Unmounting /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b.
[  237.482260] systemd-shutdown[1]: Could not unmount /run/netns/qdhcp-d2ae05cc-2fd0-4a98-b25b-cf5c0b75772b: Invalid argument
[  237.482266] systemd-shutdown[1]: Unmounting /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1.
[  237.482271] systemd-shutdown[1]: Could not unmount /run/netns/qrouter-0fd9aca1-3898-45f9-8bf8-8f78b659ffa1: Invalid argument
[  237.487535] systemd-shutdown[1]: Not all file systems unmounted, 2 left.
Comment 24 Harald Hoyer 2015-11-27 06:11:10 EST
Seems like not all net namespaces are cleanup.
Comment 25 Vincent S. Cojot 2015-11-27 08:47:13 EST
Hi Harald,
These things are OpenVswitch network namespaces..
And NO, I don't have any script that re-mounts /run read-only (that I know of).
I am not sure if we could do something about it but at any case it shouldn't prevent shutdown.
Let me do the tests again but this time with a non-OpenStack VM..
Regards,
Vincent
Comment 26 Harald Hoyer 2015-11-30 05:25:29 EST
(In reply to Vincent S. Cojot from comment #25)
> Hi Harald,
> These things are OpenVswitch network namespaces..
> And NO, I don't have any script that re-mounts /run read-only (that I know
> of).
> I am not sure if we could do something about it but at any case it shouldn't
> prevent shutdown.
> Let me do the tests again but this time with a non-OpenStack VM..
> Regards,
> Vincent

Ok, please try the following steps:

edit /usr/lib/dracut/modules.d/99shutdown/shutdown.sh

after:
. /lib/dracut-lib.sh

add:
if [ "$(stat -c '%T' -f /)" = "tmpfs" ]; then
    mount -o remount,rw /
fi

edit /usr/lib/dracut/modules.d/99shutdown/module-setup.sh
change:
    inst_multiple umount poweroff reboot halt losetup
to:
    inst_multiple umount poweroff reboot halt losetup stat


recreate the initramfs:
# dracut --force

unmask the shutdown:
# systemctl unmask dracut-shutdown.service.

and reboot.
Comment 27 Vincent S. Cojot 2015-11-30 09:33:15 EST
Hi Harald,

Prior to performing these changes, shall I undo the previous ones?

Should I remove /usr/lib/systemd/system-shutdown/debug.sh and /run/initramfs/etc/cmdline.d ? Same for the extra kernel args?

Thanks,

Vincent
Comment 28 Vincent S. Cojot 2015-11-30 10:26:37 EST
I can confirm that your change fixes the issue for my OSP7 allinone VM.
The resulting patch is:
[root@osp2 ~]# less /tmp/dracut.patch 
*** /usr/lib/dracut/modules.d/99shutdown/shutdown.sh.orig       Mon Nov 30 10:21:11 2015
--- /usr/lib/dracut/modules.d/99shutdown/shutdown.sh    Mon Nov 30 10:21:22 2015
***************
*** 13,18 ****
--- 13,21 ----
  export TERM=linux
  export PATH=/usr/sbin:/usr/bin:/sbin:/bin
  . /lib/dracut-lib.sh
+ if [ "$(stat -c '%T' -f /)" = "tmpfs" ]; then
+     mount -o remount,rw /
+ fi
  
  mkdir /oldsys
  for i in sys proc run dev; do
*** /usr/lib/dracut/modules.d/99shutdown/module-setup.sh.orig   Mon Nov 30 10:21:55 2015
--- /usr/lib/dracut/modules.d/99shutdown/module-setup.sh        Mon Nov 30 10:22:22 2015
***************
*** 13,19 ****
  
  install() {
      local _d
!     inst_multiple umount poweroff reboot halt losetup
      inst_multiple -o kexec
      inst "$moddir/shutdown.sh" "$prefix/shutdown"
      [ -e "${initdir}/lib" ] || mkdir -m 0755 -p ${initdir}/lib
--- 13,19 ----
  
  install() {
      local _d
!     inst_multiple umount poweroff reboot halt losetup stat
      inst_multiple -o kexec
      inst "$moddir/shutdown.sh" "$prefix/shutdown"
      [ -e "${initdir}/lib" ] || mkdir -m 0755 -p ${initdir}/lib
Comment 29 Harald Hoyer 2015-11-30 12:12:37 EST
(In reply to Vincent S. Cojot from comment #28)
> I can confirm that your change fixes the issue for my OSP7 allinone VM.

Thank you very much for your testing!
Comment 30 Vincent S. Cojot 2015-11-30 13:20:39 EST
You are most welcomed, Harald!
Please feel free to send me updated test-cases/patches to test, etc...
This seemed pretty major.. do you think upstream dracut could be affected by the issue as well?
Regards,
Vincent
Comment 31 Harald Hoyer 2015-12-01 02:46:19 EST
(In reply to Vincent S. Cojot from comment #30)
> You are most welcomed, Harald!
> Please feel free to send me updated test-cases/patches to test, etc...
> This seemed pretty major.. do you think upstream dracut could be affected by
> the issue as well?
> Regards,
> Vincent

already pushed to upstream:
https://github.com/haraldh/dracut/commit/54e09dfb72b557ac8ccd48f5d37089287d272ec7
Comment 32 Vincent S. Cojot 2015-12-01 09:14:26 EST
Do you think we could expect an update on 7.2.z?
Thanks,
Vincent
Comment 36 Dan Burkland 2016-01-10 21:54:40 EST
I also hit this bug however the solution posted by Harald worked flawlessly. Any ideas on when this fix will be implemented in a RHEL release?

Thanks,

Dan
Comment 37 Rama 2016-01-14 11:05:06 EST
While implementing OSP7 with versionlock we hit with the problem too. Can this bug be updated when this will be included.
Comment 40 Chris Dearborn 2016-02-02 15:32:26 EST
We just hit this problem in OSP 8 Director.  The shutdown hangs and the following message is dumped to the console in an infinite loop:

rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only filesystem
Comment 41 Shinobu KINJO 2016-02-02 16:41:49 EST
(In reply to Chris Dearborn from comment #40)
> We just hit this problem in OSP 8 Director.  The shutdown hangs and the
> following message is dumped to the console in an infinite loop:
> 
> rm: cannot remove /lib/drauct/hooks/shutdown/30-dm-shutdown.sh: Read-only
> filesystem

@Chris,

What's your kernel?

Rgds,
Shinobu
Comment 42 Chris Dearborn 2016-02-02 17:58:29 EST
We're running RHEL 7.2:

[stack@director-r13 etc]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[stack@director-r13 ~]$ uname -a
Linux director-r13.rcbd.lab 3.10.0-327.4.5.el7.x86_64 #1 SMP Thu Jan 21 04:10:29 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Oh, should have specified above that I'm running OSP 8 Beta 4.
Comment 43 Mohammed Arafa 2016-02-02 20:55:26 EST
i dont want to point out the obvious but i see several people making typo in spelling dracut leading me to believe it was not unintentional 

/lib/draUCt
Comment 45 Chris Dearborn 2016-02-03 08:42:11 EST
Oops!  Sorry about that - the correct spelling is below.  See attached ConsoleError.gif.

rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system.
Comment 46 Chris Dearborn 2016-02-03 08:43 EST
Created attachment 1120789 [details]
Screen shot of console error during shutdown
Comment 56 Jon 2016-02-16 12:02:38 EST
Steps to reproduce issue: 

 1. Create a root filesystem on LVM volume.
 2. Add a network namespace.
    # ip netns add namespace
 3. # systemctl reboot

Shutdown infinitely loops at: 
---
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system


Work-Around noted in https://bugzilla.redhat.com/show_bug.cgi?id=1178497#c26 did indeed resolve issue.  I did not need to unmask dracut-shutdown.service.  


--- /usr/lib/dracut/modules.d/99shutdown/shutdown.sh.BAK	2016-02-16 11:56:58.750828854 -0500
+++ /usr/lib/dracut/modules.d/99shutdown/shutdown.sh	2016-02-16 11:57:28.011290818 -0500
@@ -14,6 +14,10 @@ export TERM=linux
 export PATH=/usr/sbin:/usr/bin:/sbin:/bin
 . /lib/dracut-lib.sh
 
+if [ "$(stat -c '%T' -f /)" = "tmpfs" ]; then
+    mount -o remount,rw /
+fi
+
 mkdir /oldsys
 for i in sys proc run dev; do
     mkdir /oldsys/$i


--- /usr/lib/dracut/modules.d/99shutdown/module-setup.sh.BAK	2016-02-16 11:57:38.222445577 -0500
+++ /usr/lib/dracut/modules.d/99shutdown/module-setup.sh	2016-02-16 11:58:00.330736783 -0500
@@ -13,7 +13,7 @@ depends() {
 
 install() {
     local _d
-    inst_multiple umount poweroff reboot halt losetup
+    inst_multiple umount poweroff reboot halt losetup stat
     inst_multiple -o kexec
     inst "$moddir/shutdown.sh" "$prefix/shutdown"
     [ -e "${initdir}/lib" ] || mkdir -m 0755 -p ${initdir}/lib
Comment 58 cappetta 2016-03-03 08:21:56 EST
hitting this issue on CentOS-7 -- trying the stated work around but this is definitely an annoyance!
Comment 59 Gianluca Cecchi 2016-03-11 12:14:10 EST
Just to add myself to the list.
System is CentOS 7.2 fully updated where I'm testing Openstack Icehouse provided by rdo repo and the problem appears after running packstack.
At the moment applied the workaround of masking dracut-shutdown service.
I'm available to test potential complete resolutions.
Comment 63 Stuart James 2016-04-01 09:14:57 EDT
I get this same error on shutdown

My system is setup as compute/network node connected to a controller on RHEL7.2 with OSP6

[root@node2 ~(keystone_admin)]# uname -a
Linux node2.jamesnet.ca 3.10.0-327.10.1.el7.x86_64 #1 SMP Sat Jan 23 04:54:55 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@node2 ~(keystone_admin)]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

[root@node2 ~(keystone_admin)]# openstack-service status
MainPID=2462 Id=neutron-dhcp-agent.service ActiveState=active
MainPID=2472 Id=neutron-l3-agent.service ActiveState=active
MainPID=1232 Id=neutron-metadata-agent.service ActiveState=active
MainPID=2471 Id=neutron-openvswitch-agent.service ActiveState=active
MainPID=1230 Id=openstack-ceilometer-compute.service ActiveState=active
MainPID=2461 Id=openstack-nova-compute.service ActiveState=active

[root@node2 ~(keystone_admin)]# rpm -qa  | grep -i openstack
openstack-nova-common-2014.2.3-65.el7ost.noarch
openstack-neutron-2014.2.3-33.el7ost.noarch
openstack-selinux-0.6.43-1.el7ost.noarch
openstack-neutron-common-2014.2.3-33.el7ost.noarch
openstack-neutron-openvswitch-2014.2.3-33.el7ost.noarch
openstack-ceilometer-common-2014.2.3-7.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-ceilometer-compute-2014.2.3-7.el7ost.noarch
openstack-nova-compute-2014.2.3-65.el7ost.noarch

[root@node2 ~(keystone_admin)]# rpm -qa | grep dracut
dracut-033-360.el7_2.x86_64
dracut-network-033-360.el7_2.x86_64
dracut-config-rescue-033-360.el7_2.x86_64

As a workaround i did as Herald said earlier

systemctl mask dracut-shutdown.service
Comment 64 cappetta 2016-04-29 10:44:31 EDT
Issue is still present, the systemctl mask dracut-shutdown.service resolved it as well.  Not sure what the difference is between the 2 solutions but this appears to be the easiest one to apply.  

Package info is below...

[root@mini-workstation ~]# rpm -qa | grep -i dracut
dracut-network-033-161.el7.x86_64
dracut-config-rescue-033-360.el7_2.x86_64
dracut-network-033-360.el7_2.x86_64
dracut-033-360.el7_2.x86_64
dracut-033-161.el7.x86_64

[root@mini-workstation ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
Comment 65 Ian Pilcher 2016-05-06 13:01:18 EDT
If you're hitting this on a Red Hat OpenStack Platform director system (the "undercloud"), the following procedure should enable you to shut down cleanly:

$ sudo openstack-service stop
$ sudo ip netns delete $(ip netns)
$ sudo poweroff
Comment 66 ldomb 2016-05-06 14:00:38 EDT
This does not fix it for me. Even after doing this I am running into

welcome to emergency mode.....
Comment 72 Erwan Gallen 2016-05-24 07:31:59 EDT
Bug reproduced after deployment of director on customer site.

System: RHEL 7.2
Kernel: 3.10.0-327.18.2.el7.x86_64
Host: VMware ESXi 5.5

After deploying director with "openstack undercloud install".

The server do not shutdown with the error:
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh' : Read-only file system

The server do not boot, only access to partition /boot during the boot but not other partitions.
Comment 73 Martin Schuppert 2016-05-27 05:05:06 EDT
*** Bug 1318935 has been marked as a duplicate of this bug. ***
Comment 74 Ivan Bojer 2016-05-31 16:45:58 EDT
We are experiencing exactly the same issue. After deploying director with "openstack undercloud install" and attempting to reboot will yield 'rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system.' which eventually ends up with repair prompt.

System: RHEL 7.2
Kernel: 3.10.0-327.18.2.el7.x86_64
Host: VMware ESXi 5.5

-----CUT HERE----

$ uname -a
Linux localhost 3.10.0-327.18.2.el7.x86_64 #1 SMP Fri Apr 8 05:09:53 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

$ rpm -qa | grep dracut
dracut-033-360.el7_2.x86_64
dracut-network-033-360.el7_2.x86_64
dracut-config-rescue-033-360.el7_2.x86_64

$ rpm -qa  | grep -i openstack
openstack-glance-11.0.1-4.el7ost.noarch
python-openstackclient-1.7.2-1.el7ost.noarch
openstack-neutron-openvswitch-7.0.4-2.el7ost.noarch
openstack-heat-api-5.0.1-5.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-2.el7ost.noarch
openstack-ceilometer-central-5.0.2-2.el7ost.noarch
openstack-swift-2.5.0-2.el7ost.noarch
openstack-ironic-inspector-2.2.5-2.el7ost.noarch
openstack-heat-engine-5.0.1-5.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-neutron-common-7.0.4-2.el7ost.noarch
openstack-neutron-7.0.4-2.el7ost.noarch
openstack-tripleo-common-0.3.1-1.el7ost.noarch
openstack-ironic-conductor-4.2.2-4.el7ost.noarch
openstack-neutron-ml2-7.0.4-2.el7ost.noarch
openstack-keystone-8.0.1-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-11.el7ost.noarch
openstack-nova-cert-12.0.3-1.el7ost.noarch
openstack-ceilometer-common-5.0.2-2.el7ost.noarch
openstack-ceilometer-collector-5.0.2-2.el7ost.noarch
openstack-ironic-common-4.2.2-4.el7ost.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-11.el7ost.noarch
openstack-heat-templates-0-0.1.20151019.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-ceilometer-api-5.0.2-2.el7ost.noarch
openstack-ceilometer-alarm-5.0.2-2.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.1-5.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
openstack-ceilometer-notification-5.0.2-2.el7ost.noarch
openstack-ceilometer-polling-5.0.2-2.el7ost.noarch
openstack-nova-conductor-12.0.3-1.el7ost.noarch
openstack-heat-common-5.0.1-5.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-nova-common-12.0.3-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch
openstack-selinux-0.6.58-1.el7ost.noarch
openstack-nova-compute-12.0.3-1.el7ost.noarch
openstack-nova-scheduler-12.0.3-1.el7ost.noarch
openstack-nova-api-12.0.3-1.el7ost.noarch
openstack-tripleo-0.0.7-1.el7ost.noarch
openstack-ironic-api-4.2.2-4.el7ost.noarch
openstack-heat-api-cfn-5.0.1-5.el7ost.noarch

$ openstack-service status
MainPID=21251 Id=neutron-dhcp-agent.service ActiveState=active
MainPID=22404 Id=neutron-openvswitch-agent.service ActiveState=active
MainPID=22349 Id=neutron-server.service ActiveState=active
MainPID=20905 Id=openstack-ceilometer-alarm-evaluator.service ActiveState=active
MainPID=20821 Id=openstack-ceilometer-alarm-notifier.service ActiveState=active
MainPID=20860 Id=openstack-ceilometer-api.service ActiveState=active
MainPID=20782 Id=openstack-ceilometer-central.service ActiveState=active
MainPID=20723 Id=openstack-ceilometer-collector.service ActiveState=active
MainPID=20683 Id=openstack-ceilometer-notification.service ActiveState=active
MainPID=22262 Id=openstack-glance-api.service ActiveState=active
MainPID=22217 Id=openstack-glance-registry.service ActiveState=active
MainPID=22556 Id=openstack-heat-api-cfn.service ActiveState=active
MainPID=22685 Id=openstack-heat-api-cloudwatch.service ActiveState=active
MainPID=22640 Id=openstack-heat-api.service ActiveState=active
MainPID=22602 Id=openstack-heat-engine.service ActiveState=active
MainPID=19092 Id=openstack-ironic-api.service ActiveState=active
MainPID=23608 Id=openstack-ironic-conductor.service ActiveState=active
MainPID=17951 Id=openstack-ironic-inspector-dnsmasq.service ActiveState=active
MainPID=17912 Id=openstack-ironic-inspector.service ActiveState=active
MainPID=20490 Id=openstack-nova-api.service ActiveState=active
MainPID=20439 Id=openstack-nova-cert.service ActiveState=active
MainPID=24187 Id=openstack-nova-compute.service ActiveState=active
MainPID=20595 Id=openstack-nova-conductor.service ActiveState=active
MainPID=20551 Id=openstack-nova-scheduler.service ActiveState=active
MainPID=19732 Id=openstack-swift-account-auditor.service ActiveState=active
MainPID=19439 Id=openstack-swift-account-reaper.service ActiveState=active
MainPID=19943 Id=openstack-swift-account-replicator.service ActiveState=active
MainPID=19982 Id=openstack-swift-account.service ActiveState=active
MainPID=19570 Id=openstack-swift-container-auditor.service ActiveState=active
MainPID=20050 Id=openstack-swift-container-replicator.service ActiveState=active
MainPID=20272 Id=openstack-swift-container-updater.service ActiveState=active
MainPID=20087 Id=openstack-swift-container.service ActiveState=active
MainPID=19695 Id=openstack-swift-object-auditor.service ActiveState=active
MainPID=20226 Id=openstack-swift-object-replicator.service ActiveState=active
MainPID=19775 Id=openstack-swift-object-updater.service ActiveState=active
MainPID=20189 Id=openstack-swift-object.service ActiveState=active
MainPID=19887 Id=openstack-swift-proxy.service ActiveState=active


------------
Comment 75 Bret Edwards 2016-06-03 19:57:54 EDT
I've also seen this same issue many times. I'm currently running RDO Liberty. Here's the screen output from the SOL connection:

[425763.974877] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425763.980487] dracut: Disassembling device-mapper devices
[425763.983092] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425763.988309] dracut: Disassembling device-mapper devices
[425763.990867] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425763.996284] dracut: Disassembling device-mapper devices
[425764.000388] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.005928] dracut: Disassembling device-mapper devices
[425764.008458] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.013271] dracut: Disassembling device-mapper devices
[425764.015728] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.020493] dracut: Disassembling device-mapper devices
[425764.023061] device-mapper: ioctl: remove_all left 1 open device(s)
rm: cannot remove '/lib/dracut/hooks/shutdown/30-dm-shutdown.sh': Read-only file system
[425764.027925] dracut: Disassembling device-mapper devices
Comment 76 Donny Davis 2016-06-04 13:05:53 EDT
I am having the exact same issue. This was a RDO packstack deployment with Mitaka
Comment 78 AlexB 2016-08-02 21:20:29 EDT
I encountered this bug and tried updating dracut to see if the fix from https://bugzilla.redhat.com/show_bug.cgi?id=1178497#c26 was included in the latest versions. The 033-360.el7_2.1 version does include the fix and I confirmed that I was successfully able to reboot.
Comment 85 errata-xmlrpc 2016-11-04 03:56:41 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2530.html

Note You need to log in before you can comment on or make changes to this bug.