Bug 1287940 - Creating an LVM snapshot can render the system unbootable
Summary: Creating an LVM snapshot can render the system unbootable
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dracut
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: dracut-maint
QA Contact: Release Test Team
URL:
Whiteboard:
: 1272162 (view as bug list)
Depends On:
Blocks: 1289485 1313485 1469559
TreeView+ depends on / blocked
 
Reported: 2015-12-03 04:34 UTC by Gordon Messmer
Modified: 2020-07-16 08:39 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-04 08:01:48 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2359851 None None None 2016-06-08 17:02:10 UTC
Red Hat Product Errata RHBA-2016:2530 normal SHIPPED_LIVE dracut bug fix and enhancement update 2016-11-03 14:17:01 UTC

Description Gordon Messmer 2015-12-03 04:34:57 UTC
Description of problem:
If there are no LVM snapshots when dracut is run, it will not include dm-snapshot.ko in the initrd.  If a snapshot is later created, and the system reboots, the boot process will not complete successfully.

The initrd created by anaconda does include dm-snapshot, so to reproduce the problem one must install, then update, then create a snapshot.

The simple solution is to always include dm-snapshot for systems with LVM filesystems.

Version-Release number of selected component (if applicable):
dracut-033-241.el7_1.5.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install system with default LVM setup.  Use 7.0 or 7.1 so that there will be a kernel update pending after installation.
2. Install newer kernel (yum update)
3. Create a snapshot (lvcreate -L 1G -s -n root_snap /dev/VolGroup/lv_root)
4. Reboot

Actual results:
System will hang at plymouth screen for a while, then drop to a dracut shell.

Expected results:
Normal boot.

Additional info:



[    0.832316] localhost.localdomain dracut-initqueue[258]: Scanning devices vda3  for LVM logical volumes VolGroup/lv_root VolGroup/lv_swap
[    0.843334] localhost.localdomain kernel: usb 1-1: New USB device found, idVendor=0627, idProduct=0001
[    0.843338] localhost.localdomain kernel: usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
[    0.843350] localhost.localdomain kernel: usb 1-1: Product: QEMU USB Tablet
[    0.843351] localhost.localdomain kernel: usb 1-1: Manufacturer: QEMU
[    0.843353] localhost.localdomain kernel: usb 1-1: SerialNumber: 42
[    0.846908] localhost.localdomain kernel: input: QEMU QEMU USB Tablet as /devices/pci0000:00/0000:00:05.7/usb1/1-1/1-1:1.0/input/input2
[    0.847592] localhost.localdomain dracut-initqueue[258]: inactive '/dev/VolGroup/lv_var' [12.91 GiB] inherit
[    0.847805] localhost.localdomain dracut-initqueue[258]: inactive '/dev/VolGroup/lv_swap' [2.00 GiB] inherit
[    0.848030] localhost.localdomain dracut-initqueue[258]: inactive Original '/dev/VolGroup/lv_root' [20.00 GiB] inherit
[    0.848277] localhost.localdomain dracut-initqueue[258]: inactive Snapshot '/dev/VolGroup/lv_root-snap' [2.00 GiB] inherit
[    0.847069] localhost.localdomain kernel: hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU QEMU USB Tablet] on usb-0000:00:05.7-1/input0
[    0.865914] localhost.localdomain dracut-initqueue[258]: modprobe: FATAL: Module dm-snapshot not found.
[    0.866363] localhost.localdomain dracut-initqueue[258]: /usr/sbin/modprobe failed: 1
[    0.866764] localhost.localdomain dracut-initqueue[258]: Can't process LV lv_root-snap: snapshot target support missing from kernel?
[    1.253710] localhost.localdomain kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
[  124.378169] localhost.localdomain dracut-initqueue[258]: Scanning devices vda3  for LVM logical volumes VolGroup/lv_root VolGroup/lv_swap
[  124.439876] localhost.localdomain dracut-initqueue[258]: inactive '/dev/VolGroup/lv_var' [12.91 GiB] inherit
[  124.440313] localhost.localdomain dracut-initqueue[258]: ACTIVE '/dev/VolGroup/lv_swap' [2.00 GiB] inherit
[  124.440701] localhost.localdomain dracut-initqueue[258]: inactive Original '/dev/VolGroup/lv_root' [20.00 GiB] inherit
[  124.441074] localhost.localdomain dracut-initqueue[258]: inactive Snapshot '/dev/VolGroup/lv_root-snap' [2.00 GiB] inherit
[  124.444998] localhost.localdomain dracut-initqueue[258]: PARTIAL MODE. Incomplete logical volumes will be processed.
[  124.463688] localhost.localdomain dracut-initqueue[258]: modprobe: FATAL: Module dm-snapshot not found.
[  124.463978] localhost.localdomain dracut-initqueue[258]: /usr/sbin/modprobe failed: 1
[  124.464281] localhost.localdomain dracut-initqueue[258]: Can't process LV lv_root-snap: snapshot target support missing from kernel?
[  124.470154] localhost.localdomain dracut-initqueue[258]: PARTIAL MODE. Incomplete logical volumes will be processed.
[  185.193123] localhost.localdomain dracut-initqueue[258]: Warning: Could not boot.
[  185.199959] localhost.localdomain dracut-initqueue[258]: Warning: /dev/VolGroup/lv_root does not exist
[  185.201547] localhost.localdomain dracut-initqueue[258]: Warning: /dev/mapper/VolGroup-lv_root does not exist
[  185.210453] localhost.localdomain systemd[1]: Starting Dracut Emergency Shell...

Comment 1 Harald Hoyer 2015-12-08 10:47:13 UTC
Thanks for your finding!

Comment 2 ilmostro7 2016-02-15 08:33:27 UTC
Still occurring on 7.2.  From the looks of it, this has been happening for many years now on multiple distributions. Frustrating! Perhaps a workaround can be outlined here to get the system up and running, at least.

Comment 3 ilmostro7 2016-02-15 20:30:03 UTC
Would it be advisable/prudent to change the lvm package to execute 
~~~
dracut -f -v --add-driver="dm-snapshot"
~~~
at the time of installation?

I'm also wondering if this issue stems from the default mirror segtype specified in /etc/lvm/lvm.conf

mirror_segtype_default = "raid1" [raid1/mirror]

During my hours of frustration in dealing with this issue I noticed that the dm-mirror module was available/compiled in to the kernel; whereas the dm-snapshot module was missing.  Maybe it's the Multipath solution outlined at https://access.redhat.com/solutions/2042843

Comment 4 Gordon Messmer 2016-02-16 19:36:34 UTC
As far as workarounds go, I'd suggest:

1: If possible, boot the original kernel from installation, and then rebuild the initrd for your current kernel.

2: Add the module to a dracut configuration file and rebuild the initrd before rebooting with a snapshot:
# echo add_drivers+=\"dm-snapshot\" >> /etc/dracut.conf.d/snapshot.conf
# dracut -f

3: Patch 90dm/module-setup.sh so that dm-snapshot is always loaded.  I'm not positive this is the correct fix for dracut, but I think it is:

# diff -u /usr/lib/dracut/modules.d/90dm/module-setup.sh.orig /usr/lib/dracut/modules.d/90dm/module-setup.sh
--- /usr/lib/dracut/modules.d/90dm/module-setup.sh.orig	2016-02-16 11:35:29.365014902 -0800
+++ /usr/lib/dracut/modules.d/90dm/module-setup.sh	2016-02-16 11:35:22.115207698 -0800
@@ -14,7 +14,7 @@
 installkernel() {
     instmods =drivers/md
     echo 'DM-SNAPSHOT' >&2
-    instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner
+    instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner --dm-snapshot
 }
 
 install() {

Comment 5 Mike McCune 2016-03-28 22:31:02 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions

Comment 6 Ryan Sawhill 2016-06-08 15:36:48 UTC
I'd just like to confirm this is definitely an issue if people are creating LVM snapshots in their root VG.

I just spun up 3 new VMs. In RHEL 7.0, 7.1, and 7.2 (even with latest dracut packages) I saw the following:

  - The initramfs created by anaconda includes dm-snapshot (as mentioned in this BZ)

  - Running dracut post-install when no LVM snapshots are present (whether manually or as triggered by `yum update kernel`) creates initramfs LACKING dm-snapshot

  - Running dracut post-install when an LVM snapshot is present (whether manually or as triggered by `yum update kernel`) creates initramfs that INCLUDES dm-snapshot

  - Adding "add_drivers+=dm-snapshot" to dracut config results in dracut always including dm-snapshot in new initramfs files

Until this is addressed by engineering, I'll echo the work-around that was already suggested. If you might be creating snapshots in your root VG in the future, do something like:

  echo add_drivers+=dm-snapshot >/etc/dracut.conf.d/dmsnapshot.conf

And then use dracut to rebuild any initramfs files you have which were created post-installation.

Comment 7 Ryan Sawhill 2016-06-08 17:03:38 UTC
Created KCS solution to cover this issue.

  Creating an LVM snapshot in RHEL7 can render the system unbootable
  https://access.redhat.com/solutions/2359851

Comment 9 Peter Kotvan 2016-07-22 08:09:25 UTC
Verified on RHEL-7.3-20160719.1 with anaconda-21.48.22.80-1.el7.

Comment 11 errata-xmlrpc 2016-11-04 08:01:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2530.html

Comment 12 Ryan Blakley 2017-05-16 22:10:21 UTC
This appears to not be fixed in the latest dracut-033-463.el7 package. If the kernel is updated, then an lv snapshot is created on the root lv, the server will not boot due to the dm-snapshot.ko not being included by default. Below are the steps I took to recreate the issue.

root@r73-gui-ga ~ # uname -a
Linux r73-gui-ga.example.com 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

root@r73-gui-ga ~ # rpm -qa|grep dracut
dracut-network-033-463.el7.x86_64
dracut-033-463.el7.x86_64
dracut-config-rescue-033-463.el7.x86_64

root@r73-gui-ga ~ # lsinitrd /boot/initramfs-3.10.0-514.el7.x86_64.img|grep dm-snapshot
-rw-r--r--   1 root     root        70809 Oct 19  2016 usr/lib/modules/3.10.0-514.el7.x86_64/kernel/drivers/md/dm-snapshot.ko

root@r73-gui-ga ~ # yum update kernel*
Loaded plugins: auto-update-debuginfo, langpacks, product-id, search-disabled-repos, subscription-manager
----------------cut---------------- 
Installed:
  kernel.x86_64 0:3.10.0-514.16.1.el7                                                                                 

Updated:
  kernel-doc.noarch 0:3.10.0-514.16.1.el7                      kernel-tools.x86_64 0:3.10.0-514.16.1.el7              
  kernel-tools-libs.x86_64 0:3.10.0-514.16.1.el7              

Complete!

root@r73-gui-ga ~ # lsinitrd /boot/initramfs-3.10.0-514.16.1.el7.x86_64.img|grep dm-snapshot
root@r73-gui-ga ~ #

Comment 13 Gordon Messmer 2017-05-16 23:21:47 UTC
I was wrong about a third option in comment #4, BTW.  I don't see any means of unconditionally loading a module, and I suppose we'd only want that module loaded when LVM is in use anyway.

It seems like the right thing to do would be to add a task to anaconda, so that if LVM is used for the filesystem, /etc/dracut.conf.d/snapshot.conf is created with the contents:
add_drivers+="dm-snapshot"

Comment 14 Harald Hoyer 2017-06-29 09:21:41 UTC
Sidenote: please add whitespace before and after the modules add_drivers+=" dm-snapshot "

Comment 15 ilmostro7 2017-07-06 09:05:11 UTC
whitespace around the modules (args/options), as it's appending to the existing list of drivers, correct?

Comment 16 Harald Hoyer 2017-07-07 11:11:11 UTC
(In reply to ilmostro7 from comment #15)
> whitespace around the modules (args/options), as it's appending to the
> existing list of drivers, correct?

yes

Comment 17 Harald Hoyer 2017-07-07 11:11:48 UTC
(In reply to Harald Hoyer from comment #16)
> (In reply to ilmostro7 from comment #15)
> > whitespace around the modules (args/options), as it's appending to the
> > existing list of drivers, correct?
> 
> yes

not shell command line options, though... only config files

Comment 18 xiafei2011 2017-07-28 07:54:48 UTC
Hi all,
   I first install the CentOS 7.3 with default disk allocation. The kernel version is 3.10.0-514.el7.x86_64.
   Then, I compile the kernel-4.9.40 and reboot. However, the system hangs in  "A start job is running for dev-mapper-cl\x2droot.device".

   Following the above comments, I modified the /etc/dracut.conf with the contents: 
     add_drivers+=" dm-snapshot " , and compile the kernel-4.9.40 again.
   Now, the dm-snapshot.ko has been included in the /boot/initramfs-4.9.40.img.

[root@localhost ~]# lsinitrd /boot/initramfs-4.9.40.img | grep dm-snapshot
-rw-r--r--   1 root     root      1093334 Jul 28 14:48 usr/lib/modules/4.9.40/kernel/drivers/md/dm-snapshot.ko

   I reboot again. However, the system hangs in "A start job is running for dev-mapper-cl\x2droot.device".
 
   The dracut version is as follows:

[root@localhost ~]# rpm -qa | grep dracut
dracut-config-rescue-033-463.el7.x86_64
dracut-network-033-463.el7.x86_64
dracut-033-463.el7.x86_64

   Is is necessary to upgrade the dracut to the latest version as the link https://bugs.centos.org/view.php?id=12426&nbn=10 suggested.

   Any suggestions to solve this problem? Thanks very much.

Comment 19 Gordon Messmer 2017-07-28 17:12:25 UTC
(In reply to xiafei2011 from comment #18)
> [root@localhost ~]# lsinitrd /boot/initramfs-4.9.40.img | grep dm-snapshot
> -rw-r--r--   1 root     root      1093334 Jul 28 14:48
> usr/lib/modules/4.9.40/kernel/drivers/md/dm-snapshot.ko

Your issue does not appear to be related to the lack of snapshot support, and this isn't an appropriate venue for technical support.  Please ask about this in the centos user email list: centos@centos.org

Comment 20 Gordon Messmer 2017-07-28 17:13:41 UTC
BTW: This bug does not appear to be resolved (as Ryan mentioned on 5/16), but I can't modify the status to reopen it.  Can someone else re-open the bug, or should I just file a new one on the same issue?

Comment 21 Bryn M. Reeves 2017-07-31 14:39:08 UTC
*** Bug 1272162 has been marked as a duplicate of this bug. ***

Comment 23 FredrikT 2018-02-01 19:22:11 UTC
Installed CentOS 7.4 and yum updated as of today.
Kernel: 3.10.0-693.17.1.el7.x86_64
Dracut: 033-502.el7_4.1

Created a snapshot of root vol and then installed openstack via packstack.
Rebooted and was dropped into dracut shell.

Could I work myself out of this?

Comment 24 Gordon Messmer 2018-02-01 21:53:35 UTC
(In reply to FredrikT from comment #23)

I believe you can boot the original kernel.  The one that was installed by Anaconda, if it is still available, should have the dm-snapshot module and therefore should boot.  Once the system boots, run "dracut -f <kernel version>" to rebuild the newer initrd, then reboot to the new kernel.

Comment 25 FredrikT 2018-02-01 23:33:21 UTC
thanks for the reply Gordon!
Unfortunately it didn't work. Booting the previous gave me the same result.
Just reinstalled the system. Gonna try the dracut and give it a go again. Don't have snapshots (using local storage) so I want to do multiple Openstack/packstack deploys on baremetal in a quick fashion trying to use lvm snapshoting. Maybe not the best of ideas.

Comment 26 FredrikT 2018-02-01 23:33:54 UTC
thanks for the reply Gordon!
Unfortunately it didn't work. Booting the previous gave me the same result.
Just reinstalled the system. Gonna try the dracut and give it a go again. Don't have snapshots (using local storage) so I want to do multiple Openstack/packstack deploys on baremetal in a quick fashion trying to use lvm snapshoting. Maybe not the best of ideas.

Comment 27 Gordon Messmer 2018-02-02 00:37:07 UTC
Not just any previous kernel will work.  As far as I know, only the original kernel that was installed by Anaconda will have the dm-snapshot module included.

Comment 28 Gordon Messmer 2018-02-02 06:19:24 UTC
Please re-open this bug.  It is clearly not fixed.  I've re-verified it on a new installation:

# rpm -q dracut kernel
dracut-033-502.el7_4.1.x86_64
kernel-3.10.0-693.11.1.el7.x86_64
kernel-3.10.0-693.11.6.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
# lsinitrd /boot/initramfs-3.10.0-693.17.1.el7.x86_64.img | grep dm.snap

dracut was up-to-date when the newest kernel was installed.  The initramfs it created does not include the dm-snapshot module.  If I snapshot the root LV, the system will not boot.

Comment 29 FredrikT 2018-02-02 08:04:17 UTC
B.t.w Gordon,

The workaround/patch you provided moving to this:
instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner --dm-snapshot

Don't you mean:
instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner dm-snapshot

Trying it as we speak.

Comment 30 FredrikT 2018-02-02 09:46:47 UTC
I simply cannot get the module into the ramfs using your suggestion.

The one way that worked was:
dracut -f -v --add-drivers "dm-snapshot"

..as well as adding the above to /etc/dracut.confg

Not sure why it does not get picked up by your suggestion..
But I guess dracut.conf is a better way.

Server boots ok now. thanks!


Note You need to log in before you can comment on or make changes to this bug.