Bug 1287940
| Summary: | Creating an LVM snapshot can render the system unbootable | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Gordon Messmer <gordon.messmer> |
| Component: | dracut | Assignee: | dracut-maint |
| Status: | CLOSED ERRATA | QA Contact: | Release Test Team <release-test-team-automation> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.1 | CC: | dracut-maint-list, ernest.beinrohr, fredrik_tarnell, harald, ilmostro7, mbanas, mtenheuv, pasteur, pkotvan, rblakley, redhat, rsawhill, xiafei2011 |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-04 08:01:48 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1289485, 1313485, 1469559 | ||
|
Description
Gordon Messmer
2015-12-03 04:34:57 UTC
Thanks for your finding! Still occurring on 7.2. From the looks of it, this has been happening for many years now on multiple distributions. Frustrating! Perhaps a workaround can be outlined here to get the system up and running, at least. Would it be advisable/prudent to change the lvm package to execute ~~~ dracut -f -v --add-driver="dm-snapshot" ~~~ at the time of installation? I'm also wondering if this issue stems from the default mirror segtype specified in /etc/lvm/lvm.conf mirror_segtype_default = "raid1" [raid1/mirror] During my hours of frustration in dealing with this issue I noticed that the dm-mirror module was available/compiled in to the kernel; whereas the dm-snapshot module was missing. Maybe it's the Multipath solution outlined at https://access.redhat.com/solutions/2042843 As far as workarounds go, I'd suggest:
1: If possible, boot the original kernel from installation, and then rebuild the initrd for your current kernel.
2: Add the module to a dracut configuration file and rebuild the initrd before rebooting with a snapshot:
# echo add_drivers+=\"dm-snapshot\" >> /etc/dracut.conf.d/snapshot.conf
# dracut -f
3: Patch 90dm/module-setup.sh so that dm-snapshot is always loaded. I'm not positive this is the correct fix for dracut, but I think it is:
# diff -u /usr/lib/dracut/modules.d/90dm/module-setup.sh.orig /usr/lib/dracut/modules.d/90dm/module-setup.sh
--- /usr/lib/dracut/modules.d/90dm/module-setup.sh.orig 2016-02-16 11:35:29.365014902 -0800
+++ /usr/lib/dracut/modules.d/90dm/module-setup.sh 2016-02-16 11:35:22.115207698 -0800
@@ -14,7 +14,7 @@
installkernel() {
instmods =drivers/md
echo 'DM-SNAPSHOT' >&2
- instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner
+ instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner --dm-snapshot
}
install() {
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions I'd just like to confirm this is definitely an issue if people are creating LVM snapshots in their root VG. I just spun up 3 new VMs. In RHEL 7.0, 7.1, and 7.2 (even with latest dracut packages) I saw the following: - The initramfs created by anaconda includes dm-snapshot (as mentioned in this BZ) - Running dracut post-install when no LVM snapshots are present (whether manually or as triggered by `yum update kernel`) creates initramfs LACKING dm-snapshot - Running dracut post-install when an LVM snapshot is present (whether manually or as triggered by `yum update kernel`) creates initramfs that INCLUDES dm-snapshot - Adding "add_drivers+=dm-snapshot" to dracut config results in dracut always including dm-snapshot in new initramfs files Until this is addressed by engineering, I'll echo the work-around that was already suggested. If you might be creating snapshots in your root VG in the future, do something like: echo add_drivers+=dm-snapshot >/etc/dracut.conf.d/dmsnapshot.conf And then use dracut to rebuild any initramfs files you have which were created post-installation. Created KCS solution to cover this issue. Creating an LVM snapshot in RHEL7 can render the system unbootable https://access.redhat.com/solutions/2359851 Verified on RHEL-7.3-20160719.1 with anaconda-21.48.22.80-1.el7. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2530.html This appears to not be fixed in the latest dracut-033-463.el7 package. If the kernel is updated, then an lv snapshot is created on the root lv, the server will not boot due to the dm-snapshot.ko not being included by default. Below are the steps I took to recreate the issue. root@r73-gui-ga ~ # uname -a Linux r73-gui-ga.example.com 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux root@r73-gui-ga ~ # rpm -qa|grep dracut dracut-network-033-463.el7.x86_64 dracut-033-463.el7.x86_64 dracut-config-rescue-033-463.el7.x86_64 root@r73-gui-ga ~ # lsinitrd /boot/initramfs-3.10.0-514.el7.x86_64.img|grep dm-snapshot -rw-r--r-- 1 root root 70809 Oct 19 2016 usr/lib/modules/3.10.0-514.el7.x86_64/kernel/drivers/md/dm-snapshot.ko root@r73-gui-ga ~ # yum update kernel* Loaded plugins: auto-update-debuginfo, langpacks, product-id, search-disabled-repos, subscription-manager ----------------cut---------------- Installed: kernel.x86_64 0:3.10.0-514.16.1.el7 Updated: kernel-doc.noarch 0:3.10.0-514.16.1.el7 kernel-tools.x86_64 0:3.10.0-514.16.1.el7 kernel-tools-libs.x86_64 0:3.10.0-514.16.1.el7 Complete! root@r73-gui-ga ~ # lsinitrd /boot/initramfs-3.10.0-514.16.1.el7.x86_64.img|grep dm-snapshot root@r73-gui-ga ~ # I was wrong about a third option in comment #4, BTW. I don't see any means of unconditionally loading a module, and I suppose we'd only want that module loaded when LVM is in use anyway. It seems like the right thing to do would be to add a task to anaconda, so that if LVM is used for the filesystem, /etc/dracut.conf.d/snapshot.conf is created with the contents: add_drivers+="dm-snapshot" Sidenote: please add whitespace before and after the modules add_drivers+=" dm-snapshot " whitespace around the modules (args/options), as it's appending to the existing list of drivers, correct? (In reply to ilmostro7 from comment #15) > whitespace around the modules (args/options), as it's appending to the > existing list of drivers, correct? yes (In reply to Harald Hoyer from comment #16) > (In reply to ilmostro7 from comment #15) > > whitespace around the modules (args/options), as it's appending to the > > existing list of drivers, correct? > > yes not shell command line options, though... only config files Hi all,
I first install the CentOS 7.3 with default disk allocation. The kernel version is 3.10.0-514.el7.x86_64.
Then, I compile the kernel-4.9.40 and reboot. However, the system hangs in "A start job is running for dev-mapper-cl\x2droot.device".
Following the above comments, I modified the /etc/dracut.conf with the contents:
add_drivers+=" dm-snapshot " , and compile the kernel-4.9.40 again.
Now, the dm-snapshot.ko has been included in the /boot/initramfs-4.9.40.img.
[root@localhost ~]# lsinitrd /boot/initramfs-4.9.40.img | grep dm-snapshot
-rw-r--r-- 1 root root 1093334 Jul 28 14:48 usr/lib/modules/4.9.40/kernel/drivers/md/dm-snapshot.ko
I reboot again. However, the system hangs in "A start job is running for dev-mapper-cl\x2droot.device".
The dracut version is as follows：
[root@localhost ~]# rpm -qa | grep dracut
dracut-config-rescue-033-463.el7.x86_64
dracut-network-033-463.el7.x86_64
dracut-033-463.el7.x86_64
Is is necessary to upgrade the dracut to the latest version as the link https://bugs.centos.org/view.php?id=12426&nbn=10 suggested.
Any suggestions to solve this problem? Thanks very much.
(In reply to xiafei2011 from comment #18) > [root@localhost ~]# lsinitrd /boot/initramfs-4.9.40.img | grep dm-snapshot > -rw-r--r-- 1 root root 1093334 Jul 28 14:48 > usr/lib/modules/4.9.40/kernel/drivers/md/dm-snapshot.ko Your issue does not appear to be related to the lack of snapshot support, and this isn't an appropriate venue for technical support. Please ask about this in the centos user email list: centos BTW: This bug does not appear to be resolved (as Ryan mentioned on 5/16), but I can't modify the status to reopen it. Can someone else re-open the bug, or should I just file a new one on the same issue? *** Bug 1272162 has been marked as a duplicate of this bug. *** Installed CentOS 7.4 and yum updated as of today. Kernel: 3.10.0-693.17.1.el7.x86_64 Dracut: 033-502.el7_4.1 Created a snapshot of root vol and then installed openstack via packstack. Rebooted and was dropped into dracut shell. Could I work myself out of this? (In reply to FredrikT from comment #23) I believe you can boot the original kernel. The one that was installed by Anaconda, if it is still available, should have the dm-snapshot module and therefore should boot. Once the system boots, run "dracut -f <kernel version>" to rebuild the newer initrd, then reboot to the new kernel. thanks for the reply Gordon! Unfortunately it didn't work. Booting the previous gave me the same result. Just reinstalled the system. Gonna try the dracut and give it a go again. Don't have snapshots (using local storage) so I want to do multiple Openstack/packstack deploys on baremetal in a quick fashion trying to use lvm snapshoting. Maybe not the best of ideas. thanks for the reply Gordon! Unfortunately it didn't work. Booting the previous gave me the same result. Just reinstalled the system. Gonna try the dracut and give it a go again. Don't have snapshots (using local storage) so I want to do multiple Openstack/packstack deploys on baremetal in a quick fashion trying to use lvm snapshoting. Maybe not the best of ideas. Not just any previous kernel will work. As far as I know, only the original kernel that was installed by Anaconda will have the dm-snapshot module included. Please re-open this bug. It is clearly not fixed. I've re-verified it on a new installation: # rpm -q dracut kernel dracut-033-502.el7_4.1.x86_64 kernel-3.10.0-693.11.1.el7.x86_64 kernel-3.10.0-693.11.6.el7.x86_64 kernel-3.10.0-693.17.1.el7.x86_64 # lsinitrd /boot/initramfs-3.10.0-693.17.1.el7.x86_64.img | grep dm.snap dracut was up-to-date when the newest kernel was installed. The initramfs it created does not include the dm-snapshot module. If I snapshot the root LV, the system will not boot. B.t.w Gordon, The workaround/patch you provided moving to this: instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner --dm-snapshot Don't you mean: instmods dm_mod dm-cache dm-cache-mq dm-cache-cleaner dm-snapshot Trying it as we speak. I simply cannot get the module into the ramfs using your suggestion. The one way that worked was: dracut -f -v --add-drivers "dm-snapshot" ..as well as adding the above to /etc/dracut.confg Not sure why it does not get picked up by your suggestion.. But I guess dracut.conf is a better way. Server boots ok now. thanks! |