RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 651402 - dracut is accessing DM devices it shouldn't touch
Summary: dracut is accessing DM devices it shouldn't touch
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: dracut
Version: 6.1
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: rc
: ---
Assignee: Harald Hoyer
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On: 650959
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-11-09 14:12 UTC by RHEL Program Management
Modified: 2012-02-09 16:20 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Prior to this update, the udev rules used by dracut may have caused the merged logical volume management (LVM) snapshots to incorrectly appear in the /dev directory. Consequent to this, an attempt to remove these snapshots failed with an error. With this update, dracut's internal udev rules have been updated to check the value of "DM_UDEV_DISABLE_OTHER_RULES_FLAG", and dracut now works as expected.
Clone Of:
Environment:
Last Closed: 2010-11-11 12:17:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0877 0 normal SHIPPED_LIVE dracut bug fix update 2010-11-11 12:16:56 UTC

Description RHEL Program Management 2010-11-09 14:12:59 UTC
This bug has been copied from bug #650959 and has been proposed
to be backported to 6.0 z-stream (EUS).

Comment 6 Jaromir Hradilek 2010-11-10 00:18:23 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Prior to this update, the udev rules used by dracut may have caused the merged logical volume management (LVM) snapshots to incorrectly appear in the /dev directory. Consequent to this, an attempt to remove these snapshots failed with an error. With this update, dracut's internal udev rules have been updated to check the value of "DM_UDEV_DISABLE_OTHER_RULES_FLAG", and dracut now works as expected.

Comment 7 Alasdair Kergon 2010-11-10 00:23:46 UTC
Is that too much detail for a release note: do people really need exposure to these internals?

Comment 8 Mike Snitzer 2010-11-10 01:06:11 UTC
(In reply to comment #7)
> Is that too much detail for a release note: do people really need exposure to
> these internals?

Not only that but the details are wrong!

The merged (or merging) snapshots only ever appeared in /dev/mapper/ as internal DM devices.  And these udev rules never prevented the merged snapshots from being removed.

Reality is the new dracut rules simply didn't know to ignore internal DM devices -- as such those rules tried to access internal devices and got errors as a result.

Comment 9 Alexander Todorov 2010-11-10 11:09:27 UTC
In dracut-004-33.el6_0.noarch.rpm the patch from https://bugzilla.redhat.com/show_bug.cgi?id=650959#c1 is applied.

With updated dracut and updated initramfs.img I did the LVM snapshot/merge sequence from https://bugzilla.redhat.com/show_bug.cgi?id=650959#c7.

After reboot there was no I/O error in dmesg (such as in https://bugzilla.redhat.com/show_bug.cgi?id=645488#c10).

Moving to VERIFIED.

Comment 10 Alexander Todorov 2010-11-10 12:35:20 UTC
Re-tested this with updated dracut and lvm/device-mapper packages from http://people.redhat.com/jeder/bz645488/.

After the merge & reboot there are no errors in dmesg which were caused by dracut. 

After the merge is complete lvs -a shows no snapshot volume. The device node files for the snapshot volume under /dev/mapper are removed as well. 

This only verifies that previous testing was indeed correct.

Comment 12 errata-xmlrpc 2010-11-11 12:17:00 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0877.html

Comment 13 John Ruemker 2010-12-06 19:39:08 UTC
I can still reproduce this issue on the version released in the errata (dracut-004-33.el6_0.noarch).  I did remake the initrd with "dracut" after updating the dracut package.  

To reproduce, I created a snapshot:

  # lvcreate -l 511 -n lv_root-snap -s vg_jrummy/lv_root

Made a few changes to root fs, then merged the snapshot:

  # lvconvert --merge vg_jrummy/lv_root-snap
  Can't merge over open origin volume
  Merging of snapshot lv_root-snap will start next activation.
  # reboot

On reboot, I still get I/O errors on the snapshot:

dracut: Scanning devices vda2 vdb  for LVM logical volumes vg_jrummy/lv_root vg_jrummy/lv_swap 
dracut: inactive Original '/dev/vg_jrummy/lv_root' [13.54 GiB] inherit
dracut: inactive '/dev/vg_jrummy/lv_swap' [992.00 MiB] inherit
EXT4-fs (dm-2): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/mapper/vg_jrummy-lv_root
dracut: Switching root
		Welcome to Red Hat Enterprise Linux Server
Starting udev: udev: starting version 147
piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
Intel ICH 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, high) -> IRQ 11
intel8x0_measure_ac97_clock: measured 51054 usecs (4396 samples)
intel8x0: measured clock 86104 rejected
intel8x0_measure_ac97_clock: measured 50948 usecs (4404 samples)
intel8x0: measured clock 86441 rejected
intel8x0_measure_ac97_clock: measured 50967 usecs (4393 samples)
intel8x0: measured clock 86193 rejected
intel8x0: clocking to 48000
[  OK  ]
Setting hostname jrummy6-64.usersys.redhat.com:  [  OK  ]
Buffer I/O error on device dm-3, logical block 3549183
Buffer I/O error on device dm-3, logical block 3549183
ERROR: asr: reading /dev/dm-3[Input/output error]
Buffer I/O error on device dm-3, logical block 3549183
Buffer I/O error on device dm-3, logical block 3549183
ERROR: ddf1: reading /dev/dm-3[Input/output error]
Buffer I/O error on device dm-3, logical block 3549151
Buffer I/O error on device dm-3, logical block 3549151
ERROR: ddf1: reading /dev/dm-3[Input/output error]
Buffer I/O error on device dm-3, logical block 1
Buffer I/O error on device dm-3, logical block 1
ERROR: hpt37x: reading /dev/dm-3[Input/output error]
Buffer I/O error on device dm-3, logical block 3549182
Buffer I/O error on device dm-3, logical block 3549182
ERROR: hpt45x: reading /dev/dm-3[Input/output error]
ERROR: isw: reading /dev/dm-3[Input/output error]
ERROR: jmicron: reading /dev/dm-3[Input/output error]
ERROR: lsi: reading /dev/dm-3[Input/output error]
ERROR: nvidia: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: pdc: reading /dev/dm-3[Input/output error]
ERROR: sil: reading /dev/dm-3[Input/output error]
ERROR: via: reading /dev/dm-3[Input/output error]
Setting up Logical Volume Management:   2 logical volume(s) in volume group "vg_jrummy" now active
[  OK  ]

Comment 14 John Ruemker 2010-12-06 19:57:47 UTC
And just to be sure, from the unpacked initramfs:

# pwd
/root/initrd
# grep DM_UDEV_DISABLE_OTHER_RULES_FLAG etc/udev/rules.d/61-dmraid-imsm.rules
ENV{DM_UDEV_DISABLE_OTHER_RULES_FLAG}=="1", GOTO="dm_end"
# grep -A 1 -B 1 DM_UDEV_DISABLE_OTHER_RULES_FLAG etc/udev/rules.d/59-persistent-storage.rules
ENV{DM_NAME}!="?*", GOTO="ps_end"
ENV{DM_UDEV_DISABLE_OTHER_RULES_FLAG}=="1", GOTO="ps_end"
ENV{DM_UUID}=="CRYPT-TEMP-?*", GOTO="ps_end"
# grep -A 1 -B 1 DM_UDEV_DISABLE_OTHER_RULES_FLAG etc/udev/rules.d/61-persistent-storage.rules 
ACTION=="change", KERNEL=="dm-[0-9]*", ENV{DM_UDEV_DISABLE_OTHER_RULES_FLAG}!="1", GOTO="do_pss"
KERNEL=="cciss[0-9]*", GOTO="do_pss"

Comment 15 Mike Snitzer 2010-12-06 20:43:01 UTC
(In reply to comment #13)
> I can still reproduce this issue on the version released in the errata
> (dracut-004-33.el6_0.noarch).  I did remake the initrd with "dracut" after
> updating the dracut package.  

I assume you are using an updated lvm2 package.

Seems all the errors are dmraid related though... prajnoha and/or heinzm may be able to offer more insight as to why etc/udev/rules.d/61-dmraid-imsm.rules isn't ignoring the merging snapshot (/dev/dm-3).

Comment 16 John Ruemker 2010-12-06 21:20:07 UTC
I am not using an updated lvm2 package, because one is not available to my customer (it hasn't been released yet).  They pulled down this dracut erratum expecting the I/O errors during boot to go away, but understanding that the lvm issue causing the snapshot to not be cleaned up would still be present.

My understanding was that this specific bug was related to dracut only, and how it is causing I/O errors to be printed due to it accessing these devices when it shouldn't.  And https://bugzilla.redhat.com/show_bug.cgi?id=645488 is focused on correcting the fact that lvm does not clean up the merged snapshot once it is complete.  But I was under the impression that they were separate, and this one did not depend on that one to be resolved.  Is this not accurate?

Comment 17 Mike Snitzer 2010-12-06 21:35:00 UTC
(In reply to comment #16)
> I am not using an updated lvm2 package, because one is not available to my
> customer (it hasn't been released yet).  They pulled down this dracut erratum
> expecting the I/O errors during boot to go away, but understanding that the lvm
> issue causing the snapshot to not be cleaned up would still be present.
> 
> My understanding was that this specific bug was related to dracut only, and how
> it is causing I/O errors to be printed due to it accessing these devices when
> it shouldn't.  And https://bugzilla.redhat.com/show_bug.cgi?id=645488 is
> focused on correcting the fact that lvm does not clean up the merged snapshot
> once it is complete.  But I was under the impression that they were separate,
> and this one did not depend on that one to be resolved.  Is this not accurate?

That is accurate.  In the past lvm2 (e.g. via 'lvs') would also access the merging snapshot device; resulting in IO errors, e.g.:
https://bugzilla.redhat.com/show_bug.cgi?id=645488#c5

... but that isn't what is happening here.

Comment 18 Harald Hoyer 2010-12-07 07:53:49 UTC
(In reply to comment #13)
> I can still reproduce this issue on the version released in the errata
> (dracut-004-33.el6_0.noarch).  I did remake the initrd with "dracut" after
> updating the dracut package.  
> 
> To reproduce, I created a snapshot:
> 
>   # lvcreate -l 511 -n lv_root-snap -s vg_jrummy/lv_root
> 
> Made a few changes to root fs, then merged the snapshot:
> 
>   # lvconvert --merge vg_jrummy/lv_root-snap
>   Can't merge over open origin volume
>   Merging of snapshot lv_root-snap will start next activation.
>   # reboot
> 
> On reboot, I still get I/O errors on the snapshot:
> 
> dracut: Scanning devices vda2 vdb  for LVM logical volumes vg_jrummy/lv_root
> vg_jrummy/lv_swap 
> dracut: inactive Original '/dev/vg_jrummy/lv_root' [13.54 GiB] inherit
> dracut: inactive '/dev/vg_jrummy/lv_swap' [992.00 MiB] inherit
> EXT4-fs (dm-2): mounted filesystem with ordered data mode
> dracut: Mounted root filesystem /dev/mapper/vg_jrummy-lv_root
> dracut: Switching root


No more dracut after this point!!!!!!!!!


>   Welcome to Red Hat Enterprise Linux Server
> Starting udev: udev: starting version 147
> piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
> ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
> Intel ICH 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, high) -> IRQ
> 11
> intel8x0_measure_ac97_clock: measured 51054 usecs (4396 samples)
> intel8x0: measured clock 86104 rejected
> intel8x0_measure_ac97_clock: measured 50948 usecs (4404 samples)
> intel8x0: measured clock 86441 rejected
> intel8x0_measure_ac97_clock: measured 50967 usecs (4393 samples)
> intel8x0: measured clock 86193 rejected
> intel8x0: clocking to 48000
> [  OK  ]
> Setting hostname jrummy6-64.usersys.redhat.com:  [  OK  ]
> Buffer I/O error on device dm-3, logical block 3549183
> Buffer I/O error on device dm-3, logical block 3549183
> ERROR: asr: reading /dev/dm-3[Input/output error]
> Buffer I/O error on device dm-3, logical block 3549183
> Buffer I/O error on device dm-3, logical block 3549183
> ERROR: ddf1: reading /dev/dm-3[Input/output error]
> Buffer I/O error on device dm-3, logical block 3549151
> Buffer I/O error on device dm-3, logical block 3549151
> ERROR: ddf1: reading /dev/dm-3[Input/output error]
> Buffer I/O error on device dm-3, logical block 1
> Buffer I/O error on device dm-3, logical block 1
> ERROR: hpt37x: reading /dev/dm-3[Input/output error]
> Buffer I/O error on device dm-3, logical block 3549182
> Buffer I/O error on device dm-3, logical block 3549182
> ERROR: hpt45x: reading /dev/dm-3[Input/output error]
> ERROR: isw: reading /dev/dm-3[Input/output error]
> ERROR: jmicron: reading /dev/dm-3[Input/output error]
> ERROR: lsi: reading /dev/dm-3[Input/output error]
> ERROR: nvidia: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: pdc: reading /dev/dm-3[Input/output error]
> ERROR: sil: reading /dev/dm-3[Input/output error]
> ERROR: via: reading /dev/dm-3[Input/output error]
> Setting up Logical Volume Management:   2 logical volume(s) in volume group
> "vg_jrummy" now active
> [  OK  ]

Comment 19 Heinz Mauelshagen 2010-12-07 10:43:43 UTC
In case dmraid is started, it tries discovering ATARAID/DDF1 metadata signatures on any existing block devices, thus leading to io errors on any bogus ones (unclean snapshot merge device in case of this bz).

Comment 20 John Ruemker 2010-12-07 14:31:34 UTC
Yeah, removing dmraid gets rid of these.  Thanks everyone.  

-John

Comment 21 Dusty Mabe 2012-02-09 16:20:26 UTC
Hi.. I am seeing the I/O errors mentioned in comment 13 in CentOS 6.2 when merging an lvm snapshot on reboot. If I disable dmraid (by providing nodmraid on the kernel command line) then I don't see the errors. Is it expected that this should happen or should dmraid somehow detect and skip over these devices? 

- Dusty


Note You need to log in before you can comment on or make changes to this bug.