Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1125849

Summary: dmeventd[10531]: segfault at 50 ip 00007f1f2e1660e7 sp 00007f1f2f0fc398 error 4 in libc-2.12.so[7f1f2e0d8000+18b000] - dev is NULL
Product: Red Hat Enterprise Linux 6 Reporter: Marian Csontos <mcsontos>
Component: lvm2Assignee: Petr Rockai <prockai>
lvm2 sub component: dmeventd (RHEL6) QA Contact: Cluster QE <mspqa-list>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: agk, cmarthal, heinzm, jbrassow, msnitzer, nperic, prajnoha, prockai, thornber, tlavigne, zkabelac
Version: 6.6Keywords: Regression
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Not applicable, the bug arose during development and was fixed before a release.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-14 08:25:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
bt full none

Description Marian Csontos 2014-08-01 09:02:30 UTC
Description of problem:
Segfaulting dmeventd.

Version-Release number of selected component (if applicable):
lvm2-2.02.108-1.el6.x86_64
kernel-2.6.32-494.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
sda1 is a leg in RAID LV. Though likely irrelevant the raid_fault_policy is warn.
Executing these two commands:
    echo offline > /sys/block/sda/device/state
    pvscan --cache /dev/sda1
Results in segfault in dmeventd.

Messages:

Jul 31 17:41:44 barb-02c1-node01 kernel: sd 2:0:0:0: rejecting I/O to offline device
Jul 31 17:41:44 barb-02c1-node01 kernel: end_request: I/O error, dev sda, sector 0
Jul 31 17:41:44 barb-02c1-node01 kernel: sd 2:0:0:0: rejecting I/O to offline device
Jul 31 17:41:44 barb-02c1-node01 kernel: end_request: I/O error, dev sda, sector 0
Jul 31 17:41:44 barb-02c1-node01 kernel: sd 2:0:0:0: rejecting I/O to offline device
Jul 31 17:41:44 barb-02c1-node01 kernel: md: super_written gets error=-5, uptodate=0
Jul 31 17:41:44 barb-02c1-node01 kernel: md/raid10:mdX: Disk failure on dm-9, disabling device.
Jul 31 17:41:44 barb-02c1-node01 kernel: md/raid10:mdX: Operation continuing on 5 devices.
Jul 31 17:41:44 barb-02c1-node01 lvm[10528]: Device #4 of raid10 array, black_bird-synced_random_raid10_3legs_1, has failed.
Jul 31 17:41:44 barb-02c1-node01 kernel: sd 2:0:0:0: rejecting I/O to offline device
Jul 31 17:41:44 barb-02c1-node01 kernel: md: super_written gets error=-5, uptodate=0
Jul 31 17:41:44 barb-02c1-node01 lvm[10528]: PV mkHEgE-rMpc-w6eY-3y0r-Qce7-Cy1G-COvwAi not recognised. Is the device missing?
Jul 31 17:41:44 barb-02c1-node01 kernel: dmeventd[10531]: segfault at 50 ip 00007f1f2e1660e7 sp 00007f1f2f0fc398 error 4 in libc-2.12.so[7f1f2e0d8000+18b000]

Comment 2 Marian Csontos 2014-08-01 12:38:02 UTC
Created attachment 923286 [details]
bt full

Comment 3 Marian Csontos 2014-08-01 13:13:39 UTC
Nened just verified this happens only when running the two commands in close sequence like this:

    echo offline > /sys/block/sda/device/state && pvscan --cache /dev/sda1

May not be very likely to happen... Could it be inherent to ANY parallel scans?

Comment 4 Petr Rockai 2014-08-04 12:38:30 UTC
No, other parallel scans are unaffected -- it is only lvscan that is susceptible. This is because lvscan relies on device information from lvmetad. A lvmetad debug log would be very useful here (with -l wire) -- but as far as I can tell, the problem is not parallel scans at all -- dmeventd should crash the same way when any LV that is already partial is rescanned. I'll write a testcase for that and see if that's the case.

The problem is that lvscan --cache uses device information from lvmetad, which may be already partial if pvscan --cache finishes before lvscan --cache can read the device info. In this case, lvscan --cache should ignore devices that are already known to be missing, instead of trying to scan them again (and encountering a null pointer).

Comment 6 Nenad Peric 2014-08-08 13:50:34 UTC

[root@tardis-01 ~]# echo offline > /sys/block/sdb/device/state && pvscan --cache /dev/sdb1
  /dev/sdb1: open failed: No such device or address
  No PV label found on /dev/sdb1.
[root@tardis-01 ~]# lvs
  PV lkU8Mh-SKZb-zhGM-pf7T-vKR6-Wnn1-JsHjy1 not recognised. Is the device missing?
  PV 8ozawT-pdg3-6sPH-e9xv-UXyc-5kz4-hqPOa0 not recognised. Is the device missing?
  PV lkU8Mh-SKZb-zhGM-pf7T-vKR6-Wnn1-JsHjy1 not recognised. Is the device missing?
  PV 8ozawT-pdg3-6sPH-e9xv-UXyc-5kz4-hqPOa0 not recognised. Is the device missing?
  LV      VG          Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  d_lv    vg          -wi-a---p-   2.00g                                                    
  raid1   vg          rwi---r-p-   1.00g                                                    
  lv_home vg_tardis01 -wi-ao---- 224.88g                                                    
  lv_root vg_tardis01 -wi-ao----  50.00g                                                    
  lv_swap vg_tardis01 -wi-ao----   4.00g            

[root@tardis-01 ~]# ps -ef | grep dmevent
root      3824     1  0 15:45 ?        00:00:00 /sbin/dmeventd
root      3907  3655  0 15:50 pts/0    00:00:00 grep dmevent




Marking VERIFIED with:
2.6.32-495.el6.x86_64

lvm2-2.02.109-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014
lvm2-libs-2.02.109-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014
lvm2-cluster-2.02.109-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014
udev-147-2.57.el6    BUILT: Thu Jul 24 15:48:47 CEST 2014
device-mapper-1.02.88-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014
device-mapper-libs-1.02.88-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014
device-mapper-event-1.02.88-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014
device-mapper-event-libs-1.02.88-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014
device-mapper-persistent-data-0.3.2-1.el6    BUILT: Fri Apr  4 15:43:06 CEST 2014
cmirror-2.02.109-1.el6    BUILT: Tue Aug  5 17:36:23 CEST 2014

Comment 7 errata-xmlrpc 2014-10-14 08:25:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1387.html