RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1915580 - dmeventd segfault during raid integrity failure testing
Summary: dmeventd segfault during raid integrity failure testing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-13 00:27 UTC by Corey Marthaler
Modified: 2021-09-07 11:51 UTC (History)
9 users (show)

Fixed In Version: lvm2-2.03.11-2.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:02:04 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Corey Marthaler 2021-01-13 00:27:32 UTC 
Description of problem:

kernel-4.18.0-271.el8    BUILT: Fri Jan  8 03:32:43 CST 2021
lvm2-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-libs-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-dbusd-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:35:05 CST 2020


===============================================================================
Iteration 1.1 started at Tue 12 Jan 2021 05:30:16 PM CST
===============================================================================
Scenario kill_random_synced_raid1_3legs: Kill random leg of synced 3 leg raid1 volume(s)
********* RAID hash info for this scenario *********
* names:              synced_random_raid1_3legs_1
* sync:               1
* type:               raid1
* -m |-i value:       3
* leg devices:        /dev/sdb1 /dev/sde1 /dev/sdh1 /dev/sdd1
* spanned legs:       0
* manual repair:      0
* no MDA devices:     
* failpv(s):          /dev/sde1
* failnode(s):        host-093
* integrity stack:    1
 (Due to integrity stack, be mindful of false failures that are reliant on message checks
  that could be lost due to rate-limiting of corruption and other messages)
* raid fault policy:  allocate
******************************************************

Creating raids(s) on host-093...
host-093: lvcreate --yes  --type raid1 -m 3 -n synced_random_raid1_3legs_1 -L 500M black_bird /dev/sdb1:0-2400 /dev/sde1:0-2400 /dev/sdh1:0-2400 /dev/sdd1:0-2400

Current mirror/raid device structure(s):
  LV                                     Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1            rwi-a-r--- 500.00m 6.26     synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0] Iwi-aor--- 500.00m          /dev/sdb1(1)
   [synced_random_raid1_3legs_1_rimage_1] Iwi-aor--- 500.00m          /dev/sde1(1)
   [synced_random_raid1_3legs_1_rimage_2] Iwi-aor--- 500.00m          /dev/sdh1(1)
   [synced_random_raid1_3legs_1_rimage_3] Iwi-aor--- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rmeta_0]  ewi-aor---   4.00m          /dev/sdb1(0)
   [synced_random_raid1_3legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sde1(0)
   [synced_random_raid1_3legs_1_rmeta_2]  ewi-aor---   4.00m          /dev/sdh1(0)
   [synced_random_raid1_3legs_1_rmeta_3]  ewi-aor---   4.00m          /dev/sdd1(0)


Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Convert mirror/raid volume(s) to utilize integrity target volume(s) on host-093...

lvconvert --yes --raidintegrity y --raidintegritymode bitmap black_bird/synced_random_raid1_3legs_1
Creating xfs on top of mirror(s) on host-093...
Mounting mirrored xfs filesystems on host-093...

Current mirror/raid device structure(s):
  LV                                           Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                  rwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdb1(126)
   [synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdb1(1)
   [synced_random_raid1_3legs_1_rimage_1]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao----  12.00m          /dev/sde1(126)
   [synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao---- 500.00m          /dev/sde1(1)
   [synced_random_raid1_3legs_1_rimage_2]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdh1(126)
   [synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdh1(1)
   [synced_random_raid1_3legs_1_rimage_3]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao----  12.00m          /dev/sdd1(126)
   [synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rmeta_0]        ewi-aor---   4.00m          /dev/sdb1(0)
   [synced_random_raid1_3legs_1_rmeta_1]        ewi-aor---   4.00m          /dev/sde1(0)
   [synced_random_raid1_3legs_1_rmeta_2]        ewi-aor---   4.00m          /dev/sdh1(0)
   [synced_random_raid1_3legs_1_rmeta_3]        ewi-aor---   4.00m          /dev/sdd1(0)

PV=/dev/sde1
        synced_random_raid1_3legs_1_rimage_1_imeta: 1.0
        synced_random_raid1_3legs_1_rimage_1_iorig: 1.0
        synced_random_raid1_3legs_1_rmeta_1: 1.0

Writing verification files (checkit) to mirror(s) on...
        ---- host-093 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 

Verifying files (checkit) on mirror(s) on...
        ---- host-093 ----

**** Raid Integrity Corruption info for this verification *****
*  Current legs: /dev/sdb1 /dev/sde1 /dev/sdh1 /dev/sdd1
*  Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
*  PV to corrupt /dev/sdb1
*  READ (non span): lvchange --writemostly /dev/sdb1:n black_bird
*  WRITE (non span): lvchange --writemostly /dev/sde1:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdh1:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdd1:y black_bird
*  (Clearing out OLD dmesg corruption detection notifications)
***************************************************************

Verifying files (checkit) on mirror(s) on...
        ---- host-093 ----

lvchange -an black_bird/synced_random_raid1_3legs_1
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdb1 oflag=direct,sync  bs=1M seek=26 count=4
4+0 records in
4+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0861356 s, 48.7 MB/s
Verifying files (checkit) on mirror(s) on...
        ---- host-093 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1
Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1

lvs --noheadings -o integritymismatches black_bird/synced_random_raid1_3legs_1 | awk {'print $1'}
Corruption mismatches reported: 64

Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
lvs --noheadings -o integritymismatches black_bird/synced_random_raid1_3legs_1 | awk {'print $1'}
Corruption mismatches reported: 0

re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1


** NO sign of integrity correction found!! ** 


Disabling device sde on host-093rescan device...
  Error reading device /dev/sde1 at 0 length 4096.


Attempting I/O to cause mirror down conversion(s) on host-093
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0609431 s, 688 MB/s

Verifying current sanity of lvm after the failure

Current mirror/raid device structure(s):
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
  LV                                             Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                    rwi-aor-r- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_4(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]         gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta]   ewi-ao----  12.00m          /dev/sdb1(126)
   [synced_random_raid1_3legs_1_rimage_0_iorig]   -wi-ao---- 500.00m          /dev/sdb1(1)
   synced_random_raid1_3legs_1_rimage_1_extracted gwi-ao--p- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta]   ewi-ao----  12.00m          [unknown](126)
   [synced_random_raid1_3legs_1_rimage_1_iorig]   -wi-ao--p- 500.00m          [unknown](1)
   [synced_random_raid1_3legs_1_rimage_2]         gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta]   ewi-ao----  12.00m          /dev/sdh1(126)
   [synced_random_raid1_3legs_1_rimage_2_iorig]   -wi-ao---- 500.00m          /dev/sdh1(1)
   [synced_random_raid1_3legs_1_rimage_3]         gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta]   ewi-ao----  12.00m          /dev/sdd1(126)
   [synced_random_raid1_3legs_1_rimage_3_iorig]   -wi-ao---- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rimage_4]         Iwi---r--- 500.00m          /dev/sda1(1)
   synced_random_raid1_3legs_1_rimage_4_imeta     -wi-------  12.00m          /dev/sda1(126)
   [synced_random_raid1_3legs_1_rmeta_0]          ewi-aor---   4.00m          /dev/sdb1(0)
   synced_random_raid1_3legs_1_rmeta_1_extracted  -wi-ao--p-   4.00m          [unknown](0)
   [synced_random_raid1_3legs_1_rmeta_2]          ewi-aor---   4.00m          /dev/sdh1(0)
   [synced_random_raid1_3legs_1_rmeta_3]          ewi-aor---   4.00m          /dev/sdd1(0)
   [synced_random_raid1_3legs_1_rmeta_4]          ewi---r---   4.00m          /dev/sda1(0)


Verifying FAILED device /dev/sde1 is *NOT* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verifying IMAGE device /dev/sdb1 *IS* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verifying IMAGE device /dev/sdh1 *IS* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verifying IMAGE device /dev/sdd1 *IS* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_1_imeta on: host-093 

(ALLOCATE POLICY) there should not be an 'unknown' device associated with synced_random_raid1_3legs_1_rimage_1_imeta on host-093

  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
   [synced_random_raid1_3legs_1_rimage_1_imeta]            [unknown](126)                                                                                                                                                 

Attempt to trigger automatic repair again...
Attempting I/O to cause mirror down conversion(s) on host-093
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0932387 s, 450 MB/s


        [unknown] device(s) still exist in raid that should have been repaired by now
This is most likely bug 1729303


Jan 12 17:34:42 host-093 qarshd[137299]: Running cmdline: pvscan --cache /dev/sde1
Jan 12 17:34:42 host-093 kernel: sd 7:0:0:0: rejecting I/O to offline device
Jan 12 17:34:42 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:42 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55950.service: Succeeded.
Jan 12 17:34:43 host-093 systemd[1]: Started qarsh Per-Connection Server (10.3.112.236:55960).
Jan 12 17:34:43 host-093 qarshd[137304]: Talking to peer ::ffff:10.3.112.236:55960 (IPv6)
Jan 12 17:34:44 host-093 qarshd[137304]: Running cmdline: pvs -a
Jan 12 17:34:44 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:44 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:44 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55960.service: Succeeded.
Jan 12 17:34:44 host-093 systemd[1]: Started qarsh Per-Connection Server (10.3.112.236:55964).
Jan 12 17:34:44 host-093 qarshd[137311]: Talking to peer ::ffff:10.3.112.236:55964 (IPv6)
Jan 12 17:34:45 host-093 qarshd[137311]: Running cmdline: dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
Jan 12 17:34:45 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55964.service: Succeeded.
Jan 12 17:34:45 host-093 systemd[1]: Started qarsh Per-Connection Server (10.3.112.236:55970).
Jan 12 17:34:45 host-093 qarshd[137316]: Talking to peer ::ffff:10.3.112.236:55970 (IPv6)
Jan 12 17:34:46 host-093 qarshd[137316]: Running cmdline: sync
Jan 12 17:34:46 host-093 kernel: blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Jan 12 17:34:46 host-093 kernel: blk_update_request: I/O error, dev sde, sector 16432 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jan 12 17:34:46 host-093 kernel: md: super_written gets error=-5
Jan 12 17:34:46 host-093 kernel: md/raid1:mdX: Disk failure on dm-13, disabling device.#012md/raid1:mdX: Operation continuing on 3 devices.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Device #1 of raid1 array, black_bird-synced_random_raid1_3legs_1, has failed.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: waiting for resynchronization to finish before initiating repair on RAID device black_bird-synced_random_raid1_3legs_1.
Jan 12 17:34:46 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to /dev/sde1).
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
Jan 12 17:34:46 host-093 lvm[731]: Creating integrity metadata LV synced_random_raid1_3legs_1_rimage_4_imeta with size 12.00 MiB.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
Jan 12 17:34:46 host-093 lvm[731]: Logical volume "synced_random_raid1_3legs_1_rimage_4_imeta" created.
Jan 12 17:34:46 host-093 lvm[731]: Device open /dev/black_bird/synced_primary_raid1_3legs_1_rimage_4_imeta 253:23 failed errno 2
Jan 12 17:34:46 host-093 lvm[731]: Device open /dev/black_bird/synced_primary_raid1_3legs_1_rimage_4_imeta 253:23 failed errno 2
Jan 12 17:34:46 host-093 lvm[731]: Failed to open black_bird/synced_random_raid1_3legs_1_rimage_4_imeta for wiping and zeroing.
Jan 12 17:34:46 host-093 lvm[731]: Failed to zero LV for integrity metadata black_bird/synced_random_raid1_3legs_1_rimage_4_imeta
Jan 12 17:34:46 host-093 lvm[731]: Failed to deactivate LV black_bird/synced_random_raid1_3legs_1_rimage_4_imeta after zero
Jan 12 17:34:46 host-093 kernel: dmeventd[137214]: segfault at f0 ip 00007f7de39f5d84 sp 00007f7de6a7c7a8 error 4 in liblvm2cmd.so.2.03[7f7de3985000+221000]
Jan 12 17:34:46 host-093 kernel: Code: be 00 10 00 00 48 89 df e8 49 09 fb ff 48 85 c0 74 9e eb a1 e8 dd 05 fb ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa <48> 8b 97 f0 00 00 00 31 c0 48 85 d2 74 09 8b 12 31 c0 85 d2 0f 95
Jan 12 17:34:46 host-093 lvm[731]: Failed to add integrity.
Jan 12 17:34:46 host-093 systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Jan 12 17:34:46 host-093 systemd[1]: Started Process Core Dump (PID 137343/UID 0).
Jan 12 17:34:48 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55970.service: Succeeded.
Jan 12 17:34:49 host-093 systemd[1]: dm-event.service: Main process exited, code=killed, status=11/SEGV
Jan 12 17:34:49 host-093 systemd[1]: dm-event.service: Failed with result 'signal'.
Jan 12 17:34:49 host-093 systemd-coredump[137346]: Process 731 (dmeventd) of user 0 dumped core.#012#012Stack trace of thread 137214:#012#0  0x00007f7de39f5d84 lv_is_historical (liblvm2cmd.so.2.03)#012#1  0x00007f7de3a0388d lv_remove (liblvm2cmd.so.2.03)#012#2  0x00007f7de39f3f4c lv_add_integrity_to_raid (liblvm2cmd.so.2.03)#012#3  0x00007f7de3a2dadc _lv_raid_rebuild_or_replace.isra.51 (liblvm2cmd.so.2.03)#012#4  0x00007f7de3a80ac7 _lvconvert_repair_single (liblvm2cmd.so.2.03)#012#5  0x00007f7de3aad7b0 process_each_lv_in_vg (liblvm2cmd.so.2.03)#012#6  0x00007f7de3aae89b process_each_lv (liblvm2cmd.so.2.03)#012#7  0x00007f7de3a81000 lvconvert_repair_cmd (liblvm2cmd.so.2.03)#012#8  0x00007f7de3a8e636 lvm_run_command (liblvm2cmd.so.2.03)#012#9  0x00007f7de3abe37e lvm2_run (liblvm2cmd.so.2.03)#012#10 0x00007f7de40634a7 dmeventd_lvm2_run (libdevmapper-event-lvm2.so.2.03)#012#11 0x00007f7de26b2107 process_event (libdevmapper-event-lvm2raid.so)#012#12 0x000055d9abd23b2a _monitor_thread (dmeventd)#012#13 0x00007f7de559714a start_thread (libpthread.so.0)#012#14 0x00007f7de4e54db3 __clone (libc.so.6)#012#012Stack trace of thread 735:#012#0  0x00007f7de4e4b61b ioctl (libc.so.6)#012#1  0x00007f7de5373851 dm_task_run (libdevmapper.so.1.02)#012#2  0x000055d9abd23bf7 _monitor_thread (dmeventd)#012#3  0x00007f7de559714a start_thread (libpthread.so.0)#012#4  0x00007f7de4e54db3 __clone (libc.so.6)#012#012Stack trace of thread 731:#012#0  0x00007f7de4e4c28f __select (libc.so.6)#012#1  0x000055d9abd233a3 _client_read.isra.15 (dmeventd)#012#2  0x000055d9abd21f05 main (dmeventd)#012#3  0x00007f7de4d7b493 __libc_start_main (libc.so.6)#012#4  0x000055d9abd22a1e _start (dmeventd)#012#012Stack trace of thread 736:#012#0  0x00007f7de559d64a pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1  0x000055d9abd254d6 _timeout_thread (dmeventd)#012#2  0x00007f7de559714a start_thread (libpthread.so.0)#012#3  0x00007f7de4e54db3 __clone (libc.so.6)
Jan 12 17:34:49 host-093 systemd[1]: systemd-coredump: Succeeded.




[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/dmeventd -f'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  lv_is_historical (lv=lv@entry=0x0) at metadata/lv.c:33
33      metadata/lv.c: No such file or directory.
[Current thread is 1 (Thread 0x7f7de6a7f700 (LWP 137214))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-145.el8.x86_64 libaio-0.3.112-1.el8.x86_64 libblkid-2.32.1-26.el8.x86_64 libcap-2.26-4.el8.x86_64 libgcc-8.4.1-1.el8.x86_64 libgcrypt-1.8.5-4.el8.x86_64 libgpg-error-1.31-1.el8.x86_64 libmount-2.32.1-26.el8.x86_64 libselinux-2.9-5.el8.x86_64 libsepol-2.9-1.el8.x86_64 libuuid-2.32.1-26.el8.x86_64 lz4-libs-1.8.3-2.el8.x86_64 pcre2-10.32-2.el8.x86_64 systemd-libs-239-43.el8.x86_64 xz-libs-5.2.4-3.el8.x86_64
(gdb) bt
#0  lv_is_historical (lv=lv@entry=0x0) at metadata/lv.c:33
#1  0x00007f7de3a0388d in lv_remove (lv=0x0) at metadata/lv_manip.c:1755
#2  0x00007f7de39f3f4c in lv_add_integrity_to_raid (lv=lv@entry=0x7f7dd4066188, settings=0x7f7dd4068028, pvh=pvh@entry=0x0, lv_imeta_0=lv_imeta_0@entry=0x0) at metadata/integrity_manip.c:777
#3  0x00007f7de3a2dadc in _lv_raid_rebuild_or_replace (lv=lv@entry=0x7f7dd4066188, remove_pvs=remove_pvs@entry=0x7f7dd406aca0, allocate_pvs=allocate_pvs@entry=0x7f7dd4065410, 
    rebuild=rebuild@entry=0, force=<optimized out>) at metadata/raid_manip.c:7027
#4  0x00007f7de3a38441 in lv_raid_replace (lv=lv@entry=0x7f7dd4066188, force=<optimized out>, remove_pvs=remove_pvs@entry=0x7f7dd406aca0, allocate_pvs=allocate_pvs@entry=0x7f7dd4065410)
    at metadata/raid_manip.c:7126
#5  0x00007f7de3a80ac7 in _lvconvert_repair_pvs_raid (handle=0x7f7dd4008cf0, use_pvh=0x7f7dd4065410, lv=0x7f7dd4066188, cmd=0x7f7ddc000cc0) at lvconvert.c:3773
#6  _lvconvert_repair_pvs (handle=0x7f7dd4008cf0, lv=0x7f7dd4066188, cmd=0x7f7ddc000cc0) at lvconvert.c:3806
#7  _lvconvert_repair_single (cmd=cmd@entry=0x7f7ddc000cc0, lv=0x7f7dd4066188, handle=handle@entry=0x7f7dd4008cf0) at lvconvert.c:3857
#8  0x00007f7de3aad7b0 in process_each_lv_in_vg (cmd=cmd@entry=0x7f7ddc000cc0, vg=vg@entry=0x7f7dd4065350, arg_lvnames=arg_lvnames@entry=0x7f7de6a7e510, tags_in=tags_in@entry=0x7f7de6a7e4c0, 
    stop_on_error=stop_on_error@entry=0, handle=handle@entry=0x7f7dd4008cf0, check_single_lv=0x0, process_single_lv=0x7f7de3a80370 <_lvconvert_repair_single>) at toollib.c:3148
#9  0x00007f7de3aae89b in _process_lv_vgnameid_list (process_single_lv=0x7f7de3a80370 <_lvconvert_repair_single>, check_single_lv=0x0, handle=0x7f7dd4008cf0, arg_tags=0x7f7de6a7e4c0, 
    arg_lvnames=0x7f7de6a7e4e0, arg_vgnames=0x7f7de6a7e4d0, vgnameids_to_process=0x7f7de6a7e500, read_flags=1048576, cmd=0x7f7ddc000cc0) at toollib.c:3616
#10 process_each_lv (cmd=cmd@entry=0x7f7ddc000cc0, argc=argc@entry=1, argv=<optimized out>, one_vgname=one_vgname@entry=0x0, one_lvname=one_lvname@entry=0x0, read_flags=read_flags@entry=1048576, 
    handle=0x7f7dd4008cf0, check_single_lv=<optimized out>, process_single_lv=<optimized out>) at toollib.c:3773
#11 0x00007f7de3a81000 in lvconvert_repair_cmd (cmd=0x7f7ddc000cc0, argc=<optimized out>, argv=<optimized out>) at lvconvert.c:3893
#12 0x00007f7de3a8e636 in lvm_run_command (cmd=cmd@entry=0x7f7ddc000cc0, argc=<optimized out>, argv=<optimized out>, argv@entry=0x7f7de6a7e850) at lvmcmdline.c:3132
#13 0x00007f7de3abe37e in lvm2_run (handle=0x7f7ddc000cc0, cmdline=<optimized out>, cmdline@entry=0x7f7dd4039b88 "lvconvert --repair --use-policies black_bird/synced_random_raid1_3legs_1")
    at lvmcmdlib.c:93
#14 0x00007f7de40634a7 in dmeventd_lvm2_run (cmdline=cmdline@entry=0x7f7dd4039b88 "lvconvert --repair --use-policies black_bird/synced_random_raid1_3legs_1") at dmeventd_lvm.c:126
#15 0x00007f7de26b2107 in _process_raid_event (device=0x7f7dd483a7b0 "black_bird-synced_random_raid1_3legs_1", params=<optimized out>, state=0x7f7dd4039b80) at dmeventd_raid.c:98
#16 process_event (dmt=0x55d9ac76bf40, event=<optimized out>, user=<optimized out>) at dmeventd_raid.c:141
#17 0x000055d9abd23b2a in _do_process_event (thread=0x55d9ac76da10) at dmeventd.c:954
#18 _monitor_thread (arg=0x55d9ac76da10) at dmeventd.c:1042
#19 0x00007f7de559714a in start_thread () from /lib64/libpthread.so.0
#20 0x00007f7de4e54db3 in clone () from /lib64/libc.so.6
Comment 2 David Teigland 2021-01-13 19:46:34 UTC 
segfault fixed by
https://sourceware.org/git/?p=lvm2.git;a=commit;h=0534723a2de62da913dfd88d40ee6f8b8b93ac56

The segfault was caused by a bug on the error exit path while trying to revert imeta lvs that were created to replace the failed image.  There is not a way to naturally force that error exit path to be followed, so to verify the fix I added code force that code path.

before:
[ 0:07] 13:27:30.600728 lvconvert[985988] metadata/integrity_manip.c:627  forcing error prior to wipe_lv LV1_rimage_2_imeta
[ 0:07] 13:27:30.600738 lvconvert[985988] metadata/integrity_manip.c:628        <backtrace>
[ 0:07] 13:27:30.600760 lvconvert[985988] metadata/integrity_manip.c:779  Failed to add integrity.
[ 0:07] 6,6041,5001999686211,-;lvm[985988]: segfault at f0 ip 000055e75cebee47 sp 00007fff0ca0b630 error 4 in lvm[55e75cdca000+27e000]
[ 0:07] /root/lvm.git/test/shell/integrity-misc.sh: line 203: 985988 Segmentation fault      (core dumped) lvconvert -vvvv -y --repair $vg/$lv1

after:
[ 0:07] 13:24:18.505498 lvconvert[982038] metadata/integrity_manip.c:627  forcing error prior to wipe_lv LV1_rimage_2_imeta
[ 0:07] 13:24:18.505506 lvconvert[982038] metadata/integrity_manip.c:628        <backtrace>
[ 0:07] 13:24:18.505519 lvconvert[982038] metadata/integrity_manip.c:779  Failed to add integrity.


There appears to be a second bug in the backtrace related to a failure to open a new imeta LV to zero it.  That bug is what exposed the segfault fixed here.  A different bz should probably be created for the failure to open/wipe the new imeta LV.
Comment 3 Corey Marthaler 2021-02-03 03:42:15 UTC 
Marking Verified:Tested

kernel-4.18.0-277.el8    BUILT: Wed Jan 20 09:06:28 CST 2021
lvm2-2.03.11-2.el8    BUILT: Thu Jan 28 14:40:36 CST 2021
lvm2-libs-2.03.11-2.el8    BUILT: Thu Jan 28 14:40:36 CST 2021


================================================================================
Iteration 1.2 started at Tue 02 Feb 2021 09:25:30 PM CST
================================================================================
Scenario kill_random_synced_raid1_3legs: Kill random leg of synced 3 leg raid1 volume(s)
********* RAID hash info for this scenario *********
* names:              synced_random_raid1_3legs_1 synced_random_raid1_3legs_2
* sync:               1
* type:               raid1
* -m |-i value:       3
* leg devices:        /dev/sdb1 /dev/sdg1 /dev/sdc1 /dev/sdi1
* spanned legs:       0
* manual repair:      0
* no MDA devices:     
* failpv(s):          /dev/sdc1
* failnode(s):        hayes-03
* integrity stack:    1
 (Due to integrity stack, be mindful of false failures that are reliant on message checks
  that could be lost due to rate-limiting of corruption and other messages)
* raid fault policy:  allocate
******************************************************

Creating raids(s) on hayes-03...
hayes-03: lvcreate --yes  --type raid1 -m 3 -n synced_random_raid1_3legs_1 -L 500M black_bird /dev/sdb1:0-2400 /dev/sdg1:0-2400 /dev/sdc1:0-2400 /dev/sdi1:0-2400
hayes-03: lvcreate --yes  --type raid1 -m 3 -n synced_random_raid1_3legs_2 -L 500M black_bird /dev/sdb1:0-2400 /dev/sdg1:0-2400 /dev/sdc1:0-2400 /dev/sdi1:0-2400
[...]
Comment 6 Corey Marthaler 2021-02-16 21:22:34 UTC 
Fix verified in the latest nightly kernel and lvm.

kernel-4.18.0-287.el8    BUILT: Thu Feb 11 03:15:20 CST 2021
lvm2-2.03.11-4.el8    BUILT: Thu Feb 11 04:35:23 CST 2021
lvm2-libs-2.03.11-4.el8    BUILT: Thu Feb 11 04:35:23 CST 2021


================================================================================
Iteration 1.1 started at Tue 16 Feb 2021 01:38:55 PM CST
================================================================================
Scenario kill_random_synced_raid1_3legs: Kill random leg of synced 3 leg raid1 volume(s)
********* RAID hash info for this scenario *********
* names:              synced_random_raid1_3legs_1 synced_random_raid1_3legs_2
* sync:               1
* type:               raid1
* -m |-i value:       3
* leg devices:        /dev/sdj1 /dev/sdk1 /dev/sdd1 /dev/sde1
* spanned legs:       0
* manual repair:      0
* no MDA devices:     
* failpv(s):          /dev/sdd1
* failnode(s):        hayes-03
* integrity stack:    1
 (Due to integrity stack, be mindful of false failures that are reliant on message checks
  that could be lost due to rate-limiting of corruption and other messages)
* raid fault policy:  allocate
******************************************************

Creating raids(s) on hayes-03...
hayes-03: lvcreate --yes  --type raid1 -m 3 -n synced_random_raid1_3legs_1 -L 500M black_bird /dev/sdj1:0-2400 /dev/sdk1:0-2400 /dev/sdd1:0-2400 /dev/sde1:0-2400
hayes-03: lvcreate --yes  --type raid1 -m 3 -n synced_random_raid1_3legs_2 -L 500M black_bird /dev/sdj1:0-2400 /dev/sdk1:0-2400 /dev/sdd1:0-2400 /dev/sde1:0-2400

Current mirror/raid device structure(s):
  LV                                     Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1            rwi-a-r--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0] iwi-aor--- 500.00m          /dev/sdj1(1)
   [synced_random_raid1_3legs_1_rimage_1] iwi-aor--- 500.00m          /dev/sdk1(1)
   [synced_random_raid1_3legs_1_rimage_2] iwi-aor--- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rimage_3] iwi-aor--- 500.00m          /dev/sde1(1)
   [synced_random_raid1_3legs_1_rmeta_0]  ewi-aor---   4.00m          /dev/sdj1(0)
   [synced_random_raid1_3legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sdk1(0)
   [synced_random_raid1_3legs_1_rmeta_2]  ewi-aor---   4.00m          /dev/sdd1(0)
   [synced_random_raid1_3legs_1_rmeta_3]  ewi-aor---   4.00m          /dev/sde1(0)
   synced_random_raid1_3legs_2            rwi-a-r--- 500.00m 37.58    synced_random_raid1_3legs_2_rimage_0(0),synced_random_raid1_3legs_2_rimage_1(0),synced_random_raid1_3legs_2_rimage_2(0),synced_random_raid1_3legs_2_rimage_3(0)
   [synced_random_raid1_3legs_2_rimage_0] Iwi-aor--- 500.00m          /dev/sdj1(127)
   [synced_random_raid1_3legs_2_rimage_1] Iwi-aor--- 500.00m          /dev/sdk1(127)
   [synced_random_raid1_3legs_2_rimage_2] Iwi-aor--- 500.00m          /dev/sdd1(127)
   [synced_random_raid1_3legs_2_rimage_3] Iwi-aor--- 500.00m          /dev/sde1(127)
   [synced_random_raid1_3legs_2_rmeta_0]  ewi-aor---   4.00m          /dev/sdj1(126)
   [synced_random_raid1_3legs_2_rmeta_1]  ewi-aor---   4.00m          /dev/sdk1(126)
   [synced_random_raid1_3legs_2_rmeta_2]  ewi-aor---   4.00m          /dev/sdd1(126)
   [synced_random_raid1_3legs_2_rmeta_3]  ewi-aor---   4.00m          /dev/sde1(126)

Waiting until all mirror|raid volumes become fully syncd...
   2/2 mirror(s) are fully synced: ( 100.00% 100.00% )

Convert mirror/raid volume(s) to utilize integrity target volume(s) on hayes-03...

lvconvert --yes --raidintegrity y --raidintegritymode journal black_bird/synced_random_raid1_3legs_1
lvconvert --yes --raidintegrity y --raidintegritymode journal black_bird/synced_random_raid1_3legs_2
Creating xfs on top of mirror(s) on hayes-03...
Mounting mirrored xfs filesystems on hayes-03...


Current mirror/raid device structure(s):
  LV                                           Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                  rwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdj1(252)
   [synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdj1(1)
   [synced_random_raid1_3legs_1_rimage_1]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao----  12.00m          /dev/sdk1(252)
   [synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao---- 500.00m          /dev/sdk1(1)
   [synced_random_raid1_3legs_1_rimage_2]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdd1(252)
   [synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rimage_3]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao----  12.00m          /dev/sde1(252)
   [synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sde1(1)
   [synced_random_raid1_3legs_1_rmeta_0]        ewi-aor---   4.00m          /dev/sdj1(0)
   [synced_random_raid1_3legs_1_rmeta_1]        ewi-aor---   4.00m          /dev/sdk1(0)
   [synced_random_raid1_3legs_1_rmeta_2]        ewi-aor---   4.00m          /dev/sdd1(0)
   [synced_random_raid1_3legs_1_rmeta_3]        ewi-aor---   4.00m          /dev/sde1(0)
   synced_random_raid1_3legs_2                  rwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_0(0),synced_random_raid1_3legs_2_rimage_1(0),synced_random_raid1_3legs_2_rimage_2(0),synced_random_raid1_3legs_2_rimage_3(0)
   [synced_random_raid1_3legs_2_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_0_iorig(0)
   [synced_random_raid1_3legs_2_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdj1(255)
   [synced_random_raid1_3legs_2_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdj1(127)
   [synced_random_raid1_3legs_2_rimage_1]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_1_iorig(0)
   [synced_random_raid1_3legs_2_rimage_1_imeta] ewi-ao----  12.00m          /dev/sdk1(255)
   [synced_random_raid1_3legs_2_rimage_1_iorig] -wi-ao---- 500.00m          /dev/sdk1(127)
   [synced_random_raid1_3legs_2_rimage_2]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_2_iorig(0)
   [synced_random_raid1_3legs_2_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdd1(255)
   [synced_random_raid1_3legs_2_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdd1(127)
   [synced_random_raid1_3legs_2_rimage_3]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_3_iorig(0)
   [synced_random_raid1_3legs_2_rimage_3_imeta] ewi-ao----  12.00m          /dev/sde1(255)
   [synced_random_raid1_3legs_2_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sde1(127)
   [synced_random_raid1_3legs_2_rmeta_0]        ewi-aor---   4.00m          /dev/sdj1(126)
   [synced_random_raid1_3legs_2_rmeta_1]        ewi-aor---   4.00m          /dev/sdk1(126)
   [synced_random_raid1_3legs_2_rmeta_2]        ewi-aor---   4.00m          /dev/sdd1(126)
   [synced_random_raid1_3legs_2_rmeta_3]        ewi-aor---   4.00m          /dev/sde1(126)

PV=/dev/sdd1
        synced_random_raid1_3legs_1_rimage_2_imeta: 1.0
        synced_random_raid1_3legs_1_rimage_2_iorig: 1.0
        synced_random_raid1_3legs_1_rmeta_2: 1.0
        synced_random_raid1_3legs_2_rimage_2_imeta: 1.0
        synced_random_raid1_3legs_2_rimage_2_iorig: 1.0
        synced_random_raid1_3legs_2_rmeta_2: 1.0

Writing verification files (checkit) to mirror(s) on...
        ---- hayes-03 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 

Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

**** Raid Integrity Corruption info for this verification *****
*  Current legs: /dev/sdj1 /dev/sdk1 /dev/sdd1 /dev/sde1
*  Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
*  PV to corrupt /dev/sdj1
*  READ (non span): lvchange --writemostly /dev/sdj1:n black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdk1:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdd1:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sde1:y black_bird
*  (Clearing out OLD dmesg corruption detection notifications)
***************************************************************

Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

lvchange -an black_bird/synced_random_raid1_3legs_1
lvchange -an black_bird/synced_random_raid1_3legs_2
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdj1 oflag=direct,sync  bs=1M seek=25 count=28
28+0 records in
28+0 records out
29360128 bytes (29 MB, 28 MiB) copied, 0.463076 s, 63.4 MB/s
Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1
Detecting corruption on bad image one of two ways:
FULL READ: dd if=/dev/black_bird/synced_random_raid1_3legs_1 of=/dev/null iflag=direct
1024000+0 records in
1024000+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 514.189 s, 1.0 MB/s

Corruption mismatches reported: 114560
Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
Corruption mismatches reported: 0
re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1

lvchange -ay black_bird/synced_random_raid1_3legs_2
Detecting corruption on bad image one of two ways:
FULL READ: dd if=/dev/black_bird/synced_random_raid1_3legs_2 of=/dev/null iflag=direct
1024000+0 records in
1024000+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 36.816 s, 14.2 MB/s

Corruption mismatches reported: 0
Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_2)
Corruption mismatches reported: 0
re-mounting /mnt/synced_random_raid1_3legs_2
re-verifying checkit files in /mnt/synced_random_raid1_3legs_2

** SIGN of integrity correction found!! ** 
[11235.855288] md/raid1:mdX: read error corrected (1 sectors at 85823 on dm-3)

Disabling device sdd on hayes-03rescan device...
  Error reading device /dev/sdd1 at 0 length 4096.

Attempting I/O to cause mirror down conversion(s) on hayes-03
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0197865 s, 2.1 GB/s
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_2/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.020657 s, 2.0 GB/s

Verifying current sanity of lvm after the failure

Current mirror/raid device structure(s):
  WARNING: Couldn't find device with uuid k6nOBz-t9Ht-qtRk-ASjm-F2tc-Q7ea-vngB0F.
  WARNING: VG black_bird is missing PV k6nOBz-t9Ht-qtRk-ASjm-F2tc-Q7ea-vngB0F (last written to [unknown]).
  LV                                           Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                  rwi-aor--- 500.00m 55.69    synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdj1(252)
   [synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdj1(1)
   [synced_random_raid1_3legs_1_rimage_1]       gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao----  12.00m          /dev/sdk1(252)
   [synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao---- 500.00m          /dev/sdk1(1)
   [synced_random_raid1_3legs_1_rimage_2]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdb1(255)
   [synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdb1(130)
   [synced_random_raid1_3legs_1_rimage_3]       gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao----  12.00m          /dev/sde1(252)
   [synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sde1(1)
   [synced_random_raid1_3legs_1_rmeta_0]        ewi-aor---   4.00m          /dev/sdj1(0)
   [synced_random_raid1_3legs_1_rmeta_1]        ewi-aor---   4.00m          /dev/sdk1(0)
   [synced_random_raid1_3legs_1_rmeta_2]        ewi-aor---   4.00m          /dev/sdb1(129)
   [synced_random_raid1_3legs_1_rmeta_3]        ewi-aor---   4.00m          /dev/sde1(0)
   synced_random_raid1_3legs_2                  rwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_0(0),synced_random_raid1_3legs_2_rimage_1(0),synced_random_raid1_3legs_2_rimage_2(0),synced_random_raid1_3legs_2_rimage_3(0)
   [synced_random_raid1_3legs_2_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_0_iorig(0)
   [synced_random_raid1_3legs_2_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdj1(255)
   [synced_random_raid1_3legs_2_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdj1(127)
   [synced_random_raid1_3legs_2_rimage_1]       gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_1_iorig(0)
   [synced_random_raid1_3legs_2_rimage_1_imeta] ewi-ao----  12.00m          /dev/sdk1(255)
   [synced_random_raid1_3legs_2_rimage_1_iorig] -wi-ao---- 500.00m          /dev/sdk1(127)
   [synced_random_raid1_3legs_2_rimage_2]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_2_rimage_2_iorig(0)
   [synced_random_raid1_3legs_2_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdb1(126)
   [synced_random_raid1_3legs_2_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdb1(1)
   [synced_random_raid1_3legs_2_rimage_3_imeta] ewi-ao----  12.00m          /dev/sde1(255)
   [synced_random_raid1_3legs_2_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sde1(127)
   [synced_random_raid1_3legs_2_rmeta_0]        ewi-aor---   4.00m          /dev/sdj1(126)
   [synced_random_raid1_3legs_2_rmeta_1]        ewi-aor---   4.00m          /dev/sdk1(126)
   [synced_random_raid1_3legs_2_rmeta_2]        ewi-aor---   4.00m          /dev/sdb1(0)
   [synced_random_raid1_3legs_2_rmeta_3]        ewi-aor---   4.00m          /dev/sde1(126

Verifying FAILED device /dev/sdd1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdj1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdk1 *IS* in the volume(s)
Verifying IMAGE device /dev/sde1 *IS* in the volume(s)
Verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_2_imeta on: hayes-03 
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_2_iorig on: hayes-03 
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rmeta_2 on: hayes-03 
Checking EXISTENCE and STATE of synced_random_raid1_3legs_2_rimage_2_imeta on: hayes-03 
Checking EXISTENCE and STATE of synced_random_raid1_3legs_2_rimage_2_iorig on: hayes-03 
Checking EXISTENCE and STATE of synced_random_raid1_3legs_2_rmeta_2 on: hayes-03 

[...]
Comment 8 errata-xmlrpc 2021-05-18 15:02:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659
Note You need to log in before you can comment on or make changes to this bug.