Bug 1915580

Description of problem:

kernel-4.18.0-271.el8    BUILT: Fri Jan  8 03:32:43 CST 2021
lvm2-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-libs-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-dbusd-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:35:05 CST 2020


===============================================================================
Iteration 1.1 started at Tue 12 Jan 2021 05:30:16 PM CST
===============================================================================
Scenario kill_random_synced_raid1_3legs: Kill random leg of synced 3 leg raid1 volume(s)
********* RAID hash info for this scenario *********
* names:              synced_random_raid1_3legs_1
* sync:               1
* type:               raid1
* -m |-i value:       3
* leg devices:        /dev/sdb1 /dev/sde1 /dev/sdh1 /dev/sdd1
* spanned legs:       0
* manual repair:      0
* no MDA devices:     
* failpv(s):          /dev/sde1
* failnode(s):        host-093
* integrity stack:    1
 (Due to integrity stack, be mindful of false failures that are reliant on message checks
  that could be lost due to rate-limiting of corruption and other messages)
* raid fault policy:  allocate
******************************************************

Creating raids(s) on host-093...
host-093: lvcreate --yes  --type raid1 -m 3 -n synced_random_raid1_3legs_1 -L 500M black_bird /dev/sdb1:0-2400 /dev/sde1:0-2400 /dev/sdh1:0-2400 /dev/sdd1:0-2400

Current mirror/raid device structure(s):
  LV                                     Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1            rwi-a-r--- 500.00m 6.26     synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0] Iwi-aor--- 500.00m          /dev/sdb1(1)
   [synced_random_raid1_3legs_1_rimage_1] Iwi-aor--- 500.00m          /dev/sde1(1)
   [synced_random_raid1_3legs_1_rimage_2] Iwi-aor--- 500.00m          /dev/sdh1(1)
   [synced_random_raid1_3legs_1_rimage_3] Iwi-aor--- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rmeta_0]  ewi-aor---   4.00m          /dev/sdb1(0)
   [synced_random_raid1_3legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sde1(0)
   [synced_random_raid1_3legs_1_rmeta_2]  ewi-aor---   4.00m          /dev/sdh1(0)
   [synced_random_raid1_3legs_1_rmeta_3]  ewi-aor---   4.00m          /dev/sdd1(0)


Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Convert mirror/raid volume(s) to utilize integrity target volume(s) on host-093...

lvconvert --yes --raidintegrity y --raidintegritymode bitmap black_bird/synced_random_raid1_3legs_1
Creating xfs on top of mirror(s) on host-093...
Mounting mirrored xfs filesystems on host-093...

Current mirror/raid device structure(s):
  LV                                           Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                  rwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdb1(126)
   [synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdb1(1)
   [synced_random_raid1_3legs_1_rimage_1]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao----  12.00m          /dev/sde1(126)
   [synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao---- 500.00m          /dev/sde1(1)
   [synced_random_raid1_3legs_1_rimage_2]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdh1(126)
   [synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdh1(1)
   [synced_random_raid1_3legs_1_rimage_3]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao----  12.00m          /dev/sdd1(126)
   [synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rmeta_0]        ewi-aor---   4.00m          /dev/sdb1(0)
   [synced_random_raid1_3legs_1_rmeta_1]        ewi-aor---   4.00m          /dev/sde1(0)
   [synced_random_raid1_3legs_1_rmeta_2]        ewi-aor---   4.00m          /dev/sdh1(0)
   [synced_random_raid1_3legs_1_rmeta_3]        ewi-aor---   4.00m          /dev/sdd1(0)

PV=/dev/sde1
        synced_random_raid1_3legs_1_rimage_1_imeta: 1.0
        synced_random_raid1_3legs_1_rimage_1_iorig: 1.0
        synced_random_raid1_3legs_1_rmeta_1: 1.0

Writing verification files (checkit) to mirror(s) on...
        ---- host-093 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 

Verifying files (checkit) on mirror(s) on...
        ---- host-093 ----

**** Raid Integrity Corruption info for this verification *****
*  Current legs: /dev/sdb1 /dev/sde1 /dev/sdh1 /dev/sdd1
*  Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
*  PV to corrupt /dev/sdb1
*  READ (non span): lvchange --writemostly /dev/sdb1:n black_bird
*  WRITE (non span): lvchange --writemostly /dev/sde1:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdh1:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdd1:y black_bird
*  (Clearing out OLD dmesg corruption detection notifications)
***************************************************************

Verifying files (checkit) on mirror(s) on...
        ---- host-093 ----

lvchange -an black_bird/synced_random_raid1_3legs_1
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdb1 oflag=direct,sync  bs=1M seek=26 count=4
4+0 records in
4+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0861356 s, 48.7 MB/s
Verifying files (checkit) on mirror(s) on...
        ---- host-093 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1
Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1

lvs --noheadings -o integritymismatches black_bird/synced_random_raid1_3legs_1 | awk {'print $1'}
Corruption mismatches reported: 64

Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
lvs --noheadings -o integritymismatches black_bird/synced_random_raid1_3legs_1 | awk {'print $1'}
Corruption mismatches reported: 0

re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1


** NO sign of integrity correction found!! ** 


Disabling device sde on host-093rescan device...
  Error reading device /dev/sde1 at 0 length 4096.


Attempting I/O to cause mirror down conversion(s) on host-093
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0609431 s, 688 MB/s

Verifying current sanity of lvm after the failure

Current mirror/raid device structure(s):
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
  LV                                             Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                    rwi-aor-r- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_4(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]         gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta]   ewi-ao----  12.00m          /dev/sdb1(126)
   [synced_random_raid1_3legs_1_rimage_0_iorig]   -wi-ao---- 500.00m          /dev/sdb1(1)
   synced_random_raid1_3legs_1_rimage_1_extracted gwi-ao--p- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta]   ewi-ao----  12.00m          [unknown](126)
   [synced_random_raid1_3legs_1_rimage_1_iorig]   -wi-ao--p- 500.00m          [unknown](1)
   [synced_random_raid1_3legs_1_rimage_2]         gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta]   ewi-ao----  12.00m          /dev/sdh1(126)
   [synced_random_raid1_3legs_1_rimage_2_iorig]   -wi-ao---- 500.00m          /dev/sdh1(1)
   [synced_random_raid1_3legs_1_rimage_3]         gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta]   ewi-ao----  12.00m          /dev/sdd1(126)
   [synced_random_raid1_3legs_1_rimage_3_iorig]   -wi-ao---- 500.00m          /dev/sdd1(1)
   [synced_random_raid1_3legs_1_rimage_4]         Iwi---r--- 500.00m          /dev/sda1(1)
   synced_random_raid1_3legs_1_rimage_4_imeta     -wi-------  12.00m          /dev/sda1(126)
   [synced_random_raid1_3legs_1_rmeta_0]          ewi-aor---   4.00m          /dev/sdb1(0)
   synced_random_raid1_3legs_1_rmeta_1_extracted  -wi-ao--p-   4.00m          [unknown](0)
   [synced_random_raid1_3legs_1_rmeta_2]          ewi-aor---   4.00m          /dev/sdh1(0)
   [synced_random_raid1_3legs_1_rmeta_3]          ewi-aor---   4.00m          /dev/sdd1(0)
   [synced_random_raid1_3legs_1_rmeta_4]          ewi---r---   4.00m          /dev/sda1(0)


Verifying FAILED device /dev/sde1 is *NOT* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verifying IMAGE device /dev/sdb1 *IS* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verifying IMAGE device /dev/sdh1 *IS* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verifying IMAGE device /dev/sdd1 *IS* in the volume(s)
  WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
  WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to [unknown]).
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rmeta_1_extracted while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_iorig while checking used and assumed devices.
Verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_1_imeta on: host-093 

(ALLOCATE POLICY) there should not be an 'unknown' device associated with synced_random_raid1_3legs_1_rimage_1_imeta on host-093

  WARNING: Couldn't find all devices for LV black_bird/synced_random_raid1_3legs_1_rimage_1_imeta while checking used and assumed devices.
   [synced_random_raid1_3legs_1_rimage_1_imeta]            [unknown](126)                                                                                                                                                 

Attempt to trigger automatic repair again...
Attempting I/O to cause mirror down conversion(s) on host-093
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0932387 s, 450 MB/s


        [unknown] device(s) still exist in raid that should have been repaired by now
This is most likely bug 1729303


Jan 12 17:34:42 host-093 qarshd[137299]: Running cmdline: pvscan --cache /dev/sde1
Jan 12 17:34:42 host-093 kernel: sd 7:0:0:0: rejecting I/O to offline device
Jan 12 17:34:42 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:42 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55950.service: Succeeded.
Jan 12 17:34:43 host-093 systemd[1]: Started qarsh Per-Connection Server (10.3.112.236:55960).
Jan 12 17:34:43 host-093 qarshd[137304]: Talking to peer ::ffff:10.3.112.236:55960 (IPv6)
Jan 12 17:34:44 host-093 qarshd[137304]: Running cmdline: pvs -a
Jan 12 17:34:44 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:44 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:44 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55960.service: Succeeded.
Jan 12 17:34:44 host-093 systemd[1]: Started qarsh Per-Connection Server (10.3.112.236:55964).
Jan 12 17:34:44 host-093 qarshd[137311]: Talking to peer ::ffff:10.3.112.236:55964 (IPv6)
Jan 12 17:34:45 host-093 qarshd[137311]: Running cmdline: dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
Jan 12 17:34:45 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55964.service: Succeeded.
Jan 12 17:34:45 host-093 systemd[1]: Started qarsh Per-Connection Server (10.3.112.236:55970).
Jan 12 17:34:45 host-093 qarshd[137316]: Talking to peer ::ffff:10.3.112.236:55970 (IPv6)
Jan 12 17:34:46 host-093 qarshd[137316]: Running cmdline: sync
Jan 12 17:34:46 host-093 kernel: blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Jan 12 17:34:46 host-093 kernel: blk_update_request: I/O error, dev sde, sector 16432 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jan 12 17:34:46 host-093 kernel: md: super_written gets error=-5
Jan 12 17:34:46 host-093 kernel: md/raid1:mdX: Disk failure on dm-13, disabling device.#012md/raid1:mdX: Operation continuing on 3 devices.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Device #1 of raid1 array, black_bird-synced_random_raid1_3legs_1, has failed.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: waiting for resynchronization to finish before initiating repair on RAID device black_bird-synced_random_raid1_3legs_1.
Jan 12 17:34:46 host-093 kernel: blk_update_request: I/O error, dev sde, sector 40 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: VG black_bird is missing PV fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP (last written to /dev/sde1).
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
Jan 12 17:34:46 host-093 lvm[731]: Creating integrity metadata LV synced_random_raid1_3legs_1_rimage_4_imeta with size 12.00 MiB.
Jan 12 17:34:46 host-093 lvm[731]: WARNING: Couldn't find device with uuid fqQMJY-Hi4s-jg2F-XcdM-PqoO-VNuf-nXhqwP.
Jan 12 17:34:46 host-093 lvm[731]: Logical volume "synced_random_raid1_3legs_1_rimage_4_imeta" created.
Jan 12 17:34:46 host-093 lvm[731]: Device open /dev/black_bird/synced_primary_raid1_3legs_1_rimage_4_imeta 253:23 failed errno 2
Jan 12 17:34:46 host-093 lvm[731]: Device open /dev/black_bird/synced_primary_raid1_3legs_1_rimage_4_imeta 253:23 failed errno 2
Jan 12 17:34:46 host-093 lvm[731]: Failed to open black_bird/synced_random_raid1_3legs_1_rimage_4_imeta for wiping and zeroing.
Jan 12 17:34:46 host-093 lvm[731]: Failed to zero LV for integrity metadata black_bird/synced_random_raid1_3legs_1_rimage_4_imeta
Jan 12 17:34:46 host-093 lvm[731]: Failed to deactivate LV black_bird/synced_random_raid1_3legs_1_rimage_4_imeta after zero
Jan 12 17:34:46 host-093 kernel: dmeventd[137214]: segfault at f0 ip 00007f7de39f5d84 sp 00007f7de6a7c7a8 error 4 in liblvm2cmd.so.2.03[7f7de3985000+221000]
Jan 12 17:34:46 host-093 kernel: Code: be 00 10 00 00 48 89 df e8 49 09 fb ff 48 85 c0 74 9e eb a1 e8 dd 05 fb ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa <48> 8b 97 f0 00 00 00 31 c0 48 85 d2 74 09 8b 12 31 c0 85 d2 0f 95
Jan 12 17:34:46 host-093 lvm[731]: Failed to add integrity.
Jan 12 17:34:46 host-093 systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Jan 12 17:34:46 host-093 systemd[1]: Started Process Core Dump (PID 137343/UID 0).
Jan 12 17:34:48 host-093 systemd[1]: qarshd.105.93:5016-10.3.112.236:55970.service: Succeeded.
Jan 12 17:34:49 host-093 systemd[1]: dm-event.service: Main process exited, code=killed, status=11/SEGV
Jan 12 17:34:49 host-093 systemd[1]: dm-event.service: Failed with result 'signal'.
Jan 12 17:34:49 host-093 systemd-coredump[137346]: Process 731 (dmeventd) of user 0 dumped core.#012#012Stack trace of thread 137214:#012#0  0x00007f7de39f5d84 lv_is_historical (liblvm2cmd.so.2.03)#012#1  0x00007f7de3a0388d lv_remove (liblvm2cmd.so.2.03)#012#2  0x00007f7de39f3f4c lv_add_integrity_to_raid (liblvm2cmd.so.2.03)#012#3  0x00007f7de3a2dadc _lv_raid_rebuild_or_replace.isra.51 (liblvm2cmd.so.2.03)#012#4  0x00007f7de3a80ac7 _lvconvert_repair_single (liblvm2cmd.so.2.03)#012#5  0x00007f7de3aad7b0 process_each_lv_in_vg (liblvm2cmd.so.2.03)#012#6  0x00007f7de3aae89b process_each_lv (liblvm2cmd.so.2.03)#012#7  0x00007f7de3a81000 lvconvert_repair_cmd (liblvm2cmd.so.2.03)#012#8  0x00007f7de3a8e636 lvm_run_command (liblvm2cmd.so.2.03)#012#9  0x00007f7de3abe37e lvm2_run (liblvm2cmd.so.2.03)#012#10 0x00007f7de40634a7 dmeventd_lvm2_run (libdevmapper-event-lvm2.so.2.03)#012#11 0x00007f7de26b2107 process_event (libdevmapper-event-lvm2raid.so)#012#12 0x000055d9abd23b2a _monitor_thread (dmeventd)#012#13 0x00007f7de559714a start_thread (libpthread.so.0)#012#14 0x00007f7de4e54db3 __clone (libc.so.6)#012#012Stack trace of thread 735:#012#0  0x00007f7de4e4b61b ioctl (libc.so.6)#012#1  0x00007f7de5373851 dm_task_run (libdevmapper.so.1.02)#012#2  0x000055d9abd23bf7 _monitor_thread (dmeventd)#012#3  0x00007f7de559714a start_thread (libpthread.so.0)#012#4  0x00007f7de4e54db3 __clone (libc.so.6)#012#012Stack trace of thread 731:#012#0  0x00007f7de4e4c28f __select (libc.so.6)#012#1  0x000055d9abd233a3 _client_read.isra.15 (dmeventd)#012#2  0x000055d9abd21f05 main (dmeventd)#012#3  0x00007f7de4d7b493 __libc_start_main (libc.so.6)#012#4  0x000055d9abd22a1e _start (dmeventd)#012#012Stack trace of thread 736:#012#0  0x00007f7de559d64a pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1  0x000055d9abd254d6 _timeout_thread (dmeventd)#012#2  0x00007f7de559714a start_thread (libpthread.so.0)#012#3  0x00007f7de4e54db3 __clone (libc.so.6)
Jan 12 17:34:49 host-093 systemd[1]: systemd-coredump: Succeeded.




[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/dmeventd -f'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  lv_is_historical (lv=lv@entry=0x0) at metadata/lv.c:33
33      metadata/lv.c: No such file or directory.
[Current thread is 1 (Thread 0x7f7de6a7f700 (LWP 137214))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-145.el8.x86_64 libaio-0.3.112-1.el8.x86_64 libblkid-2.32.1-26.el8.x86_64 libcap-2.26-4.el8.x86_64 libgcc-8.4.1-1.el8.x86_64 libgcrypt-1.8.5-4.el8.x86_64 libgpg-error-1.31-1.el8.x86_64 libmount-2.32.1-26.el8.x86_64 libselinux-2.9-5.el8.x86_64 libsepol-2.9-1.el8.x86_64 libuuid-2.32.1-26.el8.x86_64 lz4-libs-1.8.3-2.el8.x86_64 pcre2-10.32-2.el8.x86_64 systemd-libs-239-43.el8.x86_64 xz-libs-5.2.4-3.el8.x86_64
(gdb) bt
#0  lv_is_historical (lv=lv@entry=0x0) at metadata/lv.c:33
#1  0x00007f7de3a0388d in lv_remove (lv=0x0) at metadata/lv_manip.c:1755
#2  0x00007f7de39f3f4c in lv_add_integrity_to_raid (lv=lv@entry=0x7f7dd4066188, settings=0x7f7dd4068028, pvh=pvh@entry=0x0, lv_imeta_0=lv_imeta_0@entry=0x0) at metadata/integrity_manip.c:777
#3  0x00007f7de3a2dadc in _lv_raid_rebuild_or_replace (lv=lv@entry=0x7f7dd4066188, remove_pvs=remove_pvs@entry=0x7f7dd406aca0, allocate_pvs=allocate_pvs@entry=0x7f7dd4065410, 
    rebuild=rebuild@entry=0, force=<optimized out>) at metadata/raid_manip.c:7027
#4  0x00007f7de3a38441 in lv_raid_replace (lv=lv@entry=0x7f7dd4066188, force=<optimized out>, remove_pvs=remove_pvs@entry=0x7f7dd406aca0, allocate_pvs=allocate_pvs@entry=0x7f7dd4065410)
    at metadata/raid_manip.c:7126
#5  0x00007f7de3a80ac7 in _lvconvert_repair_pvs_raid (handle=0x7f7dd4008cf0, use_pvh=0x7f7dd4065410, lv=0x7f7dd4066188, cmd=0x7f7ddc000cc0) at lvconvert.c:3773
#6  _lvconvert_repair_pvs (handle=0x7f7dd4008cf0, lv=0x7f7dd4066188, cmd=0x7f7ddc000cc0) at lvconvert.c:3806
#7  _lvconvert_repair_single (cmd=cmd@entry=0x7f7ddc000cc0, lv=0x7f7dd4066188, handle=handle@entry=0x7f7dd4008cf0) at lvconvert.c:3857
#8  0x00007f7de3aad7b0 in process_each_lv_in_vg (cmd=cmd@entry=0x7f7ddc000cc0, vg=vg@entry=0x7f7dd4065350, arg_lvnames=arg_lvnames@entry=0x7f7de6a7e510, tags_in=tags_in@entry=0x7f7de6a7e4c0, 
    stop_on_error=stop_on_error@entry=0, handle=handle@entry=0x7f7dd4008cf0, check_single_lv=0x0, process_single_lv=0x7f7de3a80370 <_lvconvert_repair_single>) at toollib.c:3148
#9  0x00007f7de3aae89b in _process_lv_vgnameid_list (process_single_lv=0x7f7de3a80370 <_lvconvert_repair_single>, check_single_lv=0x0, handle=0x7f7dd4008cf0, arg_tags=0x7f7de6a7e4c0, 
    arg_lvnames=0x7f7de6a7e4e0, arg_vgnames=0x7f7de6a7e4d0, vgnameids_to_process=0x7f7de6a7e500, read_flags=1048576, cmd=0x7f7ddc000cc0) at toollib.c:3616
#10 process_each_lv (cmd=cmd@entry=0x7f7ddc000cc0, argc=argc@entry=1, argv=<optimized out>, one_vgname=one_vgname@entry=0x0, one_lvname=one_lvname@entry=0x0, read_flags=read_flags@entry=1048576, 
    handle=0x7f7dd4008cf0, check_single_lv=<optimized out>, process_single_lv=<optimized out>) at toollib.c:3773
#11 0x00007f7de3a81000 in lvconvert_repair_cmd (cmd=0x7f7ddc000cc0, argc=<optimized out>, argv=<optimized out>) at lvconvert.c:3893
#12 0x00007f7de3a8e636 in lvm_run_command (cmd=cmd@entry=0x7f7ddc000cc0, argc=<optimized out>, argv=<optimized out>, argv@entry=0x7f7de6a7e850) at lvmcmdline.c:3132
#13 0x00007f7de3abe37e in lvm2_run (handle=0x7f7ddc000cc0, cmdline=<optimized out>, cmdline@entry=0x7f7dd4039b88 "lvconvert --repair --use-policies black_bird/synced_random_raid1_3legs_1")
    at lvmcmdlib.c:93
#14 0x00007f7de40634a7 in dmeventd_lvm2_run (cmdline=cmdline@entry=0x7f7dd4039b88 "lvconvert --repair --use-policies black_bird/synced_random_raid1_3legs_1") at dmeventd_lvm.c:126
#15 0x00007f7de26b2107 in _process_raid_event (device=0x7f7dd483a7b0 "black_bird-synced_random_raid1_3legs_1", params=<optimized out>, state=0x7f7dd4039b80) at dmeventd_raid.c:98
#16 process_event (dmt=0x55d9ac76bf40, event=<optimized out>, user=<optimized out>) at dmeventd_raid.c:141
#17 0x000055d9abd23b2a in _do_process_event (thread=0x55d9ac76da10) at dmeventd.c:954
#18 _monitor_thread (arg=0x55d9ac76da10) at dmeventd.c:1042
#19 0x00007f7de559714a in start_thread () from /lib64/libpthread.so.0
#20 0x00007f7de4e54db3 in clone () from /lib64/libc.so.6