Bug 1859794
| Summary: | Enable --syncaction check|repair on raid+integrity volumes | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Corey Marthaler <cmarthal> | |
| Component: | lvm2 | Assignee: | David Teigland <teigland> | |
| lvm2 sub component: | Mirroring and RAID | QA Contact: | cluster-qe <cluster-qe> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | medium | |||
| Priority: | high | CC: | agk, heinzm, jbrassow, mcsontos, msnitzer, pasik, prajnoha, teigland, zkabelac | |
| Version: | 8.3 | Keywords: | Triaged, ZStream | |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
|
| Target Release: | 8.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | lvm2-2.03.11-0.2.20201103git8801a86.el8 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1898639 (view as bug list) | Environment: | ||
| Last Closed: | 2021-05-18 15:01:53 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1898639 | |||
|
Description
Corey Marthaler
2020-07-23 03:08:21 UTC
pushed to master https://sourceware.org/git/?p=lvm2.git;a=commit;h=830c20d33cc8253c397e927291d51c249c2c0eb3 # lvcreate --type raid1 --raidintegrity y -m1 -L500M -n rr test Creating integrity metadata LV rr_rimage_0_imeta with size 12.00 MiB. Logical volume "rr_rimage_0_imeta" created. Creating integrity metadata LV rr_rimage_1_imeta with size 12.00 MiB. Logical volume "rr_rimage_1_imeta" created. Logical volume "rr" created. # lvs -a test -o+devices LV VG Attr LSize Origin Cpy%Sync Devices rr test rwi-a-r--- 500.00m 100.00 rr_rimage_0(0),rr_rimage_1(0) [rr_rimage_0] test gwi-aor--- 500.00m [rr_rimage_0_iorig] 100.00 rr_rimage_0_iorig(0) [rr_rimage_0_imeta] test ewi-ao---- 12.00m /dev/sdb(126) [rr_rimage_0_iorig] test -wi-ao---- 500.00m /dev/sdb(1) [rr_rimage_1] test gwi-aor--- 500.00m [rr_rimage_1_iorig] 100.00 rr_rimage_1_iorig(0) [rr_rimage_1_imeta] test ewi-ao---- 12.00m /dev/sdc(126) [rr_rimage_1_iorig] test -wi-ao---- 500.00m /dev/sdc(1) [rr_rmeta_0] test ewi-aor--- 4.00m /dev/sdb(0) [rr_rmeta_1] test ewi-aor--- 4.00m /dev/sdc(0) # lvchange -an test/rr # dd if=/dev/urandom of=/dev/sdc bs=1M seek=5 count=1 oflag=direct,sync 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395903 s, 26.5 MB/s # lvchange -ay test/rr # Note: simply activating the LV will usually detect and correct some mismatches # dmesg # Clear the mismatches that appeared from activation so we can detect more from syncaction # dmesg -C # lvs -o integritymismatches test/rr_rimage_0 IntegMismatches 0 # lvs -o integritymismatches test/rr_rimage_1 IntegMismatches 4 # lvchange --syncaction check test/rr # dmesg [1141450.097463] md: data-check of RAID array mdX [1141450.106140] device-mapper: integrity: dm-11: Checksum failed at sector 0x80 [1141450.113337] device-mapper: integrity: dm-11: Checksum failed at sector 0x100 [1141450.120615] device-mapper: integrity: dm-11: Checksum failed at sector 0x180 [1141450.127845] device-mapper: integrity: dm-11: Checksum failed at sector 0x200 [1141450.139019] device-mapper: integrity: dm-11: Checksum failed at sector 0x280 [1141450.146322] device-mapper: integrity: dm-11: Checksum failed at sector 0x300 [1141450.155495] device-mapper: integrity: dm-11: Checksum failed at sector 0x380 [1141450.162768] device-mapper: integrity: dm-11: Checksum failed at sector 0x400 [1141450.170055] device-mapper: integrity: dm-11: Checksum failed at sector 0x480 [1141450.177275] device-mapper: integrity: dm-11: Checksum failed at sector 0x500 [1141459.386917] md: mdX: data-check done. # lvs -o integritymismatches test/rr_rimage_0 IntegMismatches 0 # lvs -o integritymismatches test/rr_rimage_1 IntegMismatches 20 # Clear previous integrity checksum messages # dmesg -C # dd if=/dev/test/rr of=/dev/null bs=1M iflag=direct 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 5.17828 s, 101 MB/s # After reading the entire LV no further integrity messages appear # dmesg # and no further mismatches reported # lvs -o integritymismatches test/rr_rimage_0 IntegMismatches 0 # lvs -o integritymismatches test/rr_rimage_1 IntegMismatches 20 A few questions:
1. When used with integrity raid volumes, is "--syncaction repair" a no-op? It appears to only ever say "use check" instead.
2. What resets the IntegMismatches back to zero, if anything? It doesn't appear to be the "--syncaction check" op?
3. What does the a "--syncaction check" do? I originally thought it detected that Mismatch number, but then I tried that lvs cmd w/o running the check before and the mismatches appeared regardless. I did notice the "md: data-check of RAID array mdX" from the kernel like is expected w/ non integrity raid volumes, but I figured that check was special when run on integrity volumes?
[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
IntegMismatches
9264
[root@hayes-03 ~]# lvchange --syncaction check black_bird/synced_primary_raid1_2legs_1
[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
IntegMismatches
65673
[root@hayes-03 ~]# lvchange --syncaction repair black_bird/synced_primary_raid1_2legs_1
Use syncaction check to detect and correct integrity checksum mismatches.
[root@hayes-03 ~]# lvchange --syncaction check black_bird/synced_primary_raid1_2legs_1
[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
IntegMismatches
65673
Dec 3 14:08:26 hayes-03 lvm[2858]: raid1 array, black_bird-synced_primary_raid1_2legs_1, is now in-sync.
Dec 3 14:09:08 hayes-03 kernel: md: data-check of RAID array mdX
Dec 3 14:09:13 hayes-03 kernel: md: mdX: data-check done.
Dec 3 14:09:13 hayes-03 lvm[2858]: raid1 array, black_bird-synced_primary_raid1_2legs_1, is now in-sync.
Dec 3 14:09:39 hayes-03 kernel: md: data-check of RAID array mdX
Dec 3 14:09:43 hayes-03 kernel: md: mdX: data-check done.
Dec 3 14:09:44 hayes-03 lvm[2858]: raid1 array, black_bird-synced_primary_raid1_2legs_1, is now in-sync.
[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
IntegMismatches
65673
> 1. When used with integrity raid volumes, is "--syncaction repair" a no-op? > It appears to only ever say "use check" instead. Right, there's nothing to do for "repair" since "check" automatically triggers repair. > 2. What resets the IntegMismatches back to zero, if anything? It doesn't > appear to be the "--syncaction check" op? It should go to zero after reactivating. > 3. What does the a "--syncaction check" do? I originally thought it detected > that Mismatch number, but then I tried that lvs cmd w/o running the check > before and the mismatches appeared regardless. I did notice the "md: > data-check of RAID array mdX" from the kernel like is expected w/ non > integrity raid volumes, but I figured that check was special when run on > integrity volumes? I'm not intimately familiar with syncaction since it's strictly a raid operation, so I might need some correcting here... syncaction check is the same raid operation that has existed before. check causes raid to read and compare data on each image. Without an integrity layer, raid check will see differences in image data and respond by reporting it. With an integrity layer, raid will get an io error from one image (instead of the bad data), and respond by rewriting that block, which corrects the problem. After corrupting an image, you will often see integrity mismatches before you run syncaction check, just by activating the LV, or maybe running lvs. This is because dm-raid is reading some of the corrupted areas on the image itself, before you read anything from the LV. These automatic/internal reads trigger the integrity mismatches and corrections, so you'll often see non-zero mismatches before you check. The syncaction check will find everything, though, after which you should be able to read the entire LV without seeing any additional mismatches. Feature verified in the latest rpms.
The '--syncaction check' is now used in regression tests as one of the ways for integrity corruption detection and repair (using reported mismatches).
kernel-4.18.0-271.el8 BUILT: Fri Jan 8 03:32:43 CST 2021
lvm2-2.03.11-0.4.20201222gitb84a992.el8 BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-libs-2.03.11-0.4.20201222gitb84a992.el8 BUILT: Tue Dec 22 06:33:49 CST 2020
device-mapper-1.02.175-0.4.20201222gitb84a992.el8 BUILT: Tue Dec 22 06:33:49 CST 2020
device-mapper-libs-1.02.175-0.4.20201222gitb84a992.el8 BUILT: Tue Dec 22 06:33:49 CST 2020
device-mapper-event-1.02.175-0.4.20201222gitb84a992.el8 BUILT: Tue Dec 22 06:33:49 CST 2020
device-mapper-event-libs-1.02.175-0.4.20201222gitb84a992.el8 BUILT: Tue Dec 22 06:33:49 CST 2020
================================================================================
Iteration 1.5 started at Tue 12 Jan 2021 02:52:30 PM CST
================================================================================
Scenario kill_random_synced_raid1_3legs: Kill random leg of synced 3 leg raid1 volume(s)
********* RAID hash info for this scenario *********
* names: synced_random_raid1_3legs_1
* sync: 1
* type: raid1
* -m |-i value: 3
* leg devices: /dev/sdi /dev/sdg /dev/sdf /dev/sdo
* spanned legs: 0
* manual repair: 0
* no MDA devices:
* failpv(s): /dev/sdg
* additional snap: /dev/sdi
* failnode(s): hayes-03
* integrity stack: 1
(Due to integrity stack, be mindful of false failures that are reliant on message checks
that could be lost due to rate-limiting of corruption and other messages)
* raid fault policy: warn
******************************************************
Creating raids(s) on hayes-03...
hayes-03: lvcreate --yes --type raid1 -m 3 -n synced_random_raid1_3legs_1 -L 500M black_bird /dev/sdi:0-2400 /dev/sdg:0-2400 /dev/sdf:0-2400 /dev/sdo:0-2400
Current mirror/raid device structure(s):
LV Attr LSize Cpy%Sync Devices
synced_random_raid1_3legs_1 rwi-a-r--- 500.00m 31.31 synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
[synced_random_raid1_3legs_1_rimage_0] Iwi-aor--- 500.00m /dev/sdi(1)
[synced_random_raid1_3legs_1_rimage_1] Iwi-aor--- 500.00m /dev/sdg(1)
[synced_random_raid1_3legs_1_rimage_2] Iwi-aor--- 500.00m /dev/sdf(1)
[synced_random_raid1_3legs_1_rimage_3] Iwi-aor--- 500.00m /dev/sdo(1)
[synced_random_raid1_3legs_1_rmeta_0] ewi-aor--- 4.00m /dev/sdi(0)
[synced_random_raid1_3legs_1_rmeta_1] ewi-aor--- 4.00m /dev/sdg(0)
[synced_random_raid1_3legs_1_rmeta_2] ewi-aor--- 4.00m /dev/sdf(0)
[synced_random_raid1_3legs_1_rmeta_3] ewi-aor--- 4.00m /dev/sdo(0)
Waiting until all mirror|raid volumes become fully syncd...
1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
Convert mirror/raid volume(s) to utilize integrity target volume(s) on hayes-03...
lvconvert --yes --raidintegrity y --raidintegritymode bitmap black_bird/synced_random_raid1_3legs_1
Creating xfs on top of mirror(s) on hayes-03...
Mounting mirrored xfs filesystems on hayes-03...
Current mirror/raid device structure(s):
LV Attr LSize Cpy%Sync Devices
synced_random_raid1_3legs_1 rwi-aor--- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
[synced_random_raid1_3legs_1_rimage_0] gwi-aor--- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_0_iorig(0)
[synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao---- 12.00m /dev/sdi(126)
[synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m /dev/sdi(1)
[synced_random_raid1_3legs_1_rimage_1] gwi-aor--- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_1_iorig(0)
[synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao---- 12.00m /dev/sdg(126)
[synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao---- 500.00m /dev/sdg(1)
[synced_random_raid1_3legs_1_rimage_2] gwi-aor--- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_2_iorig(0)
[synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao---- 12.00m /dev/sdf(126)
[synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m /dev/sdf(1)
[synced_random_raid1_3legs_1_rimage_3] gwi-aor--- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_3_iorig(0)
[synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao---- 12.00m /dev/sdo(126)
[synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m /dev/sdo(1)
[synced_random_raid1_3legs_1_rmeta_0] ewi-aor--- 4.00m /dev/sdi(0)
[synced_random_raid1_3legs_1_rmeta_1] ewi-aor--- 4.00m /dev/sdg(0)
[synced_random_raid1_3legs_1_rmeta_2] ewi-aor--- 4.00m /dev/sdf(0)
[synced_random_raid1_3legs_1_rmeta_3] ewi-aor--- 4.00m /dev/sdo(0)
PV=/dev/sdg
synced_random_raid1_3legs_1_rimage_1_imeta: 2
synced_random_raid1_3legs_1_rimage_1_iorig: 2
synced_random_raid1_3legs_1_rmeta_1: 2
Writing verification files (checkit) to mirror(s) on...
---- hayes-03 ----
Sleeping 15 seconds to get some outsanding I/O locks before the failure
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
**** Raid Integrity Corruption info for this verification *****
* Current legs: /dev/sdi /dev/sdg /dev/sdf /dev/sdo
* Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
* PV to corrupt /dev/sdi
* READ (non span): lvchange --writemostly /dev/sdi:n black_bird
* WRITE (non span): lvchange --writemostly /dev/sdg:y black_bird
* WRITE (non span): lvchange --writemostly /dev/sdf:y black_bird
* WRITE (non span): lvchange --writemostly /dev/sdo:y black_bird
* (Clearing out OLD dmesg corruption detection notifications)
***************************************************************
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
lvchange -an black_bird/synced_random_raid1_3legs_1
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdi oflag=direct,sync bs=1M seek=20 count=32
32+0 records in
32+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.528359 s, 63.5 MB/s
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1
Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1
lvs --noheadings -o integritymismatches black_bird/synced_random_raid1_3legs_1 | awk {'print $1'}
Corruption mismatches reported: 512
Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
Corruption mismatches reported: 0
re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1
** NO sign of integrity correction found!! **
Corruption mismatches reported: 0
Disabling device sdg on hayes-03rescan device...
Attempting I/O to cause mirror down conversion(s) on hayes-03
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0198197 s, 2.1 GB/s
Verifying current sanity of lvm after the failure
Current mirror/raid device structure(s):
WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to /dev/sdg).
LV Attr LSize Cpy%Sync Devices
synced_random_raid1_3legs_1 rwi-aor-p- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
[synced_random_raid1_3legs_1_rimage_0] gwi-aor--- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_0_iorig(0)
[synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao---- 12.00m /dev/sdi(126)
[synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m /dev/sdi(1)
[synced_random_raid1_3legs_1_rimage_1] gwi-aor-p- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_1_iorig(0)
[synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao---- 12.00m [unknown](126)
[synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao--p- 500.00m [unknown](1)
[synced_random_raid1_3legs_1_rimage_2] gwi-aor-w- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_2_iorig(0)
[synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao---- 12.00m /dev/sdf(126)
[synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m /dev/sdf(1)
[synced_random_raid1_3legs_1_rimage_3] gwi-aor-w- 500.00m 100.00 synced_random_raid1_3legs_1_rimage_3_iorig(0)
[synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao---- 12.00m /dev/sdo(126)
[synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m /dev/sdo(1)
[synced_random_raid1_3legs_1_rmeta_0] ewi-aor--- 4.00m /dev/sdi(0)
[synced_random_raid1_3legs_1_rmeta_1] ewi-aor-p- 4.00m [unknown](0)
[synced_random_raid1_3legs_1_rmeta_2] ewi-aor--- 4.00m /dev/sdf(0)
[synced_random_raid1_3legs_1_rmeta_3] ewi-aor--- 4.00m /dev/sdo(0)
Verifying FAILED device /dev/sdg is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdi *IS* in the volume(s)
Verifying IMAGE device /dev/sdf *IS* in the volume(s)
Verifying IMAGE device /dev/sdo *IS* in the volume(s)
Verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_1_imeta on: hayes-03
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_1_iorig on: hayes-03
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rmeta_1 on: hayes-03
Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: /dev/sdi unknown /dev/sdf /dev/sdo
Found raid with dm integrity devices
ACTUAL LEG ORDER: /dev/sdi [unknown] /dev/sdf /dev/sdo
/dev/sdi ne /dev/sdi
unknown ne [unknown]
/dev/sdf ne /dev/sdf
/dev/sdo ne /dev/sdo
Fault policy is warn... Manually repairing failed raid volumes
Waiting until all mirror|raid volumes become fully syncd...
1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
hayes-03: 'lvconvert --yes --repair black_bird/synced_random_raid1_3legs_1'
Waiting until all mirror|raid volumes become fully syncd...
1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
Updating VG metadata post device failure
hayes-03: vgck --updatemetadata black_bird
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
**** Raid Integrity Corruption info for this verification *****
* Current legs: /dev/sdi /dev/sdm /dev/sdf /dev/sdo
* Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
* PV to corrupt /dev/sdi
* READ (non span): lvchange --writemostly /dev/sdi:n black_bird
* WRITE (non span): lvchange --writemostly /dev/sdm:y black_bird
* WRITE (non span): lvchange --writemostly /dev/sdf:y black_bird
* WRITE (non span): lvchange --writemostly /dev/sdo:y black_bird
* (Clearing out OLD dmesg corruption detection notifications)
***************************************************************
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
lvchange -an black_bird/synced_random_raid1_3legs_1
WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to [unknown]).
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdi oflag=direct,sync bs=1M seek=18 count=13
13+0 records in
13+0 records out
13631488 bytes (14 MB, 13 MiB) copied, 0.221666 s, 61.5 MB/s
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1
Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1
WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to [unknown]).
Cannot change VG black_bird while PVs are missing.
See vgreduce --removemissing and vgextend --restoremissing.
Cannot process volume group black_bird
check op failed, likely due to inconsistent VG after image failure
vgreduce --removemissing black_bird
WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to [unknown]).
WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
Second attempt: lvchange --syncaction check black_bird/synced_random_raid1_3legs_1
Corruption mismatches reported: 208
Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
Corruption mismatches reported: 0
re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1
Enabling device sdg on hayes-03 Running vgs to make LVM update metadata version if possible (will restore a-m PVs)
WARNING: ignoring metadata seqno 371 on /dev/sdg for seqno 382 on /dev/sdf for VG black_bird.
WARNING: Inconsistent metadata found for VG black_bird.
See vgck --updatemetadata to correct inconsistency.
WARNING: outdated PV /dev/sdg seqno 371 has been removed in current VG black_bird seqno 382.
See vgck --updatemetadata to clear outdated metadata.
Updating VG metadata post device failure
hayes-03: vgck --updatemetadata black_bird
WARNING: ignoring metadata seqno 371 on /dev/sdg for seqno 382 on /dev/sdf for VG black_bird.
WARNING: Inconsistent metadata found for VG black_bird.
See vgck --updatemetadata to correct inconsistency.
WARNING: outdated PV /dev/sdg seqno 371 has been removed in current VG black_bird seqno 382.
See vgck --updatemetadata to clear outdated metadata.
WARNING: wiping mda on outdated PV /dev/sdg
WARNING: wiping header on outdated PV /dev/sdg
Checking for leftover '-missing_0_0' or 'unknown devices'
Checking for PVs marked as missing (a-m)...
Check for the transient tag, and if present pass transient arg to hs_io_verify
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
**** Raid Integrity Corruption info for this verification *****
* Current legs: /dev/sdi /dev/sdm /dev/sdf /dev/sdo
* Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
* PV to corrupt /dev/sdi
* READ (non span): lvchange --writemostly /dev/sdi:n black_bird
* WRITE (non span): lvchange --writemostly /dev/sdm:y black_bird
* WRITE (non span): lvchange --writemostly /dev/sdf:y black_bird
* WRITE (non span): lvchange --writemostly /dev/sdo:y black_bird
* (Clearing out OLD dmesg corruption detection notifications)
***************************************************************
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
lvchange -an black_bird/synced_random_raid1_3legs_1
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdi oflag=direct,sync bs=1M seek=19 count=39
39+0 records in
39+0 records out
40894464 bytes (41 MB, 39 MiB) copied, 0.637501 s, 64.1 MB/s
Verifying files (checkit) on mirror(s) on...
---- hayes-03 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1
Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1
Corruption mismatches reported: 624
Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
Corruption mismatches reported: 0
re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1
Unmounting xfs and removing mnt point on hayes-03...
Deactivating and removing raid(s)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1659 |