RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1859794 - Enable --syncaction check|repair on raid+integrity volumes
Summary: Enable --syncaction check|repair on raid+integrity volumes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.3
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1898639
TreeView+ depends on / blocked
 
Reported: 2020-07-23 03:08 UTC by Corey Marthaler
Modified: 2023-09-04 06:52 UTC (History)
9 users (show)

Fixed In Version: lvm2-2.03.11-0.2.20201103git8801a86.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1898639 (view as bug list)
Environment:
Last Closed: 2021-05-18 15:01:53 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Corey Marthaler 2020-07-23 03:08:21 UTC 
Description of problem:
It's been brought up a few times to attempt "lvchange --syncaction check|repair" on raid integrity volumes for corruption detection. I'm filing this RFE to document that it doesn't currently work.


[root@hayes-02 ~]# lvs -a -o +segtype
  LV                                           VG         Attr       LSize   Origin                                        Cpy%Sync Type         
  synced_random_raid1_2legs_1                  black_bird rwi-aor--- 500.00m                                               100.00   raid1        
  [synced_random_raid1_2legs_1_rimage_0]       black_bird gwi-aor--- 500.00m [synced_random_raid1_2legs_1_rimage_0_iorig]  100.00   integrity
  [synced_random_raid1_2legs_1_rimage_0_imeta] black_bird ewi-ao----   8.00m                                                        linear   
  [synced_random_raid1_2legs_1_rimage_0_iorig] black_bird -wi-ao---- 500.00m                                                        linear   
  [synced_random_raid1_2legs_1_rimage_1]       black_bird gwi-aor--- 500.00m [synced_random_raid1_2legs_1_rimage_1_iorig]  100.00   integrity
  [synced_random_raid1_2legs_1_rimage_1_imeta] black_bird ewi-ao----   8.00m                                                        linear   
  [synced_random_raid1_2legs_1_rimage_1_iorig] black_bird -wi-ao---- 500.00m                                                        linear   
  [synced_random_raid1_2legs_1_rimage_2]       black_bird gwi-aor--- 500.00m [synced_random_raid1_2legs_1_rimage_2_iorig]  100.00   integrity
  [synced_random_raid1_2legs_1_rimage_2_imeta] black_bird ewi-ao----   8.00m                                                        linear   
  [synced_random_raid1_2legs_1_rimage_2_iorig] black_bird -wi-ao---- 500.00m                                                        linear   
  [synced_random_raid1_2legs_1_rmeta_0]        black_bird ewi-aor---   4.00m                                                        linear   
  [synced_random_raid1_2legs_1_rmeta_1]        black_bird ewi-aor---   4.00m                                                        linear   
  [synced_random_raid1_2legs_1_rmeta_2]        black_bird ewi-aor---   4.00m                                                        linear   

  synced_random_raid1_2legs_2                  black_bird rwi-aor--- 500.00m                                               100.00   raid1       
  [synced_random_raid1_2legs_2_rimage_0]       black_bird gwi-aor--- 500.00m [synced_random_raid1_2legs_2_rimage_0_iorig]  100.00   integrity
  [synced_random_raid1_2legs_2_rimage_0_imeta] black_bird ewi-ao----   8.00m                                                        linear   
  [synced_random_raid1_2legs_2_rimage_0_iorig] black_bird -wi-ao---- 500.00m                                                        linear   
  [synced_random_raid1_2legs_2_rimage_1]       black_bird gwi-aor--- 500.00m [synced_random_raid1_2legs_2_rimage_1_iorig]  100.00   integrity
  [synced_random_raid1_2legs_2_rimage_1_imeta] black_bird ewi-ao----   8.00m                                                        linear   
  [synced_random_raid1_2legs_2_rimage_1_iorig] black_bird -wi-ao---- 500.00m                                                        linear   
  [synced_random_raid1_2legs_2_rimage_2]       black_bird gwi-aor--- 500.00m [synced_random_raid1_2legs_2_rimage_2_iorig]  100.00   integrity
  [synced_random_raid1_2legs_2_rimage_2_imeta] black_bird ewi-ao----   8.00m                                                        linear   
  [synced_random_raid1_2legs_2_rimage_2_iorig] black_bird -wi-ao---- 500.00m                                                        linear   
  [synced_random_raid1_2legs_2_rmeta_0]        black_bird ewi-aor---   4.00m                                                        linear   
  [synced_random_raid1_2legs_2_rmeta_1]        black_bird ewi-aor---   4.00m                                                        linear   
  [synced_random_raid1_2legs_2_rmeta_2]        black_bird ewi-aor---   4.00m                                                        linear   

[root@hayes-02 ~]# lvchange --syncaction check black_bird/synced_random_raid1_2legs_1
  Integrity must be removed to use syncaction commands.
[root@hayes-02 ~]# lvchange --syncaction repair black_bird/synced_random_raid1_2legs_2
  Integrity must be removed to use syncaction commands.


Version-Release number of selected component (if applicable):
kernel-4.18.0-211.el8    BUILT: Thu Jun  4 03:33:39 CDT 2020
lvm2-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
lvm2-libs-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
lvm2-dbusd-2.03.09-3.el8    BUILT: Mon Jun 29 13:53:38 CDT 2020
lvm2-lockd-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-libs-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-event-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-event-libs-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
Comment 5 David Teigland 2020-10-26 19:37:37 UTC 
pushed to master
https://sourceware.org/git/?p=lvm2.git;a=commit;h=830c20d33cc8253c397e927291d51c249c2c0eb3


# lvcreate --type raid1 --raidintegrity y -m1 -L500M -n rr test
  Creating integrity metadata LV rr_rimage_0_imeta with size 12.00 MiB.
  Logical volume "rr_rimage_0_imeta" created.
  Creating integrity metadata LV rr_rimage_1_imeta with size 12.00 MiB.
  Logical volume "rr_rimage_1_imeta" created.
  Logical volume "rr" created.

# lvs -a test -o+devices
  LV                  VG   Attr       LSize   Origin              Cpy%Sync Devices                      
  rr                  test rwi-a-r--- 500.00m                     100.00   rr_rimage_0(0),rr_rimage_1(0)
  [rr_rimage_0]       test gwi-aor--- 500.00m [rr_rimage_0_iorig] 100.00   rr_rimage_0_iorig(0)         
  [rr_rimage_0_imeta] test ewi-ao----  12.00m                              /dev/sdb(126)                
  [rr_rimage_0_iorig] test -wi-ao---- 500.00m                              /dev/sdb(1)                  
  [rr_rimage_1]       test gwi-aor--- 500.00m [rr_rimage_1_iorig] 100.00   rr_rimage_1_iorig(0)         
  [rr_rimage_1_imeta] test ewi-ao----  12.00m                              /dev/sdc(126)                
  [rr_rimage_1_iorig] test -wi-ao---- 500.00m                              /dev/sdc(1)                  
  [rr_rmeta_0]        test ewi-aor---   4.00m                              /dev/sdb(0)                  
  [rr_rmeta_1]        test ewi-aor---   4.00m                              /dev/sdc(0)           

# lvchange -an test/rr

# dd if=/dev/urandom of=/dev/sdc bs=1M seek=5 count=1 oflag=direct,sync
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395903 s, 26.5 MB/s

# lvchange -ay test/rr

# Note: simply activating the LV will usually detect and correct some mismatches
# dmesg
# Clear the mismatches that appeared from activation so we can detect more from syncaction
# dmesg -C

# lvs -o integritymismatches test/rr_rimage_0
  IntegMismatches
                0
# lvs -o integritymismatches test/rr_rimage_1
  IntegMismatches
                4

# lvchange --syncaction check test/rr

# dmesg
[1141450.097463] md: data-check of RAID array mdX
[1141450.106140] device-mapper: integrity: dm-11: Checksum failed at sector 0x80
[1141450.113337] device-mapper: integrity: dm-11: Checksum failed at sector 0x100
[1141450.120615] device-mapper: integrity: dm-11: Checksum failed at sector 0x180
[1141450.127845] device-mapper: integrity: dm-11: Checksum failed at sector 0x200
[1141450.139019] device-mapper: integrity: dm-11: Checksum failed at sector 0x280
[1141450.146322] device-mapper: integrity: dm-11: Checksum failed at sector 0x300
[1141450.155495] device-mapper: integrity: dm-11: Checksum failed at sector 0x380
[1141450.162768] device-mapper: integrity: dm-11: Checksum failed at sector 0x400
[1141450.170055] device-mapper: integrity: dm-11: Checksum failed at sector 0x480
[1141450.177275] device-mapper: integrity: dm-11: Checksum failed at sector 0x500
[1141459.386917] md: mdX: data-check done.

# lvs -o integritymismatches test/rr_rimage_0
  IntegMismatches
                0
# lvs -o integritymismatches test/rr_rimage_1
  IntegMismatches
               20

# Clear previous integrity checksum messages
# dmesg -C


# dd if=/dev/test/rr of=/dev/null bs=1M iflag=direct
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 5.17828 s, 101 MB/s

# After reading the entire LV no further integrity messages appear
# dmesg

# and no further mismatches reported
# lvs -o integritymismatches test/rr_rimage_0
  IntegMismatches
                0
# lvs -o integritymismatches test/rr_rimage_1
  IntegMismatches
               20
Comment 11 Corey Marthaler 2020-12-03 20:30:25 UTC 
A few questions:

1. When used with integrity raid volumes, is "--syncaction repair" a no-op? It appears to only ever say "use check" instead.
2. What resets the IntegMismatches back to zero, if anything? It doesn't appear to be the "--syncaction check" op?
3. What does the a "--syncaction check" do? I originally thought it detected that Mismatch number, but then I tried that lvs cmd w/o running the check before and the mismatches appeared regardless. I did notice the "md: data-check of RAID array mdX" from the kernel like is expected w/ non integrity raid volumes, but I figured that check was special when run on integrity volumes?


[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
  IntegMismatches
             9264


[root@hayes-03 ~]# lvchange --syncaction check black_bird/synced_primary_raid1_2legs_1
[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
  IntegMismatches
            65673

[root@hayes-03 ~]# lvchange --syncaction repair black_bird/synced_primary_raid1_2legs_1
  Use syncaction check to detect and correct integrity checksum mismatches.

[root@hayes-03 ~]# lvchange --syncaction check black_bird/synced_primary_raid1_2legs_1


[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
  IntegMismatches
            65673

Dec  3 14:08:26 hayes-03 lvm[2858]: raid1 array, black_bird-synced_primary_raid1_2legs_1, is now in-sync.
Dec  3 14:09:08 hayes-03 kernel: md: data-check of RAID array mdX
Dec  3 14:09:13 hayes-03 kernel: md: mdX: data-check done.
Dec  3 14:09:13 hayes-03 lvm[2858]: raid1 array, black_bird-synced_primary_raid1_2legs_1, is now in-sync.
Dec  3 14:09:39 hayes-03 kernel: md: data-check of RAID array mdX
Dec  3 14:09:43 hayes-03 kernel: md: mdX: data-check done.
Dec  3 14:09:44 hayes-03 lvm[2858]: raid1 array, black_bird-synced_primary_raid1_2legs_1, is now in-sync.


[root@hayes-03 ~]# lvs -o integritymismatches black_bird/synced_primary_raid1_2legs_1_rimage_1
  IntegMismatches
            65673
Comment 12 David Teigland 2020-12-03 22:06:32 UTC 
> 1. When used with integrity raid volumes, is "--syncaction repair" a no-op?
> It appears to only ever say "use check" instead.

Right, there's nothing to do for "repair" since "check" automatically triggers repair.

> 2. What resets the IntegMismatches back to zero, if anything? It doesn't
> appear to be the "--syncaction check" op?

It should go to zero after reactivating.

> 3. What does the a "--syncaction check" do? I originally thought it detected
> that Mismatch number, but then I tried that lvs cmd w/o running the check
> before and the mismatches appeared regardless. I did notice the "md:
> data-check of RAID array mdX" from the kernel like is expected w/ non
> integrity raid volumes, but I figured that check was special when run on
> integrity volumes?

I'm not intimately familiar with syncaction since it's strictly a raid operation, so I might need some correcting here... syncaction check is the same raid operation that has existed before.  check causes raid to read and compare data on each image.  Without an integrity layer, raid check will see differences in image data and respond by reporting it.  With an integrity layer, raid will get an io error from one image (instead of the bad data), and respond by rewriting that block, which corrects the problem.

After corrupting an image, you will often see integrity mismatches before you run syncaction check, just by activating the LV, or maybe running lvs.  This is because dm-raid is reading some of the corrupted areas on the image itself, before you read anything from the LV.  These automatic/internal reads trigger the integrity mismatches and corrections, so you'll often see non-zero mismatches before you check.  The syncaction check will find everything, though, after which you should be able to read the entire LV without seeing any additional mismatches.
Comment 13 Corey Marthaler 2021-01-12 22:30:33 UTC 
Feature verified in the latest rpms. 

The '--syncaction check' is now used in regression tests as one of the ways for integrity corruption detection and repair (using reported mismatches).

kernel-4.18.0-271.el8    BUILT: Fri Jan  8 03:32:43 CST 2021
lvm2-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-libs-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020

device-mapper-1.02.175-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
device-mapper-libs-1.02.175-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
device-mapper-event-1.02.175-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
device-mapper-event-libs-1.02.175-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020


================================================================================
Iteration 1.5 started at Tue 12 Jan 2021 02:52:30 PM CST
================================================================================
Scenario kill_random_synced_raid1_3legs: Kill random leg of synced 3 leg raid1 volume(s)
********* RAID hash info for this scenario *********
* names:              synced_random_raid1_3legs_1
* sync:               1
* type:               raid1
* -m |-i value:       3
* leg devices:        /dev/sdi /dev/sdg /dev/sdf /dev/sdo
* spanned legs:       0
* manual repair:      0
* no MDA devices:     
* failpv(s):          /dev/sdg
* additional snap:    /dev/sdi
* failnode(s):        hayes-03
* integrity stack:    1
 (Due to integrity stack, be mindful of false failures that are reliant on message checks
  that could be lost due to rate-limiting of corruption and other messages)
* raid fault policy:  warn
******************************************************

Creating raids(s) on hayes-03...
hayes-03: lvcreate --yes  --type raid1 -m 3 -n synced_random_raid1_3legs_1 -L 500M black_bird /dev/sdi:0-2400 /dev/sdg:0-2400 /dev/sdf:0-2400 /dev/sdo:0-2400

Current mirror/raid device structure(s):
  LV                                     Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1            rwi-a-r--- 500.00m 31.31    synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0] Iwi-aor--- 500.00m          /dev/sdi(1)
   [synced_random_raid1_3legs_1_rimage_1] Iwi-aor--- 500.00m          /dev/sdg(1)
   [synced_random_raid1_3legs_1_rimage_2] Iwi-aor--- 500.00m          /dev/sdf(1)
   [synced_random_raid1_3legs_1_rimage_3] Iwi-aor--- 500.00m          /dev/sdo(1)
   [synced_random_raid1_3legs_1_rmeta_0]  ewi-aor---   4.00m          /dev/sdi(0)
   [synced_random_raid1_3legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sdg(0)
   [synced_random_raid1_3legs_1_rmeta_2]  ewi-aor---   4.00m          /dev/sdf(0)
   [synced_random_raid1_3legs_1_rmeta_3]  ewi-aor---   4.00m          /dev/sdo(0)


Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Convert mirror/raid volume(s) to utilize integrity target volume(s) on hayes-03...

lvconvert --yes --raidintegrity y --raidintegritymode bitmap black_bird/synced_random_raid1_3legs_1
Creating xfs on top of mirror(s) on hayes-03...
Mounting mirrored xfs filesystems on hayes-03...

Current mirror/raid device structure(s):
  LV                                           Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                  rwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdi(126)
   [synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdi(1)
   [synced_random_raid1_3legs_1_rimage_1]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao----  12.00m          /dev/sdg(126)
   [synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao---- 500.00m          /dev/sdg(1)
   [synced_random_raid1_3legs_1_rimage_2]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdf(126)
   [synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdf(1)
   [synced_random_raid1_3legs_1_rimage_3]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao----  12.00m          /dev/sdo(126)
   [synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sdo(1)
   [synced_random_raid1_3legs_1_rmeta_0]        ewi-aor---   4.00m          /dev/sdi(0)
   [synced_random_raid1_3legs_1_rmeta_1]        ewi-aor---   4.00m          /dev/sdg(0)
   [synced_random_raid1_3legs_1_rmeta_2]        ewi-aor---   4.00m          /dev/sdf(0)
   [synced_random_raid1_3legs_1_rmeta_3]        ewi-aor---   4.00m          /dev/sdo(0)

PV=/dev/sdg
        synced_random_raid1_3legs_1_rimage_1_imeta: 2
        synced_random_raid1_3legs_1_rimage_1_iorig: 2
        synced_random_raid1_3legs_1_rmeta_1: 2

Writing verification files (checkit) to mirror(s) on...
        ---- hayes-03 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 

Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

**** Raid Integrity Corruption info for this verification *****
*  Current legs: /dev/sdi /dev/sdg /dev/sdf /dev/sdo
*  Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
*  PV to corrupt /dev/sdi
*  READ (non span): lvchange --writemostly /dev/sdi:n black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdg:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdf:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdo:y black_bird
*  (Clearing out OLD dmesg corruption detection notifications)
***************************************************************

Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

lvchange -an black_bird/synced_random_raid1_3legs_1
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdi oflag=direct,sync  bs=1M seek=20 count=32
32+0 records in
32+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.528359 s, 63.5 MB/s
Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1

Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1

lvs --noheadings -o integritymismatches black_bird/synced_random_raid1_3legs_1 | awk {'print $1'}
Corruption mismatches reported: 512

Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
Corruption mismatches reported: 0

re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1

** NO sign of integrity correction found!! ** 
Corruption mismatches reported: 0

Disabling device sdg on hayes-03rescan device...

Attempting I/O to cause mirror down conversion(s) on hayes-03
dd if=/dev/zero of=/mnt/synced_random_raid1_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0198197 s, 2.1 GB/s

Verifying current sanity of lvm after the failure

Current mirror/raid device structure(s):
  WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
  WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to /dev/sdg).
  LV                                           Attr       LSize   Cpy%Sync Devices
   synced_random_raid1_3legs_1                  rwi-aor-p- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0(0),synced_random_raid1_3legs_1_rimage_1(0),synced_random_raid1_3legs_1_rimage_2(0),synced_random_raid1_3legs_1_rimage_3(0)
   [synced_random_raid1_3legs_1_rimage_0]       gwi-aor--- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_0_iorig(0)
   [synced_random_raid1_3legs_1_rimage_0_imeta] ewi-ao----  12.00m          /dev/sdi(126)
   [synced_random_raid1_3legs_1_rimage_0_iorig] -wi-ao---- 500.00m          /dev/sdi(1)
   [synced_random_raid1_3legs_1_rimage_1]       gwi-aor-p- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_1_iorig(0)
   [synced_random_raid1_3legs_1_rimage_1_imeta] ewi-ao----  12.00m          [unknown](126)
   [synced_random_raid1_3legs_1_rimage_1_iorig] -wi-ao--p- 500.00m          [unknown](1)
   [synced_random_raid1_3legs_1_rimage_2]       gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_2_iorig(0)
   [synced_random_raid1_3legs_1_rimage_2_imeta] ewi-ao----  12.00m          /dev/sdf(126)
   [synced_random_raid1_3legs_1_rimage_2_iorig] -wi-ao---- 500.00m          /dev/sdf(1)
   [synced_random_raid1_3legs_1_rimage_3]       gwi-aor-w- 500.00m 100.00   synced_random_raid1_3legs_1_rimage_3_iorig(0)
   [synced_random_raid1_3legs_1_rimage_3_imeta] ewi-ao----  12.00m          /dev/sdo(126)
   [synced_random_raid1_3legs_1_rimage_3_iorig] -wi-ao---- 500.00m          /dev/sdo(1)
   [synced_random_raid1_3legs_1_rmeta_0]        ewi-aor---   4.00m          /dev/sdi(0)
   [synced_random_raid1_3legs_1_rmeta_1]        ewi-aor-p-   4.00m          [unknown](0)
   [synced_random_raid1_3legs_1_rmeta_2]        ewi-aor---   4.00m          /dev/sdf(0)
   [synced_random_raid1_3legs_1_rmeta_3]        ewi-aor---   4.00m          /dev/sdo(0)


Verifying FAILED device /dev/sdg is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdi *IS* in the volume(s)
Verifying IMAGE device /dev/sdf *IS* in the volume(s)
Verifying IMAGE device /dev/sdo *IS* in the volume(s)
Verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_1_imeta on: hayes-03 
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rimage_1_iorig on: hayes-03 
Checking EXISTENCE and STATE of synced_random_raid1_3legs_1_rmeta_1 on: hayes-03 

Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: /dev/sdi unknown /dev/sdf /dev/sdo
Found raid with dm integrity devices
ACTUAL LEG ORDER: /dev/sdi [unknown] /dev/sdf /dev/sdo
/dev/sdi ne /dev/sdi
unknown ne [unknown]
/dev/sdf ne /dev/sdf
/dev/sdo ne /dev/sdo

Fault policy is warn... Manually repairing failed raid volumes
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
hayes-03: 'lvconvert --yes --repair black_bird/synced_random_raid1_3legs_1'

Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Updating VG metadata post device failure
hayes-03: vgck --updatemetadata black_bird
Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

**** Raid Integrity Corruption info for this verification *****
*  Current legs: /dev/sdi /dev/sdm /dev/sdf /dev/sdo
*  Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
*  PV to corrupt /dev/sdi
*  READ (non span): lvchange --writemostly /dev/sdi:n black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdm:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdf:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdo:y black_bird
*  (Clearing out OLD dmesg corruption detection notifications)
***************************************************************

Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

lvchange -an black_bird/synced_random_raid1_3legs_1
  WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
  WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to [unknown]).
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdi oflag=direct,sync  bs=1M seek=18 count=13
13+0 records in
13+0 records out
13631488 bytes (14 MB, 13 MiB) copied, 0.221666 s, 61.5 MB/s
Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----
lvchange -ay black_bird/synced_random_raid1_3legs_1

Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1
  WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
  WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to [unknown]).
  Cannot change VG black_bird while PVs are missing.
  See vgreduce --removemissing and vgextend --restoremissing.
  Cannot process volume group black_bird
check op failed, likely due to inconsistent VG after image failure
vgreduce --removemissing black_bird
  WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
  WARNING: VG black_bird is missing PV 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D (last written to [unknown]).
  WARNING: Couldn't find device with uuid 9p7Qh8-E9Rd-Q7zB-U7hi-DqV1-xji8-K99k6D.
Second attempt: lvchange --syncaction check black_bird/synced_random_raid1_3legs_1

Corruption mismatches reported: 208
Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
Corruption mismatches reported: 0

re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1


Enabling device sdg on hayes-03 Running vgs to make LVM update metadata version if possible (will restore a-m PVs)
  WARNING: ignoring metadata seqno 371 on /dev/sdg for seqno 382 on /dev/sdf for VG black_bird.
  WARNING: Inconsistent metadata found for VG black_bird.
  See vgck --updatemetadata to correct inconsistency.
  WARNING: outdated PV /dev/sdg seqno 371 has been removed in current VG black_bird seqno 382.
  See vgck --updatemetadata to clear outdated metadata.

Updating VG metadata post device failure
hayes-03: vgck --updatemetadata black_bird
  WARNING: ignoring metadata seqno 371 on /dev/sdg for seqno 382 on /dev/sdf for VG black_bird.
  WARNING: Inconsistent metadata found for VG black_bird.
  See vgck --updatemetadata to correct inconsistency.
  WARNING: outdated PV /dev/sdg seqno 371 has been removed in current VG black_bird seqno 382.
  See vgck --updatemetadata to clear outdated metadata.
  WARNING: wiping mda on outdated PV /dev/sdg
  WARNING: wiping header on outdated PV /dev/sdg
Checking for leftover '-missing_0_0' or 'unknown devices'
Checking for PVs marked as missing (a-m)...

Check for the transient tag, and if present pass transient arg to hs_io_verify
Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

**** Raid Integrity Corruption info for this verification *****
*  Current legs: /dev/sdi /dev/sdm /dev/sdf /dev/sdo
*  Image(s) to corrupt synced_random_raid1_3legs_1_rimage_0_iorig
*  PV to corrupt /dev/sdi
*  READ (non span): lvchange --writemostly /dev/sdi:n black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdm:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdf:y black_bird
*  WRITE (non span): lvchange --writemostly /dev/sdo:y black_bird
*  (Clearing out OLD dmesg corruption detection notifications)
***************************************************************

Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

lvchange -an black_bird/synced_random_raid1_3legs_1
* Corrupting an integrity image's PV
WRITE: dd if=/dev/urandom of=/dev/sdi oflag=direct,sync  bs=1M seek=19 count=39
39+0 records in
39+0 records out
40894464 bytes (41 MB, 39 MiB) copied, 0.637501 s, 64.1 MB/s
Verifying files (checkit) on mirror(s) on...
        ---- hayes-03 ----

lvchange -ay black_bird/synced_random_raid1_3legs_1
Detecting corruption on bad image one of two ways:
lvchange --syncaction check black_bird/synced_random_raid1_3legs_1

Corruption mismatches reported: 624

Cycling the activation to reset the mismatch counter (lvchange -a n|y black_bird/synced_random_raid1_3legs_1)
Corruption mismatches reported: 0

re-mounting /mnt/synced_random_raid1_3legs_1
re-verifying checkit files in /mnt/synced_random_raid1_3legs_1

Unmounting xfs and removing mnt point on hayes-03...
Deactivating and removing raid(s)
Comment 15 errata-xmlrpc 2021-05-18 15:01:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659
Note You need to log in before you can comment on or make changes to this bug.