RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1872130 - error code for up conversion of degraded raid should reflect actual cmd result
Summary: error code for up conversion of degraded raid should reflect actual cmd result
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.3
Hardware: x86_64
OS: Linux
urgent
medium
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-25 07:16 UTC by Roman Bednář
Modified: 2021-09-07 11:52 UTC (History)
9 users (show)

Fixed In Version: lvm2-2.03.11-0.2.20201103git8801a86.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:01:53 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Roman Bednář 2020-08-25 07:16:57 UTC 
Description of problem:

Upconversion of degraded raid seems to be enabled now (although not working properly) in latest 8.3 lvm2 build while in previous builds this feature used to be turned off (green/passing tests validated that the feature is disabled):

http://cqe-live.cluster-qe.lab.eng.brq.redhat.com/clusterqe/detail/895d2ed5-e20a-4afd-99d7-43bc5c54f805



Version-Release number of selected component (if applicable):
lvm2-2.03.09-5.el8.x86_64
kernel-4.18.0-234.el8.x86_64

How reproducible:
always

Steps to Reproduce:

SCENARIO (raid1) - [degraded_upconversion_attempt]
Create a raid, fail one of the legs to enter a degraded state, and then attmept an upconversion
lvcreate  --type raid1 -m 1 -n degraded_upconvert -L 100M raid_sanity /dev/sda1 /dev/sdj1

secondary fail=/dev/sdj1
Disabling device sdj on virt-390rescan device...

Verifying this VG is now in an "Inconsistent" state due to the missing PV
pvs /dev/sdj1
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  Error reading device /dev/sdj1 at 0 length 4096.
  Error reading device /dev/sdj1 at 0 length 4096.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: VG raid_sanity is missing PV nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ (last written to /dev/sdj1).
  WARNING: Couldn't find all devices for LV raid_sanity/degraded_upconvert_rimage_1 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV raid_sanity/degraded_upconvert_rmeta_1 while checking used and assumed devices.
  Error reading device /dev/sdj1 at 0 length 4096.
  Failed to find device for physical volume "/dev/sdj1".
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: VG raid_sanity is missing PV nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ (last written to /dev/sdj1).
  WARNING: Couldn't find all devices for LV raid_sanity/degraded_upconvert_rimage_1 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV raid_sanity/degraded_upconvert_rmeta_1 while checking used and assumed devices.

VG reduce to removing failed device and put into degraded raid mode (vgreduce --removemissing -f raid_sanity)
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: VG raid_sanity is missing PV nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ (last written to /dev/sdj1).
  WARNING: Couldn't find all devices for LV raid_sanity/degraded_upconvert_rimage_1 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV raid_sanity/degraded_upconvert_rmeta_1 while checking used and assumed devices.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.
  WARNING: Couldn't find device with uuid nMnvp4-IHSk-4ShV-14Ap-Uc9Z-Yb5D-29xNrZ.

Enabling device sdj on virt-390	Running vgs to make LVM update metadata version if possible (will restore a-m PVs)
  WARNING: ignoring metadata seqno 3 on /dev/sdj1 for seqno 5 on /dev/sda1 for VG raid_sanity.
  WARNING: Inconsistent metadata found for VG raid_sanity.
  See vgck --updatemetadata to correct inconsistency.
  WARNING: outdated PV /dev/sdj1 seqno 3 has been removed in current VG raid_sanity seqno 5.
  See vgck --updatemetadata to clear outdated metadata.

Restoring the VG back to its original state
vgck --updatemetadata raid_sanity
  WARNING: ignoring metadata seqno 3 on /dev/sdj1 for seqno 5 on /dev/sda1 for VG raid_sanity.
  WARNING: Inconsistent metadata found for VG raid_sanity.
  See vgck --updatemetadata to correct inconsistency.
  WARNING: outdated PV /dev/sdj1 seqno 3 has been removed in current VG raid_sanity seqno 5.
  See vgck --updatemetadata to clear outdated metadata.
  WARNING: wiping mda on outdated PV /dev/sdj1
  WARNING: wiping header on outdated PV /dev/sdj1
vgextend raid_sanity /dev/sdj1
Attempting upconversion of degraded raid volume using failed PV /dev/sdj1
lvconvert --yes -m 1 raid_sanity/degraded_upconvert /dev/sdj1
  Type raid1 LV raid_sanity/degraded_upconvert already has 2 images.
upconversion of a degraded raid shouldn't work



Additional info:

even though the upconversion seems to have succeeded the leg remains faulty (type - error) until repaired manually:

# lvs -a -o lv_name,segtype,devices
  LV                            Type   Devices
  degraded_upconvert            raid1  degraded_upconvert_rimage_0(0),degraded_upconvert_rimage_1(0)
  [degraded_upconvert_rimage_0] linear /dev/sda1(1)
  [degraded_upconvert_rimage_1] error
  [degraded_upconvert_rmeta_0]  linear /dev/sda1(0)
  [degraded_upconvert_rmeta_1]  error
  root                          linear /dev/vda2(205)
  swap                          linear /dev/vda2(0)


Expected results:

degraded raid upconversion should fail as it did in previous builds and suggest repair action

# lvconvert --yes -m 1 raid_sanity/degraded_upconvert /dev/sda1
   Can't change number of mirrors of degraded raid_sanity/degraded_upconvert.
   Please run "lvconvert --repair raid_sanity/degraded_upconvert" first.
   WARNING: raid_sanity/degraded_upconvert already has image count of 2.

# lvconvert --repair raid_sanity/degraded_upconvert
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  Faulty devices in raid_sanity/degraded_upconvert successfully replaced.

# lvs -a -o lv_name,segtype,devices
  LV                            Type   Devices
  degraded_upconvert            raid1  degraded_upconvert_rimage_0(0),degraded_upconvert_rimage_1(0)
  [degraded_upconvert_rimage_0] linear /dev/sda1(1)
  [degraded_upconvert_rimage_1] linear /dev/sdd1(1)
  [degraded_upconvert_rmeta_0]  linear /dev/sda1(0)
  [degraded_upconvert_rmeta_1]  linear /dev/sdd1(0)
  root                          linear /dev/vda2(205)
  swap                          linear /dev/vda2(0)
Comment 1 Corey Marthaler 2020-08-25 16:54:58 UTC 
Like Roman's commands above showed, the up conversion still doesn't actually work (like expected), but the error code now appears to have regressed in that it now reports that these attempts actually worked.


[root@hayes-01 ~]# lvs -a -o +devices
  LV                            VG          Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                                      
  degraded_upconvert            raid_sanity rwi-a-r-r- 100.00m                                    100.00           degraded_upconvert_rimage_0(0),degraded_upconvert_rimage_1(0)
  [degraded_upconvert_rimage_0] raid_sanity vwi-aor-r- 100.00m                                                                                                                  
  [degraded_upconvert_rimage_1] raid_sanity iwi-aor--- 100.00m                                                     /dev/sdj1(1)                                                 
  [degraded_upconvert_rmeta_0]  raid_sanity ewi-aor-r-   4.00m                                                                                                                  
  [degraded_upconvert_rmeta_1]  raid_sanity ewi-aor---   4.00m                                                     /dev/sdj1(0)                                                 

# Try to up convert to 2 legs (-m 1)
[root@hayes-01 ~]# lvconvert --yes -m 1 raid_sanity/degraded_upconvert /dev/sdb1
  Type raid1 LV raid_sanity/degraded_upconvert already has 2 images.
  Logical volume raid_sanity/degraded_upconvert successfully converted.
[root@hayes-01 ~]# echo $?
0

# Try to up convert to 3 legs (-m 2)
[root@hayes-01 ~]# lvconvert --yes -m 2 raid_sanity/degraded_upconvert /dev/sdb1
  Can't change number of mirrors of degraded raid_sanity/degraded_upconvert.
  Please run "lvconvert --repair raid_sanity/degraded_upconvert" first.
  Logical volume raid_sanity/degraded_upconvert successfully converted.
[root@hayes-01 ~]# echo $?
0
Comment 2 Corey Marthaler 2020-08-25 17:15:19 UTC 
I downgraded to the prior lvm version to show a contrast of error codes:

lvm2-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
lvm2-dbusd-2.03.09-5.el8    BUILT: Wed Aug 12 15:49:44 CDT 2020
lvm2-lockd-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020


[root@hayes-01 ~]# lvs -a -o +devices
  LV                            VG          Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                                      
  degraded_upconvert            raid_sanity rwi-a-r-r- 100.00m                                    100.00           degraded_upconvert_rimage_0(0),degraded_upconvert_rimage_1(0)
  [degraded_upconvert_rimage_0] raid_sanity iwi-aor--- 100.00m                                                     /dev/sdb1(1)                                                 
  [degraded_upconvert_rimage_1] raid_sanity vwi-aor-r- 100.00m                                                                                                                  
  [degraded_upconvert_rmeta_0]  raid_sanity ewi-aor---   4.00m                                                     /dev/sdb1(0)                                                 
  [degraded_upconvert_rmeta_1]  raid_sanity ewi-aor-r-   4.00m                                                                                                                  

# Expected behavior and error code
[root@hayes-01 ~]# lvconvert --yes -m 1 raid_sanity/degraded_upconvert /dev/sdj1
  Can't change number of mirrors of degraded raid_sanity/degraded_upconvert.
  Please run "lvconvert --repair raid_sanity/degraded_upconvert" first.
  WARNING: raid_sanity/degraded_upconvert already has image count of 2.
[root@hayes-01 ~]# echo $?
5

# This one was a surprise, as it appears to be the exact same as latest rpms, and even though it gives a error "Can't change number of mirrors of degraded", this command actually *does* work w/o having to do the --repair cmd first.
[root@hayes-01 ~]# lvconvert --yes -m 2 raid_sanity/degraded_upconvert /dev/sdj1
  Can't change number of mirrors of degraded raid_sanity/degraded_upconvert.
  Please run "lvconvert --repair raid_sanity/degraded_upconvert" first.
  Logical volume raid_sanity/degraded_upconvert successfully converted.
[root@hayes-01 ~]# echo $?
0
[root@hayes-01 ~]# lvs -a -o +devices
  LV                            VG          Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices                                                                                     
  degraded_upconvert            raid_sanity rwi-a-r-r- 100.00m                                    100.00           degraded_upconvert_rimage_0(0),degraded_upconvert_rimage_1(0),degraded_upconvert_rimage_2(0)
  [degraded_upconvert_rimage_0] raid_sanity iwi-aor--- 100.00m                                                     /dev/sdb1(1)                                                                                
  [degraded_upconvert_rimage_1] raid_sanity vwi-aor-r- 100.00m                                                                                                                                                 
  [degraded_upconvert_rimage_2] raid_sanity iwi-aor--- 100.00m                                                     /dev/sdj1(1)                                                                                
  [degraded_upconvert_rmeta_0]  raid_sanity ewi-aor---   4.00m                                                     /dev/sdb1(0)                                                                                
  [degraded_upconvert_rmeta_1]  raid_sanity ewi-aor-r-   4.00m                                                                                                                                                 
  [degraded_upconvert_rmeta_2]  raid_sanity ewi-aor---   4.00m                                                     /dev/sdj1(0)
Comment 4 David Teigland 2020-09-23 16:12:41 UTC 
It looks like this bug is caused by this commit:

commit 3c9177fdc0b8f94c0ae335790a485477901b685d
Author: Heinz Mauelshagen <heinzm>
Date:   Mon Jul 20 15:36:25 2020 +0200

    lvconvert: display warning if raid1 LV image count does not change
    
    Fix "lvconvert -mN $RaidLV" to display a warning in
    case the same number of images is being requested.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1774696


If I revert that commit and repeat the test described above, I believe that we get the desired behavior:

# lvs -a test -o+devices,segtype
  LV            VG   Attr       LSize   Cpy%Sync Devices                       Type  
  mm            test rwi-a-r-r- 100.00m 100.00   mm_rimage_0(0),mm_rimage_1(0) raid1 
  [mm_rimage_0] test iwi-aor--- 100.00m          /dev/sdb(1)                   linear
  [mm_rimage_1] test vwi-aor-r- 100.00m                                        error 
  [mm_rmeta_0]  test ewi-aor---   4.00m          /dev/sdb(0)                   linear
  [mm_rmeta_1]  test ewi-aor-r-   4.00m                                        error 

# lvconvert -y -m1 test/mm /dev/sdc 
  Can't change number of mirrors of degraded test/mm.
  Please run "lvconvert --repair test/mm" first.
  WARNING: test/mm already has image count of 2.

# echo $?
5


I hesitate to revert commit 3c9177fd because it claims to be fixing bug 1774696.  However, looking at 1774696, I'm not sure if the commit above is actually necessary.  bug 1774696 requests a change for "mirror", but commit 3c9177fd changes "raid1".

Heinz, can you confirm if we can safely revert commit 3c9177fd to fix this bz, without breaking bz 1774696?
Comment 5 Heinz Mauelshagen 2020-09-28 15:20:26 UTC 
(In reply to David Teigland from comment #4)
> It looks like this bug is caused by this commit:
> 
> commit 3c9177fdc0b8f94c0ae335790a485477901b685d
> Author: Heinz Mauelshagen <heinzm>
> Date:   Mon Jul 20 15:36:25 2020 +0200
> 
>     lvconvert: display warning if raid1 LV image count does not change
>     
>     Fix "lvconvert -mN $RaidLV" to display a warning in
>     case the same number of images is being requested.
>     
>     Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1774696
> 
> 
> If I revert that commit and repeat the test described above, I believe that
> we get the desired behavior:
> 
> # lvs -a test -o+devices,segtype
>   LV            VG   Attr       LSize   Cpy%Sync Devices                    
> Type  
>   mm            test rwi-a-r-r- 100.00m 100.00  
> mm_rimage_0(0),mm_rimage_1(0) raid1 
>   [mm_rimage_0] test iwi-aor--- 100.00m          /dev/sdb(1)                
> linear
>   [mm_rimage_1] test vwi-aor-r- 100.00m                                     
> error 
>   [mm_rmeta_0]  test ewi-aor---   4.00m          /dev/sdb(0)                
> linear
>   [mm_rmeta_1]  test ewi-aor-r-   4.00m                                     
> error 
> 
> # lvconvert -y -m1 test/mm /dev/sdc 
>   Can't change number of mirrors of degraded test/mm.
>   Please run "lvconvert --repair test/mm" first.
>   WARNING: test/mm already has image count of 2.
> 
> # echo $?
> 5
> 
> 
> I hesitate to revert commit 3c9177fd because it claims to be fixing bug
> 1774696.  However, looking at 1774696, I'm not sure if the commit above is
> actually necessary.  bug 1774696 requests a change for "mirror", but commit
> 3c9177fd changes "raid1".
> 
> Heinz, can you confirm if we can safely revert commit 3c9177fd to fix this
> bz, without breaking bz 1774696?

Yes, 3c9177fd can be reverted safely, a check for the same image count is in lv_raid_change_image_count() already.

Reverted in
master: 8952dcbff0c1ca9bfaa2f8fe8b6da78a10319e6b
stable-2.02: 7e2ceba2fb6ea6769608858f7ce9a88e5ff3eb68

I'll revisit bug 1774696 relative to 'mirror'.
Comment 15 errata-xmlrpc 2021-05-18 15:01:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659
Note You need to log in before you can comment on or make changes to this bug.