1572528 – readonly fs mounted on a raidlv when legs of mirror removed and replaced by new devices one by one

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1572528 - readonly fs mounted on a raidlv when legs of mirror removed and replaced by new devices one by one

Summary: readonly fs mounted on a raidlv when legs of mirror removed and replaced by n...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	LVM and device-mapper development team
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-04-27 08:48 UTC by nikhil kshirsagar
Modified:	2021-09-03 12:41 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-05-02 13:09:33 UTC
Target Upstream Version:
Embargoed:
Flags:	heinzm: needinfo-

Attachments	(Terms of Use)
Test script shoing the non-repairing behaviour of "lvconvert -m1 ..." in this use case (1001 bytes, application/x-shellscript) 2018-05-02 13:12 UTC, Heinz Mauelshagen	no flags	Details
View All

Description nikhil kshirsagar 2018-04-27 08:48:35 UTC

Description of problem:

if we create raid lv with sdf and sdg.

remove sdf. (offline)
seems fine.
extend vg with sdh and convert to mirror again of 2 legs, using sdg and sdh, ok then.
now remove sdg. (second original leg of mirror)
extend mirror with sdi
convert to mirror again with sdh and sdi
fs seems to go readonly in a minute or two, with the dmesg errors shown in the notes

Version-Release number of selected component (if applicable):
vm2-2.02.171-8.el7.x86_64
lvm2-libs-2.02.171-8.el7.x86_64

How reproducible:
see additional info

Steps to Reproduce:
see additional info





Additional info:


reproduces with 4 new devices sdf,sdg,sdh and sdi,


Disk /dev/sdf: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdg: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdh: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdi: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


[root@vm253-107 ~]# pvcreate /dev/sdf /dev/sdg /dev/sdh /dev/sdi
  WARNING: Not using lvmetad because a repair command was run.
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  Physical volume "/dev/sdf" successfully created.
  Physical volume "/dev/sdg" successfully created.
  Physical volume "/dev/sdh" successfully created.
  Physical volume "/dev/sdi" successfully created.

[root@vm253-107 ~]# vgcreate vg_mirror1 /dev/sdf /dev/sdg
  WARNING: Not using lvmetad because a repair command was run.
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  Volume group "vg_mirror1" successfully created

[root@vm253-107 ~]# vgs
  WARNING: Not using lvmetad because a repair command was run.
  VG         #PV #LV #SN Attr   VSize   VFree 
  rhel         1   5   0 wz--n- <17.50g <5.50g
  vg_mirror1   2   0   0 wz--n-   1.99g  1.99g
[root@vm253-107 ~]# lvcreate -L 500M -m 1 -n lv_mirror1 vg_mirror1
  WARNING: Not using lvmetad because a repair command was run.
  Logical volume "lv_mirror1" created.

[root@vm253-107 ~]# lvs
  WARNING: Not using lvmetad because a repair command was run.
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-a-r--- 500.00m                                    100.00          
[root@vm253-107 ~]# mkfs.xfs /dev/mapper/vg_mirror1-lv_mirror1
meta-data=/dev/mapper/vg_mirror1-lv_mirror1 isize=512    agcount=4, agsize=32000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=128000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=855, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@vm253-107 ~]# mkdir /formount1
[root@vm253-107 ~]# mount /dev/mapper/vg_mirror1-lv_mirror1 /formount1/
[root@vm253-107 ~]# touch /formount1/aa
[root@vm253-107 ~]# echo "offline" > /sys/block/sdf/device/state 
[root@vm253-107 ~]# lvs
  WARNING: Not using lvmetad because a repair command was run.
  Couldn't find device with uuid ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A.
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-p- 500.00m                                    100.00          
[root@vm253-107 ~]# vgs
  WARNING: Not using lvmetad because a repair command was run.
  Couldn't find device with uuid ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A.
  VG         #PV #LV #SN Attr   VSize   VFree 
  rhel         1   5   0 wz--n- <17.50g <5.50g
  vg_mirror1   2   1   0 wz-pn-   1.99g <1.01g
[root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1 
/dev/sda3  /dev/sdg   [unknown]  
[root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1 
  WARNING: Device for PV ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV vg_mirror1/lv_mirror1_rimage_0 while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV vg_mirror1/lv_mirror1_rmeta_0 while checking used and assumed devices.
  Wrote out consistent volume group vg_mirror1.
[root@vm253-107 ~]# vgs
  VG         #PV #LV #SN Attr   VSize    VFree  
  rhel         1   5   0 wz--n-  <17.50g  <5.50g
  vg_mirror1   1   1   0 wz--n- 1020.00m 516.00m
[root@vm253-107 ~]# lvs
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# pvs
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  PV         VG         Fmt  Attr PSize    PFree  
  /dev/sda3  rhel       lvm2 a--   <17.50g  <5.50g
  /dev/sdc   [unknown]  lvm2 u--     1.00g      0 
  /dev/sdd              lvm2 ---     1.00g   1.00g
  /dev/sde              lvm2 ---     1.00g   1.00g
  /dev/sdg   vg_mirror1 lvm2 a--  1020.00m 516.00m
  /dev/sdh              lvm2 ---     1.00g   1.00g
  /dev/sdi              lvm2 ---     1.00g   1.00g
[root@vm253-107 ~]# touch /formount1/aaa
[root@vm253-107 ~]# ls /formount1/
aa  aaa
[root@vm253-107 ~]# vgextend vg_mirror1 /dev/sdh
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  Volume group "vg_mirror1" successfully extended
[root@vm253-107 ~]# vgs
  VG         #PV #LV #SN Attr   VSize   VFree 
  rhel         1   5   0 wz--n- <17.50g <5.50g
  vg_mirror1   2   1   0 wz--n-   1.99g  1.50g
[root@vm253-107 ~]# lvs
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
(reverse-i-search)`cat': ^Ct /sys/block/sde/device/state 
[root@vm253-107 ~]# ls /formount1/
aa  aaa
[root@vm253-107 ~]# lvconvert -m 1 /dev/mapper/vg_mirror1-lv_mirror1 /dev/sdg /dev/sdh
Are you sure you want to convert raid1 LV vg_mirror1/lv_mirror1 to 2 images enhancing resilience? [y/n]: y
  WARNING: vg_mirror1/lv_mirror1 already has image count of 2.
  Logical volume vg_mirror1/lv_mirror1 successfully converted.
[root@vm253-107 ~]# lvs
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# pvs
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  PV         VG         Fmt  Attr PSize    PFree   
  /dev/sda3  rhel       lvm2 a--   <17.50g   <5.50g
  /dev/sdc   [unknown]  lvm2 u--     1.00g       0 
  /dev/sdd              lvm2 ---     1.00g    1.00g
  /dev/sde              lvm2 ---     1.00g    1.00g
  /dev/sdg   vg_mirror1 lvm2 a--  1020.00m  516.00m
  /dev/sdh   vg_mirror1 lvm2 a--  1020.00m 1020.00m
  /dev/sdi              lvm2 ---     1.00g    1.00g
[root@vm253-107 ~]# ls /formount1/
aa  aaa

[root@vm253-107 ~]# pvs
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  PV         VG         Fmt  Attr PSize    PFree   
  /dev/sda3  rhel       lvm2 a--   <17.50g   <5.50g
  /dev/sdc   [unknown]  lvm2 u--     1.00g       0 
  /dev/sdd              lvm2 ---     1.00g    1.00g
  /dev/sde              lvm2 ---     1.00g    1.00g
  /dev/sdg   vg_mirror1 lvm2 a--  1020.00m  516.00m
  /dev/sdh   vg_mirror1 lvm2 a--  1020.00m 1020.00m
  /dev/sdi              lvm2 ---     1.00g    1.00g
[root@vm253-107 ~]# echo "offline" > /sys/block/sdg/device/state 
[root@vm253-107 ~]# lvs
  /dev/sdg: open failed: No such device or address
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# vgs
  /dev/sdg: open failed: No such device or address
  VG         #PV #LV #SN Attr   VSize   VFree 
  rhel         1   5   0 wz--n- <17.50g <5.50g
  vg_mirror1   2   1   0 wz--n-   1.99g  1.50g
[root@vm253-107 ~]# vgextend vg_mirror1 /dev/sdi
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdg: open failed: No such device or address
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  WARNING: Failed to write an MDA of VG vg_mirror1.
  Volume group "vg_mirror1" successfully extended
  /dev/sdg: open failed: No such device or address
[root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1 
  /dev/sdg: open failed: No such device or address
  Volume group "vg_mirror1" is already consistent.
[root@vm253-107 ~]# lvs
  /dev/sdg: open failed: No such device or address
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# pvs
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdg: open failed: No such device or address
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  PV         VG         Fmt  Attr PSize    PFree   
  /dev/sda3  rhel       lvm2 a--   <17.50g   <5.50g
  /dev/sdc   [unknown]  lvm2 u--     1.00g       0 
  /dev/sdd              lvm2 ---     1.00g    1.00g
  /dev/sde              lvm2 ---     1.00g    1.00g
  /dev/sdg   vg_mirror1 lvm2 a--  1020.00m  516.00m
  /dev/sdh   vg_mirror1 lvm2 a--  1020.00m 1020.00m
  /dev/sdi   vg_mirror1 lvm2 a--  1020.00m 1020.00m
[root@vm253-107 ~]# ls /formount1/
aa  aaa
[root@vm253-107 ~]# ^C
[root@vm253-107 ~]# lvconvert -m 1 /dev/mapper/vg_mirror1-lv_mirror1 /dev/sdh /dev/sdi
  /dev/sdg: open failed: No such device or address
Are you sure you want to convert raid1 LV vg_mirror1/lv_mirror1 to 2 images enhancing resilience? [y/n]: y
  WARNING: vg_mirror1/lv_mirror1 already has image count of 2.
  Logical volume vg_mirror1/lv_mirror1 successfully converted.
[root@vm253-107 ~]# lvs
  /dev/sdg: open failed: No such device or address
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# pvs
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdg: open failed: No such device or address
  WARNING: PV /dev/sdc is marked in use but no VG was found using it.
  WARNING: PV /dev/sdc might need repairing.
  PV         VG         Fmt  Attr PSize    PFree   
  /dev/sda3  rhel       lvm2 a--   <17.50g   <5.50g
  /dev/sdc   [unknown]  lvm2 u--     1.00g       0 
  /dev/sdd              lvm2 ---     1.00g    1.00g
  /dev/sde              lvm2 ---     1.00g    1.00g
  /dev/sdg   vg_mirror1 lvm2 a--  1020.00m  516.00m
  /dev/sdh   vg_mirror1 lvm2 a--  1020.00m 1020.00m
  /dev/sdi   vg_mirror1 lvm2 a--  1020.00m 1020.00m
[root@vm253-107 ~]# ls /formount1/
aa  aaa
[root@vm253-107 ~]# lvs
  /dev/sdg: open failed: No such device or address
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# lvchange --refresh vg_mirror1/lv_mirror1
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  /dev/sdg: open failed: No such device or address
  Failed to remove temporary SubLVs from vg_mirror1/lv_mirror1
[root@vm253-107 ~]# ls /formount1/
aa  aaa
[root@vm253-107 ~]# ls /formount1/aa
aa   aaa  
[root@vm253-107 ~]# ls /formount1/aa*
/formount1/aa  /formount1/aaa
[root@vm253-107 ~]# lvs
  /dev/sdg: open failed: No such device or address
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# vgs
  /dev/sdg: open failed: No such device or address
  VG         #PV #LV #SN Attr   VSize   VFree 
  rhel         1   5   0 wz--n- <17.50g <5.50g
  vg_mirror1   3   1   0 wz--n-  <2.99g <2.50g
[root@vm253-107 ~]# lvs
  /dev/sdg: open failed: No such device or address
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# date
Tue Apr 24 10:40:36 IST 2018
[root@vm253-107 ~]# lvs
  /dev/sdg: open failed: No such device or address
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home       rhel       -wi-ao----   2.00g                                                    
  root       rhel       -wi-ao----   2.00g                                                    
  tmp        rhel       -wi-ao----   2.00g                                                    
  usr        rhel       -wi-ao----   4.00g                                                    
  var        rhel       -wi-ao----   2.00g                                                    
  lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m                                    100.00          
[root@vm253-107 ~]# ls /formount1/aa*
/formount1/aa  /formount1/aaa
[root@vm253-107 ~]# touch /formount1/aaas


few minutes later,


[root@vm253-107 log]# vgs
  WARNING: Not using lvmetad because a repair command was run.
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
  /dev/vg_mirror1/lv_mirror1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid r7SDmy-Kv6T-sZop-hWoA-l5zm-c17m-DxCzuL.
  VG         #PV #LV #SN Attr   VSize   VFree 
  rhel         1   5   0 wz--n- <17.50g <5.50g
  vg_mirror1   3   1   0 wz-pn-  <2.99g <2.50g
[root@vm253-107 log]# df
Filesystem                        1K-blocks    Used Available Use% Mounted on
/dev/mapper/rhel-root               1998672   52720   1824712   3% /
devtmpfs                             458356       0    458356   0% /dev
tmpfs                                474096       0    474096   0% /dev/shm
tmpfs                                474096   48932    425164  11% /run
tmpfs                                474096       0    474096   0% /sys/fs/cgroup
/dev/mapper/rhel-usr                3997376 3062072    709208  82% /usr
/dev/sda1                            499656  131476    331484  29% /boot
/dev/mapper/rhel-var                1998672  220004   1657428  12% /var
/dev/mapper/rhel-tmp                1998672    6236   1871196   1% /tmp
/dev/mapper/rhel-home               1998672   11824   1865608   1% /home
tmpfs                                 94820       4     94816   1% /run/user/992
tmpfs                                 94820      36     94784   1% /run/user/1000
tmpfs                                 94820       0     94820   0% /run/user/0
/dev/mapper/vg_mirror1-lv_mirror1    508580   25776    482804   6% /formount1
[root@vm253-107 log]# touch /formount1/aaaa
touch: cannot touch ‘/formount1/aaaa’: Input/output error
[root@vm253-107 log]# date
Tue Apr 24 10:43:50 IST 2018
[root@vm253-107 log]# 

dmesg logs show:


[8958506.464323] md/raid1:mdX: active with 2 out of 2 mirrors
[8958506.464420] dm-14: detected capacity change from 0 to 524288000
[8958506.505924] mdX: bitmap file is out of date, doing full recovery
[8958506.551661] md: resync of RAID array mdX
[8958519.309709] md: mdX: resync done.
[8958550.419780] XFS (dm-14): Mounting V5 Filesystem
[8958550.510750] XFS (dm-14): Ending clean mount
[8958580.536165] sd 2:0:0:9: rejecting I/O to offline device
[8958580.536217] blk_update_request: I/O error, dev sdf, sector 0
[8958580.536417] sd 2:0:0:9: rejecting I/O to offline device
[8958580.536420] blk_update_request: I/O error, dev sdf, sector 0
[8958580.536439] sd 2:0:0:9: rejecting I/O to offline device
[8958580.536444] md: super_written gets error=-5, uptodate=0
[8958580.536452] md/raid1:mdX: Disk failure on dm-11, disabling device.
md/raid1:mdX: Operation continuing on 1 devices.
[8958604.877385] device-mapper: raid: Failed to read superblock of device at position -1
[8958852.793906] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794128] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794151] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794176] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794185] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687397] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687617] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687644] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687660] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687688] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580780] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580960] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580984] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580998] sd 2:0:0:8: rejecting I/O to offline device
[8958891.581025] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771215] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771397] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771418] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771430] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771438] sd 2:0:0:8: rejecting I/O to offline device
[8959004.986828] sd 2:0:0:8: rejecting I/O to offline device
[8959004.986846] blk_update_request: I/O error, dev sdg, sector 0
[8959004.986861] sd 2:0:0:8: rejecting I/O to offline device
[8959004.986869] md: super_written gets error=-5, uptodate=0 <----  ??
[8959004.987414] device-mapper: raid: Failed to read superblock of device at position -1  <-----  ??
[root@vm253-107 ~]#

==============================================================================

relevant mails to engineering


---------- Forwarded message ----------
From: Heinz Mauelshagen <heinzm>
Date: Thu, Apr 26, 2018 at 7:52 PM
Subject: Re: Fwd: issues with raid type , fs goes readonly ?
To: Nikhil Kshirsagar <nkshirsa>


On 04/25/2018 11:21 AM, Nikhil Kshirsagar wrote:
Hello Heinz,

please could you check the reproducer steps, is this a bug, shall we file a bz ?

Yes, file a bz, please.

And tell the customer that after extending the VG, "lvconvert --repair nvm/r" is the recomended way to fix
such failures rather than "lvconvert -m...".

Heinz



On Tue, Apr 24, 2018 at 10:50 AM, Nikhil Kshirsagar <nkshirsa> wrote:
Hi Heinz,

I can reproduce this too on 7.4 , please take a look at my notes in the attachment. It happens when we first remove one leg of the raid lv (ok then), and then the second one..

so if we create raid with sdf and sdg.

remove sdf. (offline)
seems fine.
extend vg with sdh and convert to mirror again of 2 legs, using sdg and sdh, ok then.
now remove sdg. (second original leg of mirror)
extend mirror with sdi
convert to mirror again with sdh and sdi
fs seems to go readonly in a minute or two, with the dmesg errors shown in the notes


Thanks,
Nikhil.

Comment 3 Heinz Mauelshagen 2018-05-02 13:09:33 UTC

"lvconvert -m1 ..." does not repair the RaidLV.

It is not the recommended way to deal with such failures (see lvmraid(7) sections related to (automatic) repair and replace devices).

Please use "lvconvert --repair ..." after adding enough PVs (at least one in this case) providing enough space to allocate new SubLVs to replace the failed ones.

You can control with "lvs -ao+devices $vg", which lists the SubLVs.

The filesystem going read-only is a result of "lvconvert -m1 ..." iterations in your test _not_ fixing the degraded raid1 LV loosing data when the last raid1 leg goes offline.

Comment 4 Heinz Mauelshagen 2018-05-02 13:12:50 UTC

Created attachment 1430053 [details]
Test script shoing the non-repairing behaviour of "lvconvert -m1 ..." in this use case

Note You need to log in before you can comment on or make changes to this bug.