Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Descriptionnikhil kshirsagar
2018-04-27 08:48:35 UTC
Description of problem:
if we create raid lv with sdf and sdg.
remove sdf. (offline)
seems fine.
extend vg with sdh and convert to mirror again of 2 legs, using sdg and sdh, ok then.
now remove sdg. (second original leg of mirror)
extend mirror with sdi
convert to mirror again with sdh and sdi
fs seems to go readonly in a minute or two, with the dmesg errors shown in the notes
Version-Release number of selected component (if applicable):
vm2-2.02.171-8.el7.x86_64
lvm2-libs-2.02.171-8.el7.x86_64
How reproducible:
see additional info
Steps to Reproduce:
see additional info
Additional info:
reproduces with 4 new devices sdf,sdg,sdh and sdi,
Disk /dev/sdf: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdg: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdh: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdi: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@vm253-107 ~]# pvcreate /dev/sdf /dev/sdg /dev/sdh /dev/sdi
WARNING: Not using lvmetad because a repair command was run.
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
Physical volume "/dev/sdf" successfully created.
Physical volume "/dev/sdg" successfully created.
Physical volume "/dev/sdh" successfully created.
Physical volume "/dev/sdi" successfully created.
[root@vm253-107 ~]# vgcreate vg_mirror1 /dev/sdf /dev/sdg
WARNING: Not using lvmetad because a repair command was run.
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
Volume group "vg_mirror1" successfully created
[root@vm253-107 ~]# vgs
WARNING: Not using lvmetad because a repair command was run.
VG #PV #LV #SN Attr VSize VFree
rhel 1 5 0 wz--n- <17.50g <5.50g
vg_mirror1 2 0 0 wz--n- 1.99g 1.99g
[root@vm253-107 ~]# lvcreate -L 500M -m 1 -n lv_mirror1 vg_mirror1
WARNING: Not using lvmetad because a repair command was run.
Logical volume "lv_mirror1" created.
[root@vm253-107 ~]# lvs
WARNING: Not using lvmetad because a repair command was run.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-a-r--- 500.00m 100.00
[root@vm253-107 ~]# mkfs.xfs /dev/mapper/vg_mirror1-lv_mirror1
meta-data=/dev/mapper/vg_mirror1-lv_mirror1 isize=512 agcount=4, agsize=32000 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=128000, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=855, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@vm253-107 ~]# mkdir /formount1
[root@vm253-107 ~]# mount /dev/mapper/vg_mirror1-lv_mirror1 /formount1/
[root@vm253-107 ~]# touch /formount1/aa
[root@vm253-107 ~]# echo "offline" > /sys/block/sdf/device/state
[root@vm253-107 ~]# lvs
WARNING: Not using lvmetad because a repair command was run.
Couldn't find device with uuid ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-p- 500.00m 100.00
[root@vm253-107 ~]# vgs
WARNING: Not using lvmetad because a repair command was run.
Couldn't find device with uuid ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A.
VG #PV #LV #SN Attr VSize VFree
rhel 1 5 0 wz--n- <17.50g <5.50g
vg_mirror1 2 1 0 wz-pn- 1.99g <1.01g
[root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1
/dev/sda3 /dev/sdg [unknown]
[root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1
WARNING: Device for PV ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A not found or rejected by a filter.
WARNING: Couldn't find all devices for LV vg_mirror1/lv_mirror1_rimage_0 while checking used and assumed devices.
WARNING: Couldn't find all devices for LV vg_mirror1/lv_mirror1_rmeta_0 while checking used and assumed devices.
Wrote out consistent volume group vg_mirror1.
[root@vm253-107 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
rhel 1 5 0 wz--n- <17.50g <5.50g
vg_mirror1 1 1 0 wz--n- 1020.00m 516.00m
[root@vm253-107 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# pvs
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
PV VG Fmt Attr PSize PFree
/dev/sda3 rhel lvm2 a-- <17.50g <5.50g
/dev/sdc [unknown] lvm2 u-- 1.00g 0
/dev/sdd lvm2 --- 1.00g 1.00g
/dev/sde lvm2 --- 1.00g 1.00g
/dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m
/dev/sdh lvm2 --- 1.00g 1.00g
/dev/sdi lvm2 --- 1.00g 1.00g
[root@vm253-107 ~]# touch /formount1/aaa
[root@vm253-107 ~]# ls /formount1/
aa aaa
[root@vm253-107 ~]# vgextend vg_mirror1 /dev/sdh
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
Volume group "vg_mirror1" successfully extended
[root@vm253-107 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
rhel 1 5 0 wz--n- <17.50g <5.50g
vg_mirror1 2 1 0 wz--n- 1.99g 1.50g
[root@vm253-107 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
(reverse-i-search)`cat': ^Ct /sys/block/sde/device/state
[root@vm253-107 ~]# ls /formount1/
aa aaa
[root@vm253-107 ~]# lvconvert -m 1 /dev/mapper/vg_mirror1-lv_mirror1 /dev/sdg /dev/sdh
Are you sure you want to convert raid1 LV vg_mirror1/lv_mirror1 to 2 images enhancing resilience? [y/n]: y
WARNING: vg_mirror1/lv_mirror1 already has image count of 2.
Logical volume vg_mirror1/lv_mirror1 successfully converted.
[root@vm253-107 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# pvs
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
PV VG Fmt Attr PSize PFree
/dev/sda3 rhel lvm2 a-- <17.50g <5.50g
/dev/sdc [unknown] lvm2 u-- 1.00g 0
/dev/sdd lvm2 --- 1.00g 1.00g
/dev/sde lvm2 --- 1.00g 1.00g
/dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m
/dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m
/dev/sdi lvm2 --- 1.00g 1.00g
[root@vm253-107 ~]# ls /formount1/
aa aaa
[root@vm253-107 ~]# pvs
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
PV VG Fmt Attr PSize PFree
/dev/sda3 rhel lvm2 a-- <17.50g <5.50g
/dev/sdc [unknown] lvm2 u-- 1.00g 0
/dev/sdd lvm2 --- 1.00g 1.00g
/dev/sde lvm2 --- 1.00g 1.00g
/dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m
/dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m
/dev/sdi lvm2 --- 1.00g 1.00g
[root@vm253-107 ~]# echo "offline" > /sys/block/sdg/device/state
[root@vm253-107 ~]# lvs
/dev/sdg: open failed: No such device or address
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# vgs
/dev/sdg: open failed: No such device or address
VG #PV #LV #SN Attr VSize VFree
rhel 1 5 0 wz--n- <17.50g <5.50g
vg_mirror1 2 1 0 wz--n- 1.99g 1.50g
[root@vm253-107 ~]# vgextend vg_mirror1 /dev/sdi
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
/dev/sdg: open failed: No such device or address
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
WARNING: Failed to write an MDA of VG vg_mirror1.
Volume group "vg_mirror1" successfully extended
/dev/sdg: open failed: No such device or address
[root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1
/dev/sdg: open failed: No such device or address
Volume group "vg_mirror1" is already consistent.
[root@vm253-107 ~]# lvs
/dev/sdg: open failed: No such device or address
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# pvs
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
/dev/sdg: open failed: No such device or address
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
PV VG Fmt Attr PSize PFree
/dev/sda3 rhel lvm2 a-- <17.50g <5.50g
/dev/sdc [unknown] lvm2 u-- 1.00g 0
/dev/sdd lvm2 --- 1.00g 1.00g
/dev/sde lvm2 --- 1.00g 1.00g
/dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m
/dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m
/dev/sdi vg_mirror1 lvm2 a-- 1020.00m 1020.00m
[root@vm253-107 ~]# ls /formount1/
aa aaa
[root@vm253-107 ~]# ^C
[root@vm253-107 ~]# lvconvert -m 1 /dev/mapper/vg_mirror1-lv_mirror1 /dev/sdh /dev/sdi
/dev/sdg: open failed: No such device or address
Are you sure you want to convert raid1 LV vg_mirror1/lv_mirror1 to 2 images enhancing resilience? [y/n]: y
WARNING: vg_mirror1/lv_mirror1 already has image count of 2.
Logical volume vg_mirror1/lv_mirror1 successfully converted.
[root@vm253-107 ~]# lvs
/dev/sdg: open failed: No such device or address
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# pvs
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
/dev/sdg: open failed: No such device or address
WARNING: PV /dev/sdc is marked in use but no VG was found using it.
WARNING: PV /dev/sdc might need repairing.
PV VG Fmt Attr PSize PFree
/dev/sda3 rhel lvm2 a-- <17.50g <5.50g
/dev/sdc [unknown] lvm2 u-- 1.00g 0
/dev/sdd lvm2 --- 1.00g 1.00g
/dev/sde lvm2 --- 1.00g 1.00g
/dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m
/dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m
/dev/sdi vg_mirror1 lvm2 a-- 1020.00m 1020.00m
[root@vm253-107 ~]# ls /formount1/
aa aaa
[root@vm253-107 ~]# lvs
/dev/sdg: open failed: No such device or address
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# lvchange --refresh vg_mirror1/lv_mirror1
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
/dev/sdg: open failed: No such device or address
Failed to remove temporary SubLVs from vg_mirror1/lv_mirror1
[root@vm253-107 ~]# ls /formount1/
aa aaa
[root@vm253-107 ~]# ls /formount1/aa
aa aaa
[root@vm253-107 ~]# ls /formount1/aa*
/formount1/aa /formount1/aaa
[root@vm253-107 ~]# lvs
/dev/sdg: open failed: No such device or address
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# vgs
/dev/sdg: open failed: No such device or address
VG #PV #LV #SN Attr VSize VFree
rhel 1 5 0 wz--n- <17.50g <5.50g
vg_mirror1 3 1 0 wz--n- <2.99g <2.50g
[root@vm253-107 ~]# lvs
/dev/sdg: open failed: No such device or address
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# date
Tue Apr 24 10:40:36 IST 2018
[root@vm253-107 ~]# lvs
/dev/sdg: open failed: No such device or address
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home rhel -wi-ao---- 2.00g
root rhel -wi-ao---- 2.00g
tmp rhel -wi-ao---- 2.00g
usr rhel -wi-ao---- 4.00g
var rhel -wi-ao---- 2.00g
lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00
[root@vm253-107 ~]# ls /formount1/aa*
/formount1/aa /formount1/aaa
[root@vm253-107 ~]# touch /formount1/aaas
few minutes later,
[root@vm253-107 log]# vgs
WARNING: Not using lvmetad because a repair command was run.
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error
/dev/vg_mirror1/lv_mirror1: read failed after 0 of 2048 at 0: Input/output error
Couldn't find device with uuid r7SDmy-Kv6T-sZop-hWoA-l5zm-c17m-DxCzuL.
VG #PV #LV #SN Attr VSize VFree
rhel 1 5 0 wz--n- <17.50g <5.50g
vg_mirror1 3 1 0 wz-pn- <2.99g <2.50g
[root@vm253-107 log]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/rhel-root 1998672 52720 1824712 3% /
devtmpfs 458356 0 458356 0% /dev
tmpfs 474096 0 474096 0% /dev/shm
tmpfs 474096 48932 425164 11% /run
tmpfs 474096 0 474096 0% /sys/fs/cgroup
/dev/mapper/rhel-usr 3997376 3062072 709208 82% /usr
/dev/sda1 499656 131476 331484 29% /boot
/dev/mapper/rhel-var 1998672 220004 1657428 12% /var
/dev/mapper/rhel-tmp 1998672 6236 1871196 1% /tmp
/dev/mapper/rhel-home 1998672 11824 1865608 1% /home
tmpfs 94820 4 94816 1% /run/user/992
tmpfs 94820 36 94784 1% /run/user/1000
tmpfs 94820 0 94820 0% /run/user/0
/dev/mapper/vg_mirror1-lv_mirror1 508580 25776 482804 6% /formount1
[root@vm253-107 log]# touch /formount1/aaaa
touch: cannot touch ‘/formount1/aaaa’: Input/output error
[root@vm253-107 log]# date
Tue Apr 24 10:43:50 IST 2018
[root@vm253-107 log]#
dmesg logs show:
[8958506.464323] md/raid1:mdX: active with 2 out of 2 mirrors
[8958506.464420] dm-14: detected capacity change from 0 to 524288000
[8958506.505924] mdX: bitmap file is out of date, doing full recovery
[8958506.551661] md: resync of RAID array mdX
[8958519.309709] md: mdX: resync done.
[8958550.419780] XFS (dm-14): Mounting V5 Filesystem
[8958550.510750] XFS (dm-14): Ending clean mount
[8958580.536165] sd 2:0:0:9: rejecting I/O to offline device
[8958580.536217] blk_update_request: I/O error, dev sdf, sector 0
[8958580.536417] sd 2:0:0:9: rejecting I/O to offline device
[8958580.536420] blk_update_request: I/O error, dev sdf, sector 0
[8958580.536439] sd 2:0:0:9: rejecting I/O to offline device
[8958580.536444] md: super_written gets error=-5, uptodate=0
[8958580.536452] md/raid1:mdX: Disk failure on dm-11, disabling device.
md/raid1:mdX: Operation continuing on 1 devices.
[8958604.877385] device-mapper: raid: Failed to read superblock of device at position -1
[8958852.793906] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794128] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794151] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794176] sd 2:0:0:8: rejecting I/O to offline device
[8958852.794185] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687397] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687617] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687644] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687660] sd 2:0:0:8: rejecting I/O to offline device
[8958885.687688] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580780] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580960] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580984] sd 2:0:0:8: rejecting I/O to offline device
[8958891.580998] sd 2:0:0:8: rejecting I/O to offline device
[8958891.581025] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771215] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771397] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771418] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771430] sd 2:0:0:8: rejecting I/O to offline device
[8958973.771438] sd 2:0:0:8: rejecting I/O to offline device
[8959004.986828] sd 2:0:0:8: rejecting I/O to offline device
[8959004.986846] blk_update_request: I/O error, dev sdg, sector 0
[8959004.986861] sd 2:0:0:8: rejecting I/O to offline device
[8959004.986869] md: super_written gets error=-5, uptodate=0 <---- ??
[8959004.987414] device-mapper: raid: Failed to read superblock of device at position -1 <----- ??
[root@vm253-107 ~]#
==============================================================================
relevant mails to engineering
---------- Forwarded message ----------
From: Heinz Mauelshagen <heinzm>
Date: Thu, Apr 26, 2018 at 7:52 PM
Subject: Re: Fwd: issues with raid type , fs goes readonly ?
To: Nikhil Kshirsagar <nkshirsa>
On 04/25/2018 11:21 AM, Nikhil Kshirsagar wrote:
Hello Heinz,
please could you check the reproducer steps, is this a bug, shall we file a bz ?
Yes, file a bz, please.
And tell the customer that after extending the VG, "lvconvert --repair nvm/r" is the recomended way to fix
such failures rather than "lvconvert -m...".
Heinz
On Tue, Apr 24, 2018 at 10:50 AM, Nikhil Kshirsagar <nkshirsa> wrote:
Hi Heinz,
I can reproduce this too on 7.4 , please take a look at my notes in the attachment. It happens when we first remove one leg of the raid lv (ok then), and then the second one..
so if we create raid with sdf and sdg.
remove sdf. (offline)
seems fine.
extend vg with sdh and convert to mirror again of 2 legs, using sdg and sdh, ok then.
now remove sdg. (second original leg of mirror)
extend mirror with sdi
convert to mirror again with sdh and sdi
fs seems to go readonly in a minute or two, with the dmesg errors shown in the notes
Thanks,
Nikhil.
Comment 3Heinz Mauelshagen
2018-05-02 13:09:33 UTC
"lvconvert -m1 ..." does not repair the RaidLV.
It is not the recommended way to deal with such failures (see lvmraid(7) sections related to (automatic) repair and replace devices).
Please use "lvconvert --repair ..." after adding enough PVs (at least one in this case) providing enough space to allocate new SubLVs to replace the failed ones.
You can control with "lvs -ao+devices $vg", which lists the SubLVs.
The filesystem going read-only is a result of "lvconvert -m1 ..." iterations in your test _not_ fixing the degraded raid1 LV loosing data when the last raid1 leg goes offline.
Comment 4Heinz Mauelshagen
2018-05-02 13:12:50 UTC
Created attachment 1430053[details]
Test script shoing the non-repairing behaviour of "lvconvert -m1 ..." in this use case
Description of problem: if we create raid lv with sdf and sdg. remove sdf. (offline) seems fine. extend vg with sdh and convert to mirror again of 2 legs, using sdg and sdh, ok then. now remove sdg. (second original leg of mirror) extend mirror with sdi convert to mirror again with sdh and sdi fs seems to go readonly in a minute or two, with the dmesg errors shown in the notes Version-Release number of selected component (if applicable): vm2-2.02.171-8.el7.x86_64 lvm2-libs-2.02.171-8.el7.x86_64 How reproducible: see additional info Steps to Reproduce: see additional info Additional info: reproduces with 4 new devices sdf,sdg,sdh and sdi, Disk /dev/sdf: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdg: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdh: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sdi: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes [root@vm253-107 ~]# pvcreate /dev/sdf /dev/sdg /dev/sdh /dev/sdi WARNING: Not using lvmetad because a repair command was run. WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. Physical volume "/dev/sdf" successfully created. Physical volume "/dev/sdg" successfully created. Physical volume "/dev/sdh" successfully created. Physical volume "/dev/sdi" successfully created. [root@vm253-107 ~]# vgcreate vg_mirror1 /dev/sdf /dev/sdg WARNING: Not using lvmetad because a repair command was run. WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. Volume group "vg_mirror1" successfully created [root@vm253-107 ~]# vgs WARNING: Not using lvmetad because a repair command was run. VG #PV #LV #SN Attr VSize VFree rhel 1 5 0 wz--n- <17.50g <5.50g vg_mirror1 2 0 0 wz--n- 1.99g 1.99g [root@vm253-107 ~]# lvcreate -L 500M -m 1 -n lv_mirror1 vg_mirror1 WARNING: Not using lvmetad because a repair command was run. Logical volume "lv_mirror1" created. [root@vm253-107 ~]# lvs WARNING: Not using lvmetad because a repair command was run. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-a-r--- 500.00m 100.00 [root@vm253-107 ~]# mkfs.xfs /dev/mapper/vg_mirror1-lv_mirror1 meta-data=/dev/mapper/vg_mirror1-lv_mirror1 isize=512 agcount=4, agsize=32000 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=128000, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@vm253-107 ~]# mkdir /formount1 [root@vm253-107 ~]# mount /dev/mapper/vg_mirror1-lv_mirror1 /formount1/ [root@vm253-107 ~]# touch /formount1/aa [root@vm253-107 ~]# echo "offline" > /sys/block/sdf/device/state [root@vm253-107 ~]# lvs WARNING: Not using lvmetad because a repair command was run. Couldn't find device with uuid ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-p- 500.00m 100.00 [root@vm253-107 ~]# vgs WARNING: Not using lvmetad because a repair command was run. Couldn't find device with uuid ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A. VG #PV #LV #SN Attr VSize VFree rhel 1 5 0 wz--n- <17.50g <5.50g vg_mirror1 2 1 0 wz-pn- 1.99g <1.01g [root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1 /dev/sda3 /dev/sdg [unknown] [root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1 WARNING: Device for PV ZX2TFa-LSTJ-5gPj-Natq-ziLx-cgTm-B78s1A not found or rejected by a filter. WARNING: Couldn't find all devices for LV vg_mirror1/lv_mirror1_rimage_0 while checking used and assumed devices. WARNING: Couldn't find all devices for LV vg_mirror1/lv_mirror1_rmeta_0 while checking used and assumed devices. Wrote out consistent volume group vg_mirror1. [root@vm253-107 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel 1 5 0 wz--n- <17.50g <5.50g vg_mirror1 1 1 0 wz--n- 1020.00m 516.00m [root@vm253-107 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# pvs WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. PV VG Fmt Attr PSize PFree /dev/sda3 rhel lvm2 a-- <17.50g <5.50g /dev/sdc [unknown] lvm2 u-- 1.00g 0 /dev/sdd lvm2 --- 1.00g 1.00g /dev/sde lvm2 --- 1.00g 1.00g /dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m /dev/sdh lvm2 --- 1.00g 1.00g /dev/sdi lvm2 --- 1.00g 1.00g [root@vm253-107 ~]# touch /formount1/aaa [root@vm253-107 ~]# ls /formount1/ aa aaa [root@vm253-107 ~]# vgextend vg_mirror1 /dev/sdh WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. Volume group "vg_mirror1" successfully extended [root@vm253-107 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel 1 5 0 wz--n- <17.50g <5.50g vg_mirror1 2 1 0 wz--n- 1.99g 1.50g [root@vm253-107 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 (reverse-i-search)`cat': ^Ct /sys/block/sde/device/state [root@vm253-107 ~]# ls /formount1/ aa aaa [root@vm253-107 ~]# lvconvert -m 1 /dev/mapper/vg_mirror1-lv_mirror1 /dev/sdg /dev/sdh Are you sure you want to convert raid1 LV vg_mirror1/lv_mirror1 to 2 images enhancing resilience? [y/n]: y WARNING: vg_mirror1/lv_mirror1 already has image count of 2. Logical volume vg_mirror1/lv_mirror1 successfully converted. [root@vm253-107 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# pvs WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. PV VG Fmt Attr PSize PFree /dev/sda3 rhel lvm2 a-- <17.50g <5.50g /dev/sdc [unknown] lvm2 u-- 1.00g 0 /dev/sdd lvm2 --- 1.00g 1.00g /dev/sde lvm2 --- 1.00g 1.00g /dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m /dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m /dev/sdi lvm2 --- 1.00g 1.00g [root@vm253-107 ~]# ls /formount1/ aa aaa [root@vm253-107 ~]# pvs WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. PV VG Fmt Attr PSize PFree /dev/sda3 rhel lvm2 a-- <17.50g <5.50g /dev/sdc [unknown] lvm2 u-- 1.00g 0 /dev/sdd lvm2 --- 1.00g 1.00g /dev/sde lvm2 --- 1.00g 1.00g /dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m /dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m /dev/sdi lvm2 --- 1.00g 1.00g [root@vm253-107 ~]# echo "offline" > /sys/block/sdg/device/state [root@vm253-107 ~]# lvs /dev/sdg: open failed: No such device or address LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# vgs /dev/sdg: open failed: No such device or address VG #PV #LV #SN Attr VSize VFree rhel 1 5 0 wz--n- <17.50g <5.50g vg_mirror1 2 1 0 wz--n- 1.99g 1.50g [root@vm253-107 ~]# vgextend vg_mirror1 /dev/sdi /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error /dev/sdg: open failed: No such device or address WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address WARNING: Failed to write an MDA of VG vg_mirror1. Volume group "vg_mirror1" successfully extended /dev/sdg: open failed: No such device or address [root@vm253-107 ~]# vgreduce --removemissing --force vg_mirror1 /dev/sdg: open failed: No such device or address Volume group "vg_mirror1" is already consistent. [root@vm253-107 ~]# lvs /dev/sdg: open failed: No such device or address LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# pvs /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error /dev/sdg: open failed: No such device or address WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. PV VG Fmt Attr PSize PFree /dev/sda3 rhel lvm2 a-- <17.50g <5.50g /dev/sdc [unknown] lvm2 u-- 1.00g 0 /dev/sdd lvm2 --- 1.00g 1.00g /dev/sde lvm2 --- 1.00g 1.00g /dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m /dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m /dev/sdi vg_mirror1 lvm2 a-- 1020.00m 1020.00m [root@vm253-107 ~]# ls /formount1/ aa aaa [root@vm253-107 ~]# ^C [root@vm253-107 ~]# lvconvert -m 1 /dev/mapper/vg_mirror1-lv_mirror1 /dev/sdh /dev/sdi /dev/sdg: open failed: No such device or address Are you sure you want to convert raid1 LV vg_mirror1/lv_mirror1 to 2 images enhancing resilience? [y/n]: y WARNING: vg_mirror1/lv_mirror1 already has image count of 2. Logical volume vg_mirror1/lv_mirror1 successfully converted. [root@vm253-107 ~]# lvs /dev/sdg: open failed: No such device or address LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# pvs /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error /dev/sdg: open failed: No such device or address WARNING: PV /dev/sdc is marked in use but no VG was found using it. WARNING: PV /dev/sdc might need repairing. PV VG Fmt Attr PSize PFree /dev/sda3 rhel lvm2 a-- <17.50g <5.50g /dev/sdc [unknown] lvm2 u-- 1.00g 0 /dev/sdd lvm2 --- 1.00g 1.00g /dev/sde lvm2 --- 1.00g 1.00g /dev/sdg vg_mirror1 lvm2 a-- 1020.00m 516.00m /dev/sdh vg_mirror1 lvm2 a-- 1020.00m 1020.00m /dev/sdi vg_mirror1 lvm2 a-- 1020.00m 1020.00m [root@vm253-107 ~]# ls /formount1/ aa aaa [root@vm253-107 ~]# lvs /dev/sdg: open failed: No such device or address LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# lvchange --refresh vg_mirror1/lv_mirror1 /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address /dev/sdg: open failed: No such device or address Failed to remove temporary SubLVs from vg_mirror1/lv_mirror1 [root@vm253-107 ~]# ls /formount1/ aa aaa [root@vm253-107 ~]# ls /formount1/aa aa aaa [root@vm253-107 ~]# ls /formount1/aa* /formount1/aa /formount1/aaa [root@vm253-107 ~]# lvs /dev/sdg: open failed: No such device or address LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# vgs /dev/sdg: open failed: No such device or address VG #PV #LV #SN Attr VSize VFree rhel 1 5 0 wz--n- <17.50g <5.50g vg_mirror1 3 1 0 wz--n- <2.99g <2.50g [root@vm253-107 ~]# lvs /dev/sdg: open failed: No such device or address LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# date Tue Apr 24 10:40:36 IST 2018 [root@vm253-107 ~]# lvs /dev/sdg: open failed: No such device or address LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel -wi-ao---- 2.00g root rhel -wi-ao---- 2.00g tmp rhel -wi-ao---- 2.00g usr rhel -wi-ao---- 4.00g var rhel -wi-ao---- 2.00g lv_mirror1 vg_mirror1 rwi-aor-r- 500.00m 100.00 [root@vm253-107 ~]# ls /formount1/aa* /formount1/aa /formount1/aaa [root@vm253-107 ~]# touch /formount1/aaas few minutes later, [root@vm253-107 log]# vgs WARNING: Not using lvmetad because a repair command was run. /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 0: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524222464: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 524279808: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 512 at 4096: Input/output error /dev/vg_mirror1/lv_mirror1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid r7SDmy-Kv6T-sZop-hWoA-l5zm-c17m-DxCzuL. VG #PV #LV #SN Attr VSize VFree rhel 1 5 0 wz--n- <17.50g <5.50g vg_mirror1 3 1 0 wz-pn- <2.99g <2.50g [root@vm253-107 log]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/rhel-root 1998672 52720 1824712 3% / devtmpfs 458356 0 458356 0% /dev tmpfs 474096 0 474096 0% /dev/shm tmpfs 474096 48932 425164 11% /run tmpfs 474096 0 474096 0% /sys/fs/cgroup /dev/mapper/rhel-usr 3997376 3062072 709208 82% /usr /dev/sda1 499656 131476 331484 29% /boot /dev/mapper/rhel-var 1998672 220004 1657428 12% /var /dev/mapper/rhel-tmp 1998672 6236 1871196 1% /tmp /dev/mapper/rhel-home 1998672 11824 1865608 1% /home tmpfs 94820 4 94816 1% /run/user/992 tmpfs 94820 36 94784 1% /run/user/1000 tmpfs 94820 0 94820 0% /run/user/0 /dev/mapper/vg_mirror1-lv_mirror1 508580 25776 482804 6% /formount1 [root@vm253-107 log]# touch /formount1/aaaa touch: cannot touch ‘/formount1/aaaa’: Input/output error [root@vm253-107 log]# date Tue Apr 24 10:43:50 IST 2018 [root@vm253-107 log]# dmesg logs show: [8958506.464323] md/raid1:mdX: active with 2 out of 2 mirrors [8958506.464420] dm-14: detected capacity change from 0 to 524288000 [8958506.505924] mdX: bitmap file is out of date, doing full recovery [8958506.551661] md: resync of RAID array mdX [8958519.309709] md: mdX: resync done. [8958550.419780] XFS (dm-14): Mounting V5 Filesystem [8958550.510750] XFS (dm-14): Ending clean mount [8958580.536165] sd 2:0:0:9: rejecting I/O to offline device [8958580.536217] blk_update_request: I/O error, dev sdf, sector 0 [8958580.536417] sd 2:0:0:9: rejecting I/O to offline device [8958580.536420] blk_update_request: I/O error, dev sdf, sector 0 [8958580.536439] sd 2:0:0:9: rejecting I/O to offline device [8958580.536444] md: super_written gets error=-5, uptodate=0 [8958580.536452] md/raid1:mdX: Disk failure on dm-11, disabling device. md/raid1:mdX: Operation continuing on 1 devices. [8958604.877385] device-mapper: raid: Failed to read superblock of device at position -1 [8958852.793906] sd 2:0:0:8: rejecting I/O to offline device [8958852.794128] sd 2:0:0:8: rejecting I/O to offline device [8958852.794151] sd 2:0:0:8: rejecting I/O to offline device [8958852.794176] sd 2:0:0:8: rejecting I/O to offline device [8958852.794185] sd 2:0:0:8: rejecting I/O to offline device [8958885.687397] sd 2:0:0:8: rejecting I/O to offline device [8958885.687617] sd 2:0:0:8: rejecting I/O to offline device [8958885.687644] sd 2:0:0:8: rejecting I/O to offline device [8958885.687660] sd 2:0:0:8: rejecting I/O to offline device [8958885.687688] sd 2:0:0:8: rejecting I/O to offline device [8958891.580780] sd 2:0:0:8: rejecting I/O to offline device [8958891.580960] sd 2:0:0:8: rejecting I/O to offline device [8958891.580984] sd 2:0:0:8: rejecting I/O to offline device [8958891.580998] sd 2:0:0:8: rejecting I/O to offline device [8958891.581025] sd 2:0:0:8: rejecting I/O to offline device [8958973.771215] sd 2:0:0:8: rejecting I/O to offline device [8958973.771397] sd 2:0:0:8: rejecting I/O to offline device [8958973.771418] sd 2:0:0:8: rejecting I/O to offline device [8958973.771430] sd 2:0:0:8: rejecting I/O to offline device [8958973.771438] sd 2:0:0:8: rejecting I/O to offline device [8959004.986828] sd 2:0:0:8: rejecting I/O to offline device [8959004.986846] blk_update_request: I/O error, dev sdg, sector 0 [8959004.986861] sd 2:0:0:8: rejecting I/O to offline device [8959004.986869] md: super_written gets error=-5, uptodate=0 <---- ?? [8959004.987414] device-mapper: raid: Failed to read superblock of device at position -1 <----- ?? [root@vm253-107 ~]# ============================================================================== relevant mails to engineering ---------- Forwarded message ---------- From: Heinz Mauelshagen <heinzm> Date: Thu, Apr 26, 2018 at 7:52 PM Subject: Re: Fwd: issues with raid type , fs goes readonly ? To: Nikhil Kshirsagar <nkshirsa> On 04/25/2018 11:21 AM, Nikhil Kshirsagar wrote: Hello Heinz, please could you check the reproducer steps, is this a bug, shall we file a bz ? Yes, file a bz, please. And tell the customer that after extending the VG, "lvconvert --repair nvm/r" is the recomended way to fix such failures rather than "lvconvert -m...". Heinz On Tue, Apr 24, 2018 at 10:50 AM, Nikhil Kshirsagar <nkshirsa> wrote: Hi Heinz, I can reproduce this too on 7.4 , please take a look at my notes in the attachment. It happens when we first remove one leg of the raid lv (ok then), and then the second one.. so if we create raid with sdf and sdg. remove sdf. (offline) seems fine. extend vg with sdh and convert to mirror again of 2 legs, using sdg and sdh, ok then. now remove sdg. (second original leg of mirror) extend mirror with sdi convert to mirror again with sdh and sdi fs seems to go readonly in a minute or two, with the dmesg errors shown in the notes Thanks, Nikhil.