Bug 1633167
| Summary: | lvm tried to deactivate subLV of raid1 lv in a tagged VG when Vg deactivated | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | nikhil kshirsagar <nkshirsa> | ||||
| Component: | lvm2 | Assignee: | Heinz Mauelshagen <heinzm> | ||||
| lvm2 sub component: | Mirroring and RAID | QA Contact: | cluster-qe <cluster-qe> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | urgent | ||||||
| Priority: | urgent | CC: | agk, cmarthal, heinzm, jbrassow, loberman, lvm-team, mcsontos, msnitzer, nkshirsa, nwahl, prajnoha, rbednar, rhandlin, yoliynyk, zkabelac | ||||
| Version: | 7.5 | ||||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | lvm2-2.02.184-1.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-08-06 13:10:41 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1577173 | ||||||
| Attachments: |
|
||||||
Looking over th BZ description - there is one issue standing out - when using 'cluster' (clvmd) and passing --config option with activation command (vgchnage) is unsupported combination as the 'clvmd' activation service is running autonomously from the command itself - going to take deeper analysis of the original issue. So I'm still unclear what is going on - but how has happened that _rmeta_ LVs are visible ?? I'm looking over the archive and it seems the LV itself was created on different node ? The core trouble seems to be, that _rmeta_ LVs are present with this tag - which likely failed to be checked on validation code - and code let it store in metadata (also in 'lvs -a' you can notice missing [] around the name). So is there something special going on during lvcreate ? I've not yet been able to reproduce creation of 'raid1' LV with visible _rmeta_ devices. There is definitely a bug in validation code - it's not checking internal LVs is invisible - so during run of command: vgchange -an all visible LVs are tried for deactivation and if _rmeta_ is visible and is used by raid1 LV itself - user will get error about problem with deactivation of LV which is openned. A workaround should be to 'vgcfgbackup' such VG - remove "VISIBLE" attribute from _rmeta_ LVs and vgcfgrestore them back. Also user should be always able to deactivate direct raid1 LV by name with 'lvchange -an vg/lv' command. lvm2 surely needs extension in validation of metadata - so trial to store '_rmeta_' is caught before it hits metadata storage. We need to see 'sos' report from all nodes - please provide attachemnt from all clustered nodes. The reported case is not reproducible with shown steps so far. Sos reports is needed for better analysis how the raid's _rmeta_ images were left visible in metadata. Provided single 'sos' report file just shows raid LV appeared there already invalid. So after verification - there really is during RAID creation as short moment, when raid LV is committed on this with VISIBLE flag - those _rmeta_ LVs are then zerod and new commit making them invisible follows shortly. However if there is 'crash' just in this tiny time window - raid LV is left in unknown state - user likely should remove such LV and create it again. This is seen as creation sequence bug in raidLV. If the customer has deadlocking issue - he should probably switch to use older '--type mirror' - where the order sequence with zeroing is correct. Upstream commit16ae968d24b4fe3264dc9b46063345ff2846957b to avoid commiting SubLVs for wipinng thus not causing them to turn remnant on crashs. (In reply to Zdenek Kabelac from comment #22) > So after verification - there really is during RAID creation as short > moment, when raid LV is committed on this with VISIBLE flag - those > _rmeta_ LVs are then zerod and new commit making them invisible follows > shortly. > > However if there is 'crash' just in this tiny time window - raid LV is > left in unknown state - user likely should remove such LV and create it > again. > > This is seen as creation sequence bug in raidLV. > > If the customer has deadlocking issue - he should probably switch to use > older '--type mirror' - where the order sequence with zeroing is correct. Switching to 'mirror' type is not mandatory with patch as of previous comment. The patch 16ae968d24b4fe3264dc9b46063345ff2846957b still seems to lack fixing the commit of raid metadata with visible _rmeta_ LVs. And the patch is also introducing other potential problem where the _rmeta_ LVs present in DM table can be misidentified as other type of block devices. While the very generic activation for wipe_fs was a bit 'annoying' we've been sure there is later zero chance to see this device being i.e. identified as some other device/fs/mdraid member. With clearing only 1st. sector as proposed patch is doing - we are leaving other signatures in place - so in case there is any issue with raid activation - such device can be possibly misused. To stay 'reasonably' secure we probably need to erase at least 64K from the front and end of the device - although there are weird filesystem like ZFS where its signature is stored (and later identified) in way more complicated way. It would be probably inefficient to clear whole _rmeta_ device though - since in case of large extent size we might end with erasing lots of space. Another slight advantage from previous 'activation' method was the option to introduce usage of TRIM ioctl in a very simple way for such device - although we already do provide support for trimming 'removed' PV space - so we might consider to generalize this concept of use it also for 'allocation' of new LVs. (In reply to Zdenek Kabelac from comment #25) > The patch 16ae968d24b4fe3264dc9b46063345ff2846957b still seems to lack > fixing the commit of raid metadata with visible _rmeta_ LVs. > > And the patch is also introducing other potential problem where the _rmeta_ > LVs present in DM table can be misidentified as other type of block devices. > The patch does not introduce the potential problem you describe anew at all! It keeps the semantics we had before (wiping one sector at the beginning). So your statement points at an enhancement request to reduce any potential bogus discoveries by e.g. libblkid even further from what we always allowed for. I can add an additional patch changing the given semantics after studying which signatures of what size we actually should take into consideration so that we can minimize wiping overhead. (In reply to Heinz Mauelshagen from comment #26) > (In reply to Zdenek Kabelac from comment #25) > > The patch 16ae968d24b4fe3264dc9b46063345ff2846957b still seems to lack > > fixing the commit of raid metadata with visible _rmeta_ LVs. > > > > And the patch is also introducing other potential problem where the _rmeta_ > > LVs present in DM table can be misidentified as other type of block devices. > > > > The patch does not introduce the potential problem you describe anew at all! > As of discussion with Zdenek: not using wipe_lv actually looses us wiping all known signatures, so we need to keep using that API for the time being. Reverting and coming up with a new patch... lvm2 upstream commit commit dd5716ddf258c4a44819fa90d3356833ccf767b4 Stable branch commit 9b04851fc574ce9cffd30a51d2b750955239f316 Created attachment 1586603 [details] test results Marking verified. Testing consisted of running the reproducer from initial comment (100 times) and raid sanity regression check for raid1 in singlenode environment. See attachment for logs and reproducer script. Regression run: https://beaker.cluster-qe.lab.eng.brq.redhat.com/bkr/jobs/96718 lvm2-2.02.185-2.el7.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2253 |
Description of problem: When a raid1 type lv is in a tagged VG, and we try to deactivate the VG, lvm tries to deactivate the underlying sublv (_rmeta_0 and _rmeta_1) while the raid1 lv is active, therefore causing failures in cluster resource agents and scripts. Version-Release number of selected component (if applicable): lvm2-2.02.177-4.el7.x86_64 Additional info: [nkshirsa@foobar sosreport-em50008.02175265-20180914100002]$ cat sos_commands/lvm2/vgs_-v_-o_vg_mda_count_vg_mda_free_vg_mda_size_vg_mda_used_count_vg_tags_--config_global_locking_type_0 Reloading config files Wiping internal VG cache WARNING: Locking disabled. Be careful! This could corrupt your metadata. VG Attr Ext #PV #LV #SN VSize VFree VG UUID VProfile #VMda VMdaFree VMdaSize #VMdaUse VG Tags lsv30013_rootvg wz--n- 4.00m 1 18 0 <35.51g 2.50g 1ayAcQ-PhDw-jK93-eZvV-BGqT-krAo-ED7XOm 1 501.50k 1020.00k 1 murex_raid_vg wz--n- 4.00m 2 3 0 339.99g 139.98g ptuQYT-7cnI-vspJ-zGWl-D3B9-K3b0-CPLKbt 2 505.50k 1020.00k 2 pacemaker Reloading config files Wiping internal VG cache [nkshirsa@foobar sosreport-em50008.02175265-20180914100002]$ cat sos_commands/lvm2/lvs_-a_-o_lv_tags_devices_--config_global_locking_type_0 WARNING: Locking disabled. Be careful! This could corrupt your metadata. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert LV Tags Devices lv_home lsv30013_rootvg -wi-ao---- 32.00m /dev/mapper/mpatha2(6458) lv_opt lsv30013_rootvg -wi-ao---- 1.00g /dev/mapper/mpatha2(6202) lv_opt_controlm lsv30013_rootvg -wi-ao---- 1.00g /dev/mapper/mpatha2(5946) lv_opt_manage lsv30013_rootvg -wi-ao---- 32.00m /dev/mapper/mpatha2(5938) lv_opt_managesoft lsv30013_rootvg -wi-ao---- 200.00m /dev/mapper/mpatha2(5888) lv_opt_microsoft lsv30013_rootvg -wi-ao---- 256.00m /dev/mapper/mpatha2(5760) lv_opt_tivoli_cit lsv30013_rootvg -wi-ao---- 256.00m /dev/mapper/mpatha2(5824) lv_root lsv30013_rootvg -wi-ao---- 768.00m /dev/mapper/mpatha2(6466) lv_swap lsv30013_rootvg -wi-ao---- 2.00g /dev/mapper/mpatha2(6658) lv_tmp lsv30013_rootvg -wi-ao---- 5.00g /dev/mapper/mpatha2(4480) lv_usr lsv30013_rootvg -wi-ao---- 9.00g /dev/mapper/mpatha2(2176) lv_usr_local lsv30013_rootvg -wi-ao---- 256.00m /dev/mapper/mpatha2(2112) lv_var lsv30013_rootvg -wi-ao---- 3.00g /dev/mapper/mpatha2(1344) lv_var_crash lsv30013_rootvg -wi-ao---- 7.00g /dev/mapper/mpatha2(832) lv_var_crash lsv30013_rootvg -wi-ao---- 7.00g /dev/mapper/mpatha2(7170) lv_var_log lsv30013_rootvg -wi-ao---- 2.00g /dev/mapper/mpatha2(288) lv_var_log_audit lsv30013_rootvg -wi-ao---- 1.00g /dev/mapper/mpatha2(32) lv_var_opt_besclient lsv30013_rootvg -wi-ao---- 128.00m /dev/mapper/mpatha2(800) lv_var_opt_managesoft lsv30013_rootvg -wi-ao---- 128.00m /dev/mapper/mpatha2(0) lv_murex_raid1 murex_raid_vg rwi-a-r--- 100.00g 100.00 lv_murex_raid1_rimage_0(0),lv_murex_raid1_rimage_1(0) [lv_murex_raid1_rimage_0] murex_raid_vg iwi-aor--- 100.00g /dev/mapper/mpathb(1) [lv_murex_raid1_rimage_1] murex_raid_vg iwi-aor--- 100.00g /dev/mapper/mpathc(1) lv_murex_raid1_rmeta_0 murex_raid_vg ewi-aor--- 4.00m /dev/mapper/mpathb(0) lv_murex_raid1_rmeta_1 murex_raid_vg ewi-aor--- 4.00m /dev/mapper/mpathc(0) [nkshirsa@foobar sosreport-em50008.02175265-20180914100002]$ script that customer ran was # pcs resource disable murex_grp # vgchange -anvvv murex_raid_vg # vgchange --deltag pacemaker murex_raid_vg Reply with results. # vgchange --addtag pacemaker murex_raid_vg # vgchange -ayvvv murex_raid_vg --config 'activation { volume_list = [ "@pacemaker" ]}' # vgchange -anvvv murex_raid_vg ]0;root@lsv30013:/home/em50008[root@lsv30013 em50008]# vgchange -anvvv murex_raid_vg[1@ [1@- Parsing: vgchange -an -vvv murex_raid_vg Recognised command vgchange_activate (id 116 / enum 98). devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ] Setting global/locking_type to 1 ... Processing command: vgchange -an -vvv murex_raid_vg Deactivated 3 logical volumes in volume group murex_raid_vg Getting device info for murex_raid_vg-lv_murex_raid1 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtyX8RBceOdQpc1TzhNrPBGNgv5fWFDTGv]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtyX8RBceOdQpc1TzhNrPBGNgv5fWFDTGv [ noopencount flush ] [16384] (*1) Getting device info for murex_raid_vg-lv_murex_raid1_rmeta_0 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtI1aB5gfkZghsmJHKjcxOP7jqV6Kn7v75]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtI1aB5gfkZghsmJHKjcxOP7jqV6Kn7v75 [ noopencount flush ] [16384] (*1) Getting device info for murex_raid_vg-lv_murex_raid1_rmeta_1 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtczgD1345cAYx7nNAFGKDwntWENDxADy5]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtczgD1345cAYx7nNAFGKDwntWENDxADy5 [ noopencount flush ] [16384] (*1) Counted 0 active LVs in VG murex_raid_vg 0 logical volume(s) in volume group "murex_raid_vg" now active Completed: vgchange -an -vvv murex_raid_vg ]0;root@lsv30013:/home/em50008[root@lsv30013 em50008]# [K[root@lsv30013 em50008]# vgchange --deltag pacemaker murex_raid_vg Volume group "murex_raid_vg" successfully changed ]0;root@lsv30013:/home/em50008[root@lsv30013 em50008]# vgchange --addtag pacemaker murex_raid_vg Volume group "murex_raid_vg" successfully changed ]0;root@lsv30013:/home/em50008[root@lsv30013 em50008]# vgchange -ayvvv murex_raid_vg --config 'activation { volume_list = [ "@pacemaker" Parsing: vgchange -ay -vvv murex_raid_vg --config 'activation { volume_list = [ "@pacemaker" ]}' Recognised command vgchange_activate (id 116 / enum 98). Reloading config files Syncing device names Wiping internal VG cache Freeing VG #orphans_lvm1 at 0x5646a23883f0. Freeing VG #orphans_pool at 0x5646a238cc10. Freeing VG #orphans_lvm2 at 0x5646a2391430. ... ... Processing command: vgchange -ay -vvv murex_raid_vg --config 'activation { volume_list = [ "@pacemaker" ]}' ... ... murex_raid_vg-lv_murex_raid1: Skipping NODE_ADD (253,27) 0:6 0660 [trust_udev] murex_raid_vg-lv_murex_raid1: Processing NODE_READ_AHEAD 8192 (flags=1) murex_raid_vg-lv_murex_raid1 (253:27): read ahead is 256 murex_raid_vg-lv_murex_raid1 (253:27): Setting read ahead to 8192 Activated 3 logical volumes in volume group murex_raid_vg ... ... ]0;root@lsv30013:/home/em50008[root@lsv30013 em50008]# vgchange -anvvv murex_raid_vg[C[C[1@ [1@- Parsing: vgchange -an -vvv murex_raid_vg ... Processing command: vgchange -an -vvv murex_raid_vg ... ... /dev/mapper/murex_raid_vg-lv_murex_raid1_rmeta_0: Aliased to /dev/dm-23 in device cache (preferred name) (253:23) /dev/murex_raid_vg/lv_murex_raid1_rmeta_0: Aliased to /dev/mapper/murex_raid_vg-lv_murex_raid1_rmeta_0 in device cache (preferred name) (253:23) /dev/dm-24: Added to device cache (253:24) /dev/mapper/murex_raid_vg-lv_murex_raid1_rimage_0: Aliased to /dev/dm-24 in device cache (preferred name) (253:24) /dev/dm-25: Added to device cache (253:25) /dev/mapper/murex_raid_vg-lv_murex_raid1_rmeta_1: Aliased to /dev/dm-25 in device cache (preferred name) (253:25) /dev/murex_raid_vg/lv_murex_raid1_rmeta_1: Aliased to /dev/mapper/murex_raid_vg-lv_murex_raid1_rmeta_1 in device cache (preferred name) (253:25) /dev/dm-26: Added to device cache (253:26) /dev/mapper/murex_raid_vg-lv_murex_raid1_rimage_1: Aliased to /dev/dm-26 in device cache (preferred name) (253:26) /dev/dm-27: Added to device cache (253:27) /dev/disk/by-id/dm-name-murex_raid_vg-lv_murex_raid1: Aliased to /dev/dm-27 in device cache (preferred name) (253:27) /dev/disk/by-id/dm-uuid-LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtyX8RBceOdQpc1TzhNrPBGNgv5fWFDTGv: Aliased to /dev/disk/by-id/dm-name-murex_raid_vg-lv_murex_raid1 in device cache (253:27) /dev/disk/by-uuid/7931b442-613f-41eb-aba4-e04db87ddac3: Aliased to /dev/disk/by-id/dm-name-murex_raid_vg-lv_murex_raid1 in device cache (253:27) /dev/mapper/murex_raid_vg-lv_murex_raid1: Aliased to /dev/disk/by-id/dm-name-murex_raid_vg-lv_murex_raid1 in device cache (preferred name) (253:27) /dev/murex_raid_vg/lv_murex_raid1: Aliased to /dev/mapper/murex_raid_vg-lv_murex_raid1 in device cache (preferred name) (253:27) /dev/dm-3: Added to device cache (253:3) ... ... /dev/mapper/mpathc: Found metadata at 69632 size 2711 with wrap 0 (in area at 4096 size 1044480) for murex_raid_vg (ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbt) lvmcache has no info for vgname "murex_raid_vg" with VGID ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbt. lvmcache has no info for vgname "murex_raid_vg". lvmcache /dev/mapper/mpathc: now in VG murex_raid_vg with 1 mda(s). lvmcache /dev/mapper/mpathc: VG murex_raid_vg: set VGID to ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbt. lvmcache /dev/mapper/mpathc: VG murex_raid_vg: set creation host to lsv30013.linux.internalcorp.net. lvmcache /dev/mapper/mpathc: VG murex_raid_vg: stored metadata checksum 0x9537abd9 with size 2711. ... .. /dev/mapper/mpathb: Using cached metadata at 69632 size 2711 with wrap 0 (in area at 4096 size 1044480) for murex_raid_vg (ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbt) lvmcache /dev/mapper/mpathb: now in VG murex_raid_vg (ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbt) with 1 mda(s). ... ... Stack murex_raid_vg/lv_murex_raid1:0[0] on LV murex_raid_vg/lv_murex_raid1_rmeta_0:0. Adding murex_raid_vg/lv_murex_raid1:0 as an user of murex_raid_vg/lv_murex_raid1_rmeta_0. Stack murex_raid_vg/lv_murex_raid1:0[0] on LV murex_raid_vg/lv_murex_raid1_rimage_0:0. Adding murex_raid_vg/lv_murex_raid1:0 as an user of murex_raid_vg/lv_murex_raid1_rimage_0. Stack murex_raid_vg/lv_murex_raid1:0[1] on LV murex_raid_vg/lv_murex_raid1_rmeta_1:0. Adding murex_raid_vg/lv_murex_raid1:0 as an user of murex_raid_vg/lv_murex_raid1_rmeta_1. Stack murex_raid_vg/lv_murex_raid1:0[1] on LV murex_raid_vg/lv_murex_raid1_rimage_1:0. Adding murex_raid_vg/lv_murex_raid1:0 as an user of murex_raid_vg/lv_murex_raid1_rimage_1. Read murex_raid_vg metadata (16) from /dev/mapper/mpathc at 69632 size 2711 Widening request for 512 bytes at 4096 to 4096 bytes at 4096 on /dev/mapper/mpathb (for VG metadata header) Read /dev/mapper/mpathb: 4096 bytes (sync) at 4096 (for VG metadata header) Widening request for 130 bytes at 69632 to 4096 bytes at 69632 on /dev/mapper/mpathb (for VG metadata content) Read /dev/mapper/mpathb: 4096 bytes (sync) at 69632 (for VG metadata content) Widening request for 2711 bytes at 69632 to 4096 bytes at 69632 on /dev/mapper/mpathb (for VG metadata content) Read /dev/mapper/mpathb: 4096 bytes (sync) at 69632 (for VG metadata content) Skipped reading metadata from /dev/mapper/mpathb at 69632 size 2711 with matching checksum. lvmcache /dev/mapper/mpathb: VG murex_raid_vg: set system_id to . lvmcache: VG murex_raid_vg (ptuQYT-7cnI-vspJ-zGWl-D3B9-K3b0-CPLKbt) stored (2711 bytes). /dev/mapper/mpathb: using cached size 356515840 sectors /dev/mapper/mpathc: using cached size 356515840 sectors /dev/mapper/mpathb 0: 0 1: lv_murex_raid1_rmeta_0(0:0) /dev/mapper/mpathb 1: 1 25600: lv_murex_raid1_rimage_0(0:0) /dev/mapper/mpathb 2: 25601 17918: NULL(0:0) /dev/mapper/mpathc 0: 0 1: lv_murex_raid1_rmeta_1(0:0) /dev/mapper/mpathc 1: 1 25600: lv_murex_raid1_rimage_1(0:0) /dev/mapper/mpathc 2: 25601 17918: NULL(0:0) Processing VG murex_raid_vg ptuQYT-7cnI-vspJ-zGWl-D3B9-K3b0-CPLKbt Getting device info for murex_raid_vg-lv_murex_raid1 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtyX8RBceOdQpc1TzhNrPBGNgv5fWFDTGv]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtyX8RBceOdQpc1TzhNrPBGNgv5fWFDTGv [ opencount flush ] [16384] (*1) Getting device info for murex_raid_vg-lv_murex_raid1_rmeta_0 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtI1aB5gfkZghsmJHKjcxOP7jqV6Kn7v75]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtI1aB5gfkZghsmJHKjcxOP7jqV6Kn7v75 [ opencount flush ] [16384] (*1) Getting device info for murex_raid_vg-lv_murex_raid1_rmeta_1 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtczgD1345cAYx7nNAFGKDwntWENDxADy5]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtczgD1345cAYx7nNAFGKDwntWENDxADy5 [ opencount flush ] [16384] (*1) Counted 2 open LVs in VG murex_raid_vg. Getting device info for murex_raid_vg-lv_murex_raid1 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtyX8RBceOdQpc1TzhNrPBGNgv5fWFDTGv]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtyX8RBceOdQpc1TzhNrPBGNgv5fWFDTGv [ opencount flush ] [16384] (*1) Getting device info for murex_raid_vg-lv_murex_raid1_rmeta_0 [LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtI1aB5gfkZghsmJHKjcxOP7jqV6Kn7v75]. dm info LVM-ptuQYT7cnIvspJzGWlD3B9K3b0CPLKbtI1aB5gfkZghsmJHKjcxOP7jqV6Kn7v75 [ opencount flush ] [16384] (*1) Logical volume murex_raid_vg/lv_murex_raid1_rmeta_0 is used by another device. Can't deactivate volume group "murex_raid_vg" with 2 open logical volume(s) Unlock: Memlock counters: prioritized:0 locked:0 critical:0 daemon:0 suspended:0 Syncing device namesca lvmcache: VG murex_raid_vg wiped. Unlocking /run/lock/lvm/V_murex_raid_vg _undo_flock /run/lock/lvm/V_murex_raid_vg Closed /dev/mapper/mpathc Closed /dev/mapper/mpathb Freeing VG murex_raid_vg at 0x564dc9540580. Setting global/notify_dbus to 1 Completed: vgchange -an -vvv murex_raid_vg