Red Hat Bugzilla – Bug 834327
Virtual snapshot does not get removed after removal attempt on a busy device
Last modified: 2013-11-21 18:17:25 EST
Description of problem: The lvremove does not actually remove a virtual snapshot after attempted to remove such a LV with a busy device. lvremove finishes successfuly but the _vorigin remains. This causes later LV creation of a same named virt snapshot to fail. Version-Release number of selected component (if applicable): lvm2-2.02.95-10.el6.x86_64 lvm2-libs-2.02.95-10.el6.x86_64 lvm2-cluster-2.02.95-10.el6.x86_64 cmirror-2.02.95-10.el6.x86_64 device-mapper-1.02.74-10.el6.x86_64 device-mapper-event-1.02.74-10.el6.x86_64 device-mapper-libs-1.02.74-10.el6.x86_64 device-mapper-event-libs-1.02.74-10.el6.x86_64 How reproducible: Every time Steps to Reproduce: 1. Create LV lvcreate -n badboy -L 1g --virtualsize 1g vg_virts 2. Create some load on the newly created LV: xxd /dev/vg_virts/badboy 3. Try to remove the LV: lvremove -f vg_virts/badboy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy Unable to deactivate vg_virts-badboy (253:5) Unable to deactivate logical volume "badboy" Stop the load and remove the LV: lvremove -f vg_virts/badboy Logical volume "badboy" successfully removed dmsetup ls vg_virts-badboy_vorigin (253:6) vg_virts-c1--node01 (253:4) vg_virts-disk (253:2) vg_systemgroup-lv_home (253:1) vg_virts-disk2 (253:3) vg_systemgroup-lv_root (253:0) Try to create it again with the same name: lvcreate -n badboy -L 1g --virtualsize 1g vg_virts Unable to create LV badboy_vorigin in Volume Group vg_virts: name already in use. Couldn't create virtual origin for LV badboy Actual results: The creation fails because _vorigin remains: lvs -a | grep bad [badboy_vorigin] vg_virts vwi-a-v- 1.00g Expected results: lvremove should properly remove all the remnants of the LV.
So this is trying to remove the LV while it is still in use? Does the current release have the same bug?
I can still reproduce this: [root@virt-011 ~]# lvcreate -n badboy -L 1g --virtualsize 1g vg Logical volume "badboy" created [root@virt-011 ~]# for i in {1..1000}; do dd if=/dev/urandom of=/dev/vg/badboy bs=10240 count=10000 >& out.log; done & [1] 5920 [root@virt-011 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert badboy vg swi-aos--- 1.00g [badboy_vorigin] 9.57 [badboy_vorigin] vg owi-a-s--- 1.00g lv_root vg_virt011 -wi-ao---- 6.71g lv_swap vg_virt011 -wi-ao---- 816.00m [root@virt-011 ~]# lvremove -f vg/badboy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy device-mapper: remove ioctl on failed: Device or resource busy Unable to deactivate vg-badboy (253:2) Unable to deactivate logical volume "badboy" [root@virt-011 ~]# kill %1 [root@virt-011 ~]# [1]+ Terminated for i in {1..1000}; do dd if=/dev/urandom of=/dev/vg/badboy bs=10240 count=10000 &>out.log; done [root@virt-011 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert badboy vg -wi-a----- 1.00g [badboy_vorigin] vg vwi-a-v--- 1.00g lv_root vg_virt011 -wi-ao---- 6.71g lv_swap vg_virt011 -wi-ao---- 816.00m Now trying the removal: [root@virt-011 ~]# lvremove -f vg/badboy Logical volume "badboy" successfully removed [root@virt-011 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert [badboy_vorigin] vg vwi-a-v--- 1.00g lv_root vg_virt011 -wi-ao---- 6.71g lv_swap vg_virt011 -wi-ao---- 816.00m [root@virt-011 ~]# dmsetup ls vg-badboy_vorigin (253:3) vg_virt011-lv_swap (253:1) vg_virt011-lv_root (253:0) [root@virt-011 ~]#
Ehm.. Forgot to write which versions I used: lvm2-2.02.100-5.el6.x86_64 lvm2-libs-2.02.100-5.el6.x86_64 device-mapper-1.02.79-5.el6.x86_64 device-mapper-libs-1.02.79-5.el6.x86_64 device-mapper-event-1.02.79-5.el6.x86_64 kernel-2.6.32-422.el6.x86_64
Peter is going to try a simple patch - to reinsert a "is the device open?" check (before the command changes anything) that got taken out a while back, with a new wait loop around it in case it's only been open temporarily by udev.
This has been recently fixed upstream: https://www.redhat.com/archives/lvm-devel/2013-October/msg00043.html It's a kind of workaround - there is a still hidden race - however it's currently hard to fix this with current libdm code - it needs more thinking. But the reported problem should be handled.
Also requires a little retry loop around the open to avoid race with any asynchronous udev processing. prajnoha has patch for that
(In reply to Alasdair Kergon from comment #9) > Also requires a little retry loop around the open to avoid race with any > asynchronous udev processing. https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=48df36b8c531a43da8602b50759c9e841e1d9f44
Tested, and marking as VERIFIED with lvm2-2.02.100-6.el6. [root@virt-013 ~]# lvcreate -n badboy -L 1g --virtualsize 1g vg Logical volume "badboy" created added load [root@virt-013 ~]# lvremove -f vg/badboy Logical volume vg/badboy in use. [root@virt-013 ~]# kill %1 [root@virt-013 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert badboy vg swi-a-s--- 1.00g [badboy_vorigin] 9.57 [badboy_vorigin] vg owi-a-s--- 1.00g lv_root vg_virt013 -wi-ao---- 6.71g lv_swap vg_virt013 -wi-ao---- 816.00m [root@virt-013 ~]# lvremove -f vg/badboy Logical volume "badboy" successfully removed [root@virt-013 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert lv_root vg_virt013 -wi-ao---- 6.71g lv_swap vg_virt013 -wi-ao---- 816.00m [root@virt-013 ~]# dmsetup ls vg_virt013-lv_swap (253:1) vg_virt013-lv_root (253:0)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1704.html