Bug 834327 - Virtual snapshot does not get removed after removal attempt on a busy device
Virtual snapshot does not get removed after removal attempt on a busy device
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.3
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Peter Rajnoha
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-21 10:46 EDT by Nenad Peric
Modified: 2013-11-21 18:17 EST (History)
12 users (show)

See Also:
Fixed In Version: lvm2-2.02.100-6.el6
Doc Type: Bug Fix
Doc Text:
Virtual snapshot device removal failed if the device was still open. The lvremove left <virtual_snashot_name>_vorigin device-mapper device in the system. This device could not be removed with LVM tools (as it's internal LVM device) nor any other virtual snapshot with the <virtual_snapshot_name> name created again unless the <virtual_snapshot_name>_vorigin device was manually removed using dmsetup remove. This has been fixed by properly checking the LV open count status before proceeding with the removal operation.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-21 18:17:25 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nenad Peric 2012-06-21 10:46:59 EDT
Description of problem:

The lvremove does not actually remove a virtual snapshot after attempted to remove such a LV with a busy device.

lvremove finishes successfuly but the _vorigin remains. 
This causes later LV creation of a same named virt snapshot to fail. 



Version-Release number of selected component (if applicable):

lvm2-2.02.95-10.el6.x86_64
lvm2-libs-2.02.95-10.el6.x86_64
lvm2-cluster-2.02.95-10.el6.x86_64
cmirror-2.02.95-10.el6.x86_64
device-mapper-1.02.74-10.el6.x86_64
device-mapper-event-1.02.74-10.el6.x86_64
device-mapper-libs-1.02.74-10.el6.x86_64
device-mapper-event-libs-1.02.74-10.el6.x86_64

How reproducible:

Every time

Steps to Reproduce:
1. Create LV 

lvcreate -n badboy -L 1g --virtualsize 1g vg_virts

2. Create some load on the newly created LV:

xxd /dev/vg_virts/badboy

3.
  
Try to remove the LV:

lvremove -f vg_virts/badboy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  Unable to deactivate vg_virts-badboy (253:5)
  Unable to deactivate logical volume "badboy"

Stop the load and remove the LV:

lvremove -f vg_virts/badboy
  Logical volume "badboy" successfully removed

dmsetup ls
vg_virts-badboy_vorigin (253:6)
vg_virts-c1--node01     (253:4)
vg_virts-disk   (253:2)
vg_systemgroup-lv_home  (253:1)
vg_virts-disk2  (253:3)
vg_systemgroup-lv_root  (253:0)



Try to create it again with the same name:

lvcreate -n badboy -L 1g --virtualsize 1g vg_virts
  Unable to create LV badboy_vorigin in Volume Group vg_virts: name already in use.
  Couldn't create virtual origin for LV badboy

Actual results:

The creation fails because _vorigin remains:

lvs -a | grep bad
  [badboy_vorigin]          vg_virts       vwi-a-v-    1.00g    



Expected results:

lvremove should properly remove all the remnants of the LV.
Comment 3 Alasdair Kergon 2013-10-10 06:53:42 EDT
So this is trying to remove the LV while it is still in use?
Does the current release have the same bug?
Comment 4 Nenad Peric 2013-10-11 05:42:49 EDT
I can still reproduce this:

[root@virt-011 ~]# lvcreate -n badboy -L 1g --virtualsize 1g vg
  Logical volume "badboy" created

[root@virt-011 ~]# for i in {1..1000}; do dd if=/dev/urandom of=/dev/vg/badboy bs=10240 count=10000 >& out.log; done &
[1] 5920

[root@virt-011 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin           Data%  Move Log Cpy%Sync Convert
  badboy           vg         swi-aos---   1.00g      [badboy_vorigin]   9.57                          
  [badboy_vorigin] vg         owi-a-s---   1.00g                                                       
  lv_root          vg_virt011 -wi-ao----   6.71g                                                       
  lv_swap          vg_virt011 -wi-ao---- 816.00m                                                       
[root@virt-011 ~]# lvremove -f vg/badboy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  Unable to deactivate vg-badboy (253:2)
  Unable to deactivate logical volume "badboy"
[root@virt-011 ~]# kill %1
[root@virt-011 ~]# 
[1]+  Terminated              for i in {1..1000};
do
    dd if=/dev/urandom of=/dev/vg/badboy bs=10240 count=10000 &>out.log;
done
[root@virt-011 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  badboy           vg         -wi-a-----   1.00g                                             
  [badboy_vorigin] vg         vwi-a-v---   1.00g                                             
  lv_root          vg_virt011 -wi-ao----   6.71g                                             
  lv_swap          vg_virt011 -wi-ao---- 816.00m                              

Now trying the removal:

[root@virt-011 ~]# lvremove -f vg/badboy
  Logical volume "badboy" successfully removed
[root@virt-011 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  [badboy_vorigin] vg         vwi-a-v---   1.00g                                             
  lv_root          vg_virt011 -wi-ao----   6.71g                                             
  lv_swap          vg_virt011 -wi-ao---- 816.00m                                             
[root@virt-011 ~]# dmsetup ls
vg-badboy_vorigin	(253:3)
vg_virt011-lv_swap	(253:1)
vg_virt011-lv_root	(253:0)
[root@virt-011 ~]#
Comment 5 Nenad Peric 2013-10-11 05:44:31 EDT
Ehm.. Forgot to write which versions I used:

lvm2-2.02.100-5.el6.x86_64
lvm2-libs-2.02.100-5.el6.x86_64
device-mapper-1.02.79-5.el6.x86_64
device-mapper-libs-1.02.79-5.el6.x86_64
device-mapper-event-1.02.79-5.el6.x86_64
kernel-2.6.32-422.el6.x86_64
Comment 6 Alasdair Kergon 2013-10-11 20:25:43 EDT
Peter is going to try a simple patch - to reinsert a "is the device open?" check (before the command changes anything) that got taken out a while back, with a new wait loop around it in case it's only been open temporarily by udev.
Comment 8 Zdenek Kabelac 2013-10-14 11:19:07 EDT
This has been recently fixed upstream:

https://www.redhat.com/archives/lvm-devel/2013-October/msg00043.html

It's a kind of workaround - there is a still hidden race - however it's currently hard to fix this with current libdm code - it needs more thinking.

But the reported problem should be handled.
Comment 9 Alasdair Kergon 2013-10-14 13:43:45 EDT
Also requires a little retry loop around the open to avoid race with any asynchronous udev processing.

prajnoha has patch for that
Comment 11 Peter Rajnoha 2013-10-15 07:29:08 EDT
(In reply to Alasdair Kergon from comment #9)
> Also requires a little retry loop around the open to avoid race with any
> asynchronous udev processing.

https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=48df36b8c531a43da8602b50759c9e841e1d9f44
Comment 13 Nenad Peric 2013-10-16 10:03:52 EDT
Tested, and marking as VERIFIED with lvm2-2.02.100-6.el6.

[root@virt-013 ~]# lvcreate -n badboy -L 1g --virtualsize 1g vg
  Logical volume "badboy" created

added load

[root@virt-013 ~]# lvremove -f vg/badboy
  Logical volume vg/badboy in use.
[root@virt-013 ~]# kill %1
[root@virt-013 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin           Data%  Move Log Cpy%Sync Convert
  badboy           vg         swi-a-s---   1.00g      [badboy_vorigin]   9.57                          
  [badboy_vorigin] vg         owi-a-s---   1.00g                                                       
  lv_root          vg_virt013 -wi-ao----   6.71g                                                       
  lv_swap          vg_virt013 -wi-ao---- 816.00m                                                       
[root@virt-013 ~]# lvremove -f vg/badboy
  Logical volume "badboy" successfully removed
[root@virt-013 ~]# lvs -a
  LV      VG         Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root vg_virt013 -wi-ao----   6.71g                                             
  lv_swap vg_virt013 -wi-ao---- 816.00m               
[root@virt-013 ~]# dmsetup ls
vg_virt013-lv_swap	(253:1)
vg_virt013-lv_root	(253:0)
Comment 14 errata-xmlrpc 2013-11-21 18:17:25 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1704.html

Note You need to log in before you can comment on or make changes to this bug.