Bug 834327 - Virtual snapshot does not get removed after removal attempt on a busy device
Summary: Virtual snapshot does not get removed after removal attempt on a busy device
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Peter Rajnoha
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-06-21 14:46 UTC by Nenad Peric
Modified: 2013-11-21 23:17 UTC (History)
12 users (show)

Fixed In Version: lvm2-2.02.100-6.el6
Doc Type: Bug Fix
Doc Text:
Virtual snapshot device removal failed if the device was still open. The lvremove left <virtual_snashot_name>_vorigin device-mapper device in the system. This device could not be removed with LVM tools (as it's internal LVM device) nor any other virtual snapshot with the <virtual_snapshot_name> name created again unless the <virtual_snapshot_name>_vorigin device was manually removed using dmsetup remove. This has been fixed by properly checking the LV open count status before proceeding with the removal operation.
Clone Of:
Environment:
Last Closed: 2013-11-21 23:17:25 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1704 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2013-11-20 21:52:01 UTC

Description Nenad Peric 2012-06-21 14:46:59 UTC
Description of problem:

The lvremove does not actually remove a virtual snapshot after attempted to remove such a LV with a busy device.

lvremove finishes successfuly but the _vorigin remains. 
This causes later LV creation of a same named virt snapshot to fail. 



Version-Release number of selected component (if applicable):

lvm2-2.02.95-10.el6.x86_64
lvm2-libs-2.02.95-10.el6.x86_64
lvm2-cluster-2.02.95-10.el6.x86_64
cmirror-2.02.95-10.el6.x86_64
device-mapper-1.02.74-10.el6.x86_64
device-mapper-event-1.02.74-10.el6.x86_64
device-mapper-libs-1.02.74-10.el6.x86_64
device-mapper-event-libs-1.02.74-10.el6.x86_64

How reproducible:

Every time

Steps to Reproduce:
1. Create LV 

lvcreate -n badboy -L 1g --virtualsize 1g vg_virts

2. Create some load on the newly created LV:

xxd /dev/vg_virts/badboy

3.
  
Try to remove the LV:

lvremove -f vg_virts/badboy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  Unable to deactivate vg_virts-badboy (253:5)
  Unable to deactivate logical volume "badboy"

Stop the load and remove the LV:

lvremove -f vg_virts/badboy
  Logical volume "badboy" successfully removed

dmsetup ls
vg_virts-badboy_vorigin (253:6)
vg_virts-c1--node01     (253:4)
vg_virts-disk   (253:2)
vg_systemgroup-lv_home  (253:1)
vg_virts-disk2  (253:3)
vg_systemgroup-lv_root  (253:0)



Try to create it again with the same name:

lvcreate -n badboy -L 1g --virtualsize 1g vg_virts
  Unable to create LV badboy_vorigin in Volume Group vg_virts: name already in use.
  Couldn't create virtual origin for LV badboy

Actual results:

The creation fails because _vorigin remains:

lvs -a | grep bad
  [badboy_vorigin]          vg_virts       vwi-a-v-    1.00g    



Expected results:

lvremove should properly remove all the remnants of the LV.

Comment 3 Alasdair Kergon 2013-10-10 10:53:42 UTC
So this is trying to remove the LV while it is still in use?
Does the current release have the same bug?

Comment 4 Nenad Peric 2013-10-11 09:42:49 UTC
I can still reproduce this:

[root@virt-011 ~]# lvcreate -n badboy -L 1g --virtualsize 1g vg
  Logical volume "badboy" created

[root@virt-011 ~]# for i in {1..1000}; do dd if=/dev/urandom of=/dev/vg/badboy bs=10240 count=10000 >& out.log; done &
[1] 5920

[root@virt-011 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin           Data%  Move Log Cpy%Sync Convert
  badboy           vg         swi-aos---   1.00g      [badboy_vorigin]   9.57                          
  [badboy_vorigin] vg         owi-a-s---   1.00g                                                       
  lv_root          vg_virt011 -wi-ao----   6.71g                                                       
  lv_swap          vg_virt011 -wi-ao---- 816.00m                                                       
[root@virt-011 ~]# lvremove -f vg/badboy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  Unable to deactivate vg-badboy (253:2)
  Unable to deactivate logical volume "badboy"
[root@virt-011 ~]# kill %1
[root@virt-011 ~]# 
[1]+  Terminated              for i in {1..1000};
do
    dd if=/dev/urandom of=/dev/vg/badboy bs=10240 count=10000 &>out.log;
done
[root@virt-011 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  badboy           vg         -wi-a-----   1.00g                                             
  [badboy_vorigin] vg         vwi-a-v---   1.00g                                             
  lv_root          vg_virt011 -wi-ao----   6.71g                                             
  lv_swap          vg_virt011 -wi-ao---- 816.00m                              

Now trying the removal:

[root@virt-011 ~]# lvremove -f vg/badboy
  Logical volume "badboy" successfully removed
[root@virt-011 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  [badboy_vorigin] vg         vwi-a-v---   1.00g                                             
  lv_root          vg_virt011 -wi-ao----   6.71g                                             
  lv_swap          vg_virt011 -wi-ao---- 816.00m                                             
[root@virt-011 ~]# dmsetup ls
vg-badboy_vorigin	(253:3)
vg_virt011-lv_swap	(253:1)
vg_virt011-lv_root	(253:0)
[root@virt-011 ~]#

Comment 5 Nenad Peric 2013-10-11 09:44:31 UTC
Ehm.. Forgot to write which versions I used:

lvm2-2.02.100-5.el6.x86_64
lvm2-libs-2.02.100-5.el6.x86_64
device-mapper-1.02.79-5.el6.x86_64
device-mapper-libs-1.02.79-5.el6.x86_64
device-mapper-event-1.02.79-5.el6.x86_64
kernel-2.6.32-422.el6.x86_64

Comment 6 Alasdair Kergon 2013-10-12 00:25:43 UTC
Peter is going to try a simple patch - to reinsert a "is the device open?" check (before the command changes anything) that got taken out a while back, with a new wait loop around it in case it's only been open temporarily by udev.

Comment 8 Zdenek Kabelac 2013-10-14 15:19:07 UTC
This has been recently fixed upstream:

https://www.redhat.com/archives/lvm-devel/2013-October/msg00043.html

It's a kind of workaround - there is a still hidden race - however it's currently hard to fix this with current libdm code - it needs more thinking.

But the reported problem should be handled.

Comment 9 Alasdair Kergon 2013-10-14 17:43:45 UTC
Also requires a little retry loop around the open to avoid race with any asynchronous udev processing.

prajnoha has patch for that

Comment 11 Peter Rajnoha 2013-10-15 11:29:08 UTC
(In reply to Alasdair Kergon from comment #9)
> Also requires a little retry loop around the open to avoid race with any
> asynchronous udev processing.

https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=48df36b8c531a43da8602b50759c9e841e1d9f44

Comment 13 Nenad Peric 2013-10-16 14:03:52 UTC
Tested, and marking as VERIFIED with lvm2-2.02.100-6.el6.

[root@virt-013 ~]# lvcreate -n badboy -L 1g --virtualsize 1g vg
  Logical volume "badboy" created

added load

[root@virt-013 ~]# lvremove -f vg/badboy
  Logical volume vg/badboy in use.
[root@virt-013 ~]# kill %1
[root@virt-013 ~]# lvs -a
  LV               VG         Attr       LSize   Pool Origin           Data%  Move Log Cpy%Sync Convert
  badboy           vg         swi-a-s---   1.00g      [badboy_vorigin]   9.57                          
  [badboy_vorigin] vg         owi-a-s---   1.00g                                                       
  lv_root          vg_virt013 -wi-ao----   6.71g                                                       
  lv_swap          vg_virt013 -wi-ao---- 816.00m                                                       
[root@virt-013 ~]# lvremove -f vg/badboy
  Logical volume "badboy" successfully removed
[root@virt-013 ~]# lvs -a
  LV      VG         Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root vg_virt013 -wi-ao----   6.71g                                             
  lv_swap vg_virt013 -wi-ao---- 816.00m               
[root@virt-013 ~]# dmsetup ls
vg_virt013-lv_swap	(253:1)
vg_virt013-lv_root	(253:0)

Comment 14 errata-xmlrpc 2013-11-21 23:17:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1704.html


Note You need to log in before you can comment on or make changes to this bug.