Bug 638711

Summary: [regression] 'lvremove' fails sometimes to remove snapshot volumes
Product: Red Hat Enterprise Linux 6 Reporter: Brian Likosar <liko>
Component: lvm2Assignee: Peter Rajnoha <prajnoha>
Status: CLOSED DUPLICATE QA Contact: Corey Marthaler <cmarthal>
Severity: medium Docs Contact:
Priority: high    
Version: 6.1CC: agk, bmarzins, bmr, centaur, coughlan, cyberrider, davidz, dwysocha, frank, ftaylor, heinzm, jbrassow, joe.thornber, jonathan, kay.sievers, kparal, kueda, liko, llowrey, lvm-team, mbroz, msnitzer, ndevos, nls1729, prajnoha, prockai, rh-bugzilla, walter.haidinger
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 577798 Environment:
Last Closed: 2011-05-30 09:33:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 577798    
Bug Blocks:    

Description Brian Likosar 2010-09-29 18:07:41 UTC
+++ This bug was initially created as a clone of Bug #577798 +++

Description of problem:

# lvdisplay

  --- Logical volume ---
  LV Name                /dev/vg01/.local.backup
  VG Name                vg01
  LV UUID                hkAyO4-M31g-LJw5-Kdcu-AfK1-Bquw-buVrWA
  LV Write Access        read only
  LV snapshot status     active destination for /dev/vg01/local
  LV Status              available
  # open                 0
  LV Size                2,00 GiB
  Current LE             512
  COW-table size         1,00 GiB
  COW-table LE           256
  Allocated to snapshot  0,01% 
  Snapshot chunk size    4,00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:29

# /dev/vg01/.local.backup
  Can't remove open logical volume ".local.backup"

  ... repeating this several times ...

# lvremove  /dev/vg01/.local.backup
Do you really want to remove active logical volume .local.backup? [y/n]: y
  Logical volume ".local.backup" successfully removed


Version-Release number of selected component (if applicable):

kernel-2.6.33.1-19.fc13.x86_64
lvm2-2.02.61-1.fc13.x86_64

--- Additional comment from enrico.scholz.de on 2010-03-29 05:52:00 EDT ---

Created attachment 403250 [details]
strace in non-working case

--- Additional comment from enrico.scholz.de on 2010-03-29 05:52:28 EDT ---

Created attachment 403251 [details]
strace in working case

--- Additional comment from prajnoha on 2010-03-29 06:08:36 EDT ---

Do you have "udisks" package installed? There is one udev rule that could have possibly caused this...

For starters, just a quick check - could you please try to kill udev daemon temporarily and see if you can reproduce the problem? Thanks.

--- Additional comment from enrico.scholz.de on 2010-03-30 07:36:02 EDT ---

yes; udisks is installed and I can not reproduce the issue after its removal.

'udevadm control --stop-exec-queue' before lvremove seems to work too.

--- Additional comment from prajnoha on 2010-03-30 08:08:22 EDT ---

Just for the record, the rule we have problem supporting is this one exactly (in /lib/udev/rules.d/80-udisks.rules which is a part of udisks package):

# Make udevd synthesize a 'change' uevent when last opener of a rw-fd closes the fd - this
# should be part of the device-mapper rules
KERNEL=="dm-*", OPTIONS+="watch"

We have added udev synchronisation feature in device-mapper/lvm2 recently so we always wait until udev processing is settled down to cope with such problems where devices are accessed from within udev rules and also to provide a way to wait for nodes/symlinks to be created.

However, we can't synchronize with events synthesized as a result of this rule (like we can't with events originating in "udevadm trigger" which generates such events as well). The synchronisation could be done on events we know about (events originated in device-mapper itself).

There are still ongoing discussions with udev team to properly deal with this issue though...

Now, could you please keep the udisks package and also keep udev running while having the "watch" rule commented out and see if the problem is gone? Thanks.

(..so we're really sure this is exactly the case and as a proof)

--- Additional comment from enrico.scholz.de on 2010-03-30 08:58:11 EDT ---

yes; after commenting out this line I can not reproduce the error anymore.

--- Additional comment from prajnoha on 2010-03-30 09:20:24 EDT ---

So we have another nice and practical example how the "watch" rule disrupts an idea to properly synchronize with udev events...

--- Additional comment from prajnoha on 2010-03-30 09:26:06 EDT ---

...also, CC-ing David (the "udisks" package maintainer).

--- Additional comment from enrico.scholz.de on 2010-05-13 06:19:18 EDT ---

still with

lvm2-2.02.61-1.fc13.x86_64
udisks-1.0.1-1.fc13.x86_64

--- Additional comment from prajnoha on 2010-05-17 09:07:01 EDT ---

(In reply to comment #9)
> still with
> 
> lvm2-2.02.61-1.fc13.x86_64
> udisks-1.0.1-1.fc13.x86_64    

Unfortunately, we still don't have a proper solution for synchronization with the events like the ones originated in the "watch" rule.

Comment 3 Scott Marshall 2010-11-27 09:49:00 UTC
Bug 577798 (the original report) has further information on this issue, starting here: https://bugzilla.redhat.com/show_bug.cgi?id=577798#c11

Comment 7 Peter Rajnoha 2011-05-30 08:32:29 UTC
We've applied a patch upstream that tries to minimize device RW open calls
within the LVM itself. This should also prevent the events based on the watch
rule from being fired when not necessary, at least with respect to internal LVM
handling of devices:

  https://www.redhat.com/archives/lvm-devel/2011-May/msg00025.html (LVM2
v2.02.86)

However, there's still a possibility that somone else, externally, will open a
device for read-write and close it (which will cause the uevent to occur) just
before the device is removed and so we could end up with the same problem as
reported here - in this case, we have no control over this asynchronicity.

(For a hassle about the watch rule and more related information see also
https://bugzilla.redhat.com/show_bug.cgi?id=561424)