Bug 889469

Summary: Properly synchronize with udev and do not leave dangling synchronization cookies if pvscan --cache -aay is used (autoactivation)
Product: Red Hat Enterprise Linux 6 Reporter: Peter Rajnoha <prajnoha>
Component: lvm2Assignee: Peter Rajnoha <prajnoha>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: urgent Docs Contact:
Priority: high    
Version: 6.4CC: agk, dwysocha, heinzm, jbrassow, mcsontos, msnitzer, nperic, prajnoha, prockai, thornber, tlavigne, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.98-7.el6 Doc Type: Bug Fix
Doc Text:
No documentation needed. The autoactivation - pvscan --cache -aay feature is new in 6.4.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 08:15:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 895654    

Description Peter Rajnoha 2012-12-21 10:26:57 UTC
When using pvscan --cache -aay either within the udev rule automatically to autoactivate VGs/LVs (69-dm-lvm-metad.rules) or directly on the command line, we need to add an extra sync_local_dev_names call for the pvscan --cache to properly synchronize with udev. Otherwise, we end up unsynchronized and with *dangling udev_sync cookies* (sempahores) in the system as the activation code called within the pvscan sets up cookies and expects the synchronization to be called later.

The bug was caused by the fact that pvscan --cache uses VG_GLOBAL lock and it does not lock each VG separately. But the code responsible for calling the udev synchronization is used only for real VGs and VG_GLOBAL is not a real VG:

define unlock_vg(cmd, vol)
  do { \
    if (is_real_vg(vol)) \
      sync_dev_names(cmd); \
   (void) lock_vol(cmd, vol, LCK_VG_UNLOCK); \
  } while (0)

Therefore, in this particular case with pvscan --cache, we need to explicitly add the extra sync_dev_names call to synchronize with udev properly.

Comment 1 Peter Rajnoha 2012-12-21 10:27:29 UTC
The patch is a simple one-liner and it is upstream now:

http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=ad85b0c526b40a4e78ac83426f54c935c147c787

Comment 3 Marian Csontos 2012-12-21 20:21:01 UTC
LVM would hang once the limit for semaphores is reached - this could be a root cause of other unresolved issues. It is a "Must have" if lvmetad is going to be used.

Have not seen any dangling cookies since the patch was applied.

Comment 4 Peter Rajnoha 2013-01-03 15:03:25 UTC
To QA:
To test this issue, you can use this sequence:

[root@rhel6-a ~]# pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created

[root@rhel6-a ~]# vgcreate vg /dev/sda
  Volume group "vg" successfully created

[root@rhel6-a ~]# lvcreate -l1 vg
  Logical volume "lvol0" created

[root@rhel6-a ~]# vgchange -an vg
  0 logical volume(s) in volume group "vg" now active

[root@rhel6-a ~]# lvs
  LV      VG        Attr      LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  lvol0   vg        -wi------   4.00m                                             

[root@rhel6-a ~]# dmsetup udevcookies
Cookie       Semid      Value      Last semop time           Last change time

[root@rhel6-a ~]# pvscan --cache -aay /dev/sda
  1 logical volume(s) in volume group "vg" now active

[root@rhel6-a ~]# lvs
  LV      VG        Attr      LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  lvol0   vg        -wi-a----   4.00m                                             

[root@rhel6-a ~]# dmsetup udevcookies
Cookie       Semid      Value      Last semop time           Last change time
0xd4df189    229378     1          Thu Jan  3 15:00:37 2013  Thu Jan  3 15:00:37 2013


The last one is incorrect! There should not be any cookie left after the pvscan --cache -aay command called before.

Comment 6 Nenad Peric 2013-01-09 14:22:30 UTC
I can confirma "stuck cookie" wven with simplest commands in the -6 version.
The cookie leftover was:

(08:06:54) [root@r6-node01:~]$ dmsetup udevcookies
Cookie       Semid      Value      Last semop time           Last change time
0xd4d5b78    393219     1          Wed Jan  9 07:53:18 2013  Wed Jan  9 07:53:18 2013

just after  the follwoing commands:

pvcreate  /dev/sda1
vgcreate vg /dev/sda1 
lvcreate -l1 vg
vgchange -an vg
vgextend vg /dev/sdb1 
vgchange -an vg

(08:20:35) [root@r6-node01:~]$ lvextend  vg/lvol0 /dev/sdb1
  Extending logical volume lvol0 to 10.00 GiB
  Logical volume lvol0 successfully resized
(08:20:43) [root@r6-node01:~]$ vgs
  VG       #PV #LV #SN Attr   VSize  VFree
  VolGroup   1   2   0 wz--n-  9.51g    0 
  vg         2   1   0 wz--n- 19.98g 9.99g
(08:20:49) [root@r6-node01:~]$ lvs
  LV      VG       Attr      LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root VolGroup -wi-ao---  7.54g                                             
  lv_swap VolGroup -wi-ao---  1.97g                                             
  lvol0   vg       -wi------ 10.00g                                             

(08:21:04) [root@r6-node01:~]$ dmsetup udevcookies
Cookie       Semid      Value      Last semop time           Last change time
(08:21:12) [root@r6-node01:~]$ pvscan --cache -aay /dev/sdb1 
  1 logical volume(s) in volume group "vg" now active
(08:21:35) [root@r6-node01:~]$ dmsetup udevcookies
Cookie       Semid      Value      Last semop time           Last change time
(08:21:38) [root@r6-node01:~]$ lvs
  LV      VG       Attr      LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root VolGroup -wi-ao---  7.54g                                             
  lv_swap VolGroup -wi-ao---  1.97g                                             
  lvol0   vg       -wi-a---- 10.00g                                             
(08:21:41) [root@r6-node01:~]$ 

Verified with:

lvm2-2.02.98-7.el6.x86_64
lvm2-devel-2.02.98-7.el6.x86_64
lvm2-libs-2.02.98-7.el6.x86_64
udev-147-2.43.el6.x86_64
device-mapper-1.02.77-7.el6.x86_64

Comment 7 errata-xmlrpc 2013-02-21 08:15:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0501.html