Created attachment 793232 [details]
log

Description of problem:

because of a bug in delete snapshot I had several stuck tasks for delete snapshots. 
I tried running create volume task which also got stuck and running a simple vgs on the system will get stuck as well. 
if I terminate the vgs we can see that it's a lock issue with lvn: 

[root@opens-vdsb lvm(keystone_admin)]# vgs
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-69f4bd94-f7df-40bf-94ba-f9e1f6661a3f: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-24d4e530-5eef-4f4c-8b3b-c1dd51f1b295: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-0f48b93d-32bb-4078-89be-76ff6468ee97: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-5a14b4c4-9a14-4de6-abd2-236208bc1429: read failed after 0 of 4096 at 4096: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 10737352704: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 10737410048: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 0: Input/output error
  /dev/cinder-volumes/_snapshot-9a68a5b8-781c-4661-be5c-6a791ff79370: read failed after 0 of 4096 at 4096: Input/output error


^C  CTRL-c detected: giving up waiting for lock
  /var/lock/lvm/V_cinder-volumes:aux: flock failed: Interrupted system call
  Can't get lock for cinder-volumes
  VG   #PV #LV #SN Attr   VSize   VFree
  vg0    1   2   0 wz--n- 232.69g    0 

Version-Release number of selected component (if applicable):

openstack-cinder-2013.1.3-2.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. install Grizzly/2013-08-29.1
2. create a volume + snapshots
3. delete the snapshots
4. create a new volume 
5. run vgs

Actual results:

since the delete snapshots tasks are stuck, all other tasks are stuck as well (create or even simple vgs). seems like a lock issue on vg during create/delete of tasks 

Expected results:

just because one task is stuck does not mean that all others should get stuck as well. 
we need to change the way we take a lock on vg during delete/create and create a time out in case an exclusive lock is stuck. 

Additional info: cinder volumes log