Bug 685100

Summary: [vdsm][storage]attachStorageDomain fails on Could not obtain lock
Product: Red Hat Enterprise Linux 6 Reporter: Moran Goldboim <mgoldboi>
Component: vdsmAssignee: Dan Kenigsberg <danken>
Status: CLOSED DUPLICATE QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1CC: abaron, bazulay, ewarszaw, iheim, smizrahi, srevivo, syeghiay, ykaul
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-04-26 11:36:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
vdsm log none

Description Moran Goldboim 2011-03-15 09:56:28 UTC
Created attachment 484420 [details]
vdsm log

Description of problem:
large scale deployment 35 SDs 3 luns each, when attach storage domain failed on:
Thread-223::ERROR::2011-03-10 08:59:45,929::task::854::TaskManager.Task::(_setError) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 862, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/spm.py", line 114, in run
    return self.func(*args, **kwargs)
  File "/usr/share/vdsm/storage/spm.py", line 1133, in public_attachStorageDomain
    pool.attachSD(sdUUID)
  File "/usr/share/vdsm/storage/sp.py", line 799, in attachSD
    self.acquireDomainLock(sdUUID)
  File "/usr/share/vdsm/storage/sp.py", line 228, in acquireDomainLock
    raise e
AcquireLockFailure: Could not obtain lock: "id=d392b479-5781-495b-ba2a-0672fef7a8ef, rc=1, out=['error - lease file does not exist or is not writeable', 'usage: /usr/libexec/vdsm/spmprotect.sh COMMAND PARAMETERS', 'Commands:', '  start { spUUID hostId renewal_interval_sec lease_path[:offset] lease_time_ms io_op_timeout_ms fail_retries }', 'Parameters:', '  spUUID -                pool uuid', '  hostId -                host id in pool', '  renewal_interval_sec -  intervals for lease renewals attempts', '  lease_path -            path to lease file/volume', '  offset -                offset of lease within file', '  lease_time_ms -         time limit within which lease must be renewed (at least 2*renewal_interval_sec)', '  io_op_timeout_ms -      I/O operation timeout', '  fail_retries -          Maximal number of attempts to retry to renew the lease before fencing (<= lease_time_ms/renewal_interval_sec)'], err=[]"
Thread-223::DEBUG::2011-03-10 08:59:45,931::task::491::TaskManager.Task::(_debug) Task 0ac7e2c1-de21-4f4e-8836-480e84b2a8fe: Task._run: 0ac7e2c1-de21-4f4e-8836-480e84b2a8fe ('d392b479-5781-495b-ba2a-0672fef7a8ef', '232e5637-d880-4d7c-bb0a-db62197bfc04') {} failed - stopping task

interleaving with a failed refresh storage pool. 

Version-Release number of selected component (if applicable):
vdsm-4.9-53.el6.x86_64

How reproducible:
happened several times

Steps to Reproduce:
1.attach storage domain on a large deployment setup
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 2 RHEL Program Management 2011-04-04 02:14:12 UTC
Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 3 Saggi Mizrahi 2011-04-26 11:36:42 UTC
This happens because of a performance problem in attach. Marking as duplicate. Fix already in gerrit.

*** This bug has been marked as a duplicate of bug 679123 ***