Created attachment 484420 [details]
vdsm log

Description of problem:
large scale deployment 35 SDs 3 luns each, when attach storage domain failed on:
Thread-223::ERROR::2011-03-10 08:59:45,929::task::854::TaskManager.Task::(_setError) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 862, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/spm.py", line 114, in run
    return self.func(*args, **kwargs)
  File "/usr/share/vdsm/storage/spm.py", line 1133, in public_attachStorageDomain
    pool.attachSD(sdUUID)
  File "/usr/share/vdsm/storage/sp.py", line 799, in attachSD
    self.acquireDomainLock(sdUUID)
  File "/usr/share/vdsm/storage/sp.py", line 228, in acquireDomainLock
    raise e
AcquireLockFailure: Could not obtain lock: "id=d392b479-5781-495b-ba2a-0672fef7a8ef, rc=1, out=['error - lease file does not exist or is not writeable', 'usage: /usr/libexec/vdsm/spmprotect.sh COMMAND PARAMETERS', 'Commands:', '  start { spUUID hostId renewal_interval_sec lease_path[:offset] lease_time_ms io_op_timeout_ms fail_retries }', 'Parameters:', '  spUUID -                pool uuid', '  hostId -                host id in pool', '  renewal_interval_sec -  intervals for lease renewals attempts', '  lease_path -            path to lease file/volume', '  offset -                offset of lease within file', '  lease_time_ms -         time limit within which lease must be renewed (at least 2*renewal_interval_sec)', '  io_op_timeout_ms -      I/O operation timeout', '  fail_retries -          Maximal number of attempts to retry to renew the lease before fencing (<= lease_time_ms/renewal_interval_sec)'], err=[]"
Thread-223::DEBUG::2011-03-10 08:59:45,931::task::491::TaskManager.Task::(_debug) Task 0ac7e2c1-de21-4f4e-8836-480e84b2a8fe: Task._run: 0ac7e2c1-de21-4f4e-8836-480e84b2a8fe ('d392b479-5781-495b-ba2a-0672fef7a8ef', '232e5637-d880-4d7c-bb0a-db62197bfc04') {} failed - stopping task

interleaving with a failed refresh storage pool. 

Version-Release number of selected component (if applicable):
vdsm-4.9-53.el6.x86_64

How reproducible:
happened several times

Steps to Reproduce:
1.attach storage domain on a large deployment setup
2.
3.
  
Actual results:


Expected results:


Additional info: