Description of problem:

deadlock in sampling method during connectStoragePool task trying to refresh storage pool. 

I see new command sent from backend started on Thread-145:

Thread-145::INFO::2012-02-26 14:12:58,562::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='711a080f-4702-450c-9f5f-bf54f1e99383', hostID=2, scsiKey='711a080f-4702-450c-9f5f-bf54f1e99383', msdUUID='45290416-1dd4-494d-8ac9-29471752bd21', masterVersion=2, options=None)
Thread-145::DEBUG::2012-02-26 14:12:58,563::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.711a080f-4702-450c-9f5f-bf54f1e99383`ReqID=`2345be07-7e6e-4ab5-b9cd-13d935d2d2d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '756' at '_connectStoragePool

then, the last log on this thread is: 

Thread-145::DEBUG::2012-02-26 14:12:58,568::misc::1024::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)

command doesn't return - storage resource remain locked. 

running gdb on vdsm main process, I see the following:


(gdb) bt
#0  0x000000345a4e8933 in select () from /lib64/libc.so.6
#1  0x00007f5682034fcd in select_select (self=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-2.7.2/Modules/selectmodule.c:278
#2  0x000000345b0dffbb in call_function (oparg=<optimized out>, pp_stack=0x7fff3e6afbc8) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4090
#3  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:2740
#4  0x000000345b0e0580 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3e6afd08, func=<function at remote 0x99eb18>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4176
#5  call_function (oparg=<optimized out>, pp_stack=0x7fff3e6afd08) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4111
#6  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:2740
#7  0x000000345b0e0580 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3e6afe48, func=<function at remote 0x121f758>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4176
#8  call_function (oparg=<optimized out>, pp_stack=0x7fff3e6afe48) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4111
#9  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:2740
#10 0x000000345b0e0580 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3e6aff88, func=<function at remote 0x1223488>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4176
#11 call_function (oparg=<optimized out>, pp_stack=0x7fff3e6aff88) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4111
#12 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:2740
#13 0x000000345b0e15a5 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0xa8e4c0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at /usr/src/debug/Python-2.7.2/Python/ceval.c:3330
#14 0x000000345b0dfadb in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3e6b0178, func=<function at remote 0x9a8c08>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4186
#15 call_function (oparg=<optimized out>, pp_stack=0x7fff3e6b0178) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4111
#16 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:2740
#17 0x000000345b0e0580 in fast_function (nk=<optimized out>, na=0, n=<optimized out>, pp_stack=0x7fff3e6b02b8, func=<function at remote 0x9a8de8>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4176
#18 call_function (oparg=<optimized out>, pp_stack=0x7fff3e6b02b8) at /usr/src/debug/Python-2.7.2/Python/ceval.c:4111
#19 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:2740
#20 0x000000345b0e15a5 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at /usr/src/debug/Python-2.7.2/Python/ceval.c:3330
#21 0x000000345b0e16d2 in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/ceval.c:689
#22 0x000000345b0fb9ec in run_mod (mod=<optimized out>, filename=<optimized out>, globals=
    {'betterThreading': <module at remote 0x9abc90>, 'dsaversion': <module at remote 0x9ab980>, 'logging': <module at remote 0x907788>, 'signal': <module at remote 0x7f5683b869b8>, 'lconfig': <module at remote 0x978ad0>, '__builtins__': <module at remote 0x7f5683b52ad0>, '__file__': '/usr/share/vdsm/vdsm', 'traceback': <module at remote 0x7f5683ac2088>, 'loggerConfFile': '/etc/vdsm/logger.conf', '__package__': None, 'sys': <module at remote 0x7f5683b52b78>, 'threading': <module at remote 0x8ad718>, 'parse_args': <function at remote 0xa286e0>, 'run': <function at remote 0x9a8de8>, 'getopt': <module at remote 0x907280>, 'usage': <function at remote 0x917c08>, '__name__': '__main__', 'os': <module at remote 0x7f5683b18cc8>, '__doc__': None, 'constants': <module at remote 0x9ab868>, 'serve_clients': <function at remote 0x9a8c08>}, locals=
    {'betterThreading': <module at remote 0x9abc90>, 'dsaversion': <module at remote 0x9ab980>, 'logging': <module at remote 0x907788>, 'signal': <module at remote 0x7f5683b869b8>, 'lconfig': <module at remote 0x978ad0>, '__builtins__': <module at remote 0x7f5683b52ad0>, '__file__': '/usr/share/vdsm/vdsm', 'traceback': <module at remote 0x7f5683ac2088>, 'loggerConfFile': '/etc/vdsm/logger.conf', '__package__': None, 'sys': <module at remote 0x7f5683b52b78>, 'threading': <module at remote 0x8ad718>, 'parse_args': <function at remote 0xa286e0>, 'run': <function at remote 0x9a8de8>, 'getopt': <module at remote 0x907280>, 'usage': <function at remote 0x917c08>, '__name__': '__main__', 'os': <module at remote 0x7f5683b18cc8>, '__doc__': None, 'constants': <module at remote 0x9ab868>, 'serve_clients': <function at remote 0x9a8c08>}, flags=<optimized out>, arena=<optimized out>) at /usr/src/debug/Python-2.7.2/Python/pythonrun.c:1354
#23 0x000000345b0fc7f0 in PyRun_FileExFlags (fp=0x8d8570, filename=0x7fff3e6b0e5b "/usr/share/vdsm/vdsm", start=<optimized out>, globals=
    {'betterThreading': <module at remote 0x9abc90>, 'dsaversion': <module at remote 0x9ab980>, 'logging': <module at remote 0x907788>, 'signal': <module at remote 0x7f5683b869b8>, 'lconfig': <module at remote 0x978ad0>, '__builtins__': <module at remote 0x7f5683b52ad0>, '__file__': '/usr/share/vdsm/vdsm', 'traceback': <module at remote 0x7f5683ac2088>, 'loggerConfFile': '/etc/vdsm/logger.conf', '__package__': None, 'sys': <module at remote 0x7f5683b52b78>, 'threading': <module at remote 0x8ad718>, 'parse_args': <function at remote 0xa286e0>, 'run': <function at remote 0x9a8de8>, 'getopt': <module at remote 0x907280>, 'usage': <function at remote 0x917c08>, '__name__': '__main__', 'os': <module at remote 0x7f5683b18cc8>, '__doc__': None, 'constants': <module at remote 0x9ab868>, 'serve_clients': <function at remote 0x9a8c08>}, locals=
    {'betterThreading': <module at remote 0x9abc90>, 'dsaversion': <module at remote 0x9ab980>, 'logging': <module at remote 0x907788>, 'signal': <module at remote 0x7f5683b869b8>, 'lconfig': <module at remote 0x978ad0>, '__builtins__': <module at remote 0x7f5683b52ad0>, '__file__': '/usr/share/vdsm/vdsm', 'traceback': <module at remote 0x7f5683ac2088>, 'loggerConfFile': '/etc/vdsm/logger.conf', '__package__': None, 'sys': <module at remote 0x7f5683b52b78>, 'threading': <module at remote 0x8ad718>, 'parse_args': <function at remote 0xa286e0>, 'run': <function at remote 0x9a8de8>, 'getopt': <module at remote 0x907280>, 'usage': <function at remote 0x917c08>, '__name__': '__main__', 'os': <module at remote 0x7f5683b18cc8>, '__doc__': None, 'constants': <module at remote 0x9ab868>, 'serve_clients': <function at remote 0x9a8c08>}, closeit=1, flags=0x7fff3e6b05e0) at /usr/src/debug/Python-2.7.2/Python/pythonrun.c:1340
#24 0x000000345b0fd26f in PyRun_SimpleFileExFlags (fp=0x8d8570, filename=0x7fff3e6b0e5b "/usr/share/vdsm/vdsm", closeit=1, flags=0x7fff3e6b05e0) at /usr/src/debug/Python-2.7.2/Python/pythonrun.c:944
#25 0x000000345b10e745 in Py_Main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/Python-2.7.2/Modules/main.c:599
#26 0x000000345a42169d in __libc_start_main () from /lib64/libc.so.6
#27 0x0000000000400651 in _start ()