664509 – 2.2.6 connectStoragePool doesn't return with valid return response but with exception

Bug 664509 - 2.2.6 connectStoragePool doesn't return with valid return response but with exception

Summary: 2.2.6 connectStoragePool doesn't return with valid return response but with e...

Keywords:
Status:	CLOSED NEXTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 5
Classification:	Red Hat
Component:	vdsm22
Sub Component:
Version:	5.6
Hardware:	x86_64
OS:	Linux
Priority:	urgent
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Eduardo Warszawski
QA Contact:	yeylon@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	568128 675133
TreeView+	depends on / blocked

Reported:	2010-12-20 15:59 UTC by Haim
Modified:	2016-04-18 06:36 UTC (History)
CC List:	8 users (show)
Fixed In Version:	vdsm22-4.5-63.13
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2011-02-07 17:11:34 UTC
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
vdsm.log (1.42 MB, application/x-gzip) 2010-12-20 15:59 UTC, Haim	no flags	Details
vdsm.log (790.19 KB, application/x-gzip) 2011-01-25 14:08 UTC, Daniel Paikov	no flags	Details
View All

Description Haim 2010-12-20 15:59:13 UTC

Created attachment 469799 [details]
vdsm.log

Description of problem:

on rhel5.6, with vdsm 2.2.6, which includes NFS time-out fix, it seems as connectStoragePool and reconstructMaster both fails on 'OSError: [Errno 62] Operation stuck on remote handler' and doesn't return valid return response to rhevm.

guess that problem will occur when ever Run command will fail on the above, we should catch that exception, and return valid response. 

repro steps: 

1) 1 host (spm)
2) 2 storage domains, 1 ISO (each storage domain on different storage server)
3) block communication to master storage domain on storage X 
4) spm-protect will kill vdsm, and start it again
5) spmStop will be sent 
6) connect storage server (will fail)
7) connect storage pool will fail, but no return response (see below)

also see full log. 

[root@nott-vds4 ~]# cat /tmp/s 
Thread-731::INFO::2010-12-20 16:58:29,811::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                            
Thread-735::INFO::2010-12-20 16:58:39,854::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-747::INFO::2010-12-20 16:59:24,237::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-755::INFO::2010-12-20 17:00:08,643::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-763::INFO::2010-12-20 17:00:53,207::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-771::INFO::2010-12-20 17:01:37,611::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-779::INFO::2010-12-20 17:02:21,974::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-787::INFO::2010-12-20 17:03:06,335::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-795::INFO::2010-12-20 17:03:50,716::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-799::INFO::2010-12-20 17:04:25,064::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                            
Thread-799::INFO::2010-12-20 17:04:25,068::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                            
Thread-803::INFO::2010-12-20 17:04:35,110::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-815::INFO::2010-12-20 17:05:19,478::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-823::INFO::2010-12-20 17:06:03,863::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-831::INFO::2010-12-20 17:06:48,243::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-839::INFO::2010-12-20 17:07:32,625::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-847::INFO::2010-12-20 17:08:17,014::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-855::INFO::2010-12-20 17:09:01,414::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-863::INFO::2010-12-20 17:09:45,881::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-867::INFO::2010-12-20 17:10:20,438::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                            
Thread-867::INFO::2010-12-20 17:10:20,439::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                            
Thread-871::INFO::2010-12-20 17:10:30,452::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-883::INFO::2010-12-20 17:11:14,831::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-891::INFO::2010-12-20 17:11:59,207::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-899::INFO::2010-12-20 17:12:43,634::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-907::INFO::2010-12-20 17:13:28,022::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-915::INFO::2010-12-20 17:14:12,443::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-923::INFO::2010-12-20 17:14:56,844::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-931::INFO::2010-12-20 17:15:41,301::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-935::INFO::2010-12-20 17:16:15,649::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                            
Thread-935::INFO::2010-12-20 17:16:15,651::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                            
Thread-939::INFO::2010-12-20 17:16:25,705::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-951::INFO::2010-12-20 17:17:10,074::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-959::INFO::2010-12-20 17:17:54,509::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-967::INFO::2010-12-20 17:18:38,920::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-975::INFO::2010-12-20 17:19:23,294::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-983::INFO::2010-12-20 17:20:07,724::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-991::INFO::2010-12-20 17:20:52,094::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-999::INFO::2010-12-20 17:21:36,619::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15)                                                 
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1003::INFO::2010-12-20 17:22:10,985::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                           
Thread-1003::INFO::2010-12-20 17:22:10,987::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                           
Thread-1007::INFO::2010-12-20 17:22:21,021::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1019::INFO::2010-12-20 17:23:05,409::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1027::INFO::2010-12-20 17:23:49,802::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1035::INFO::2010-12-20 17:24:34,267::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1043::INFO::2010-12-20 17:25:18,654::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1051::INFO::2010-12-20 17:26:03,082::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1059::INFO::2010-12-20 17:26:47,466::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1067::INFO::2010-12-20 17:27:31,848::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1071::INFO::2010-12-20 17:28:06,264::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                           
Thread-1071::INFO::2010-12-20 17:28:06,265::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                           
Thread-1075::INFO::2010-12-20 17:28:16,290::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1087::INFO::2010-12-20 17:29:00,667::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1095::INFO::2010-12-20 17:29:45,023::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1103::INFO::2010-12-20 17:30:29,429::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1111::INFO::2010-12-20 17:31:13,788::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1119::INFO::2010-12-20 17:31:58,315::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1127::INFO::2010-12-20 17:32:42,710::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1135::INFO::2010-12-20 17:33:27,119::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1139::INFO::2010-12-20 17:34:01,483::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                           
Thread-1139::INFO::2010-12-20 17:34:01,485::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                           
Thread-1143::INFO::2010-12-20 17:34:11,519::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1155::INFO::2010-12-20 17:34:55,957::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1163::INFO::2010-12-20 17:35:40,379::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1171::INFO::2010-12-20 17:36:24,931::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1179::INFO::2010-12-20 17:37:09,311::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1187::INFO::2010-12-20 17:37:53,642::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1195::INFO::2010-12-20 17:38:38,022::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1203::INFO::2010-12-20 17:39:22,439::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1207::INFO::2010-12-20 17:39:56,803::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                           
Thread-1207::INFO::2010-12-20 17:39:56,805::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                           
Thread-1211::INFO::2010-12-20 17:40:06,828::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1223::INFO::2010-12-20 17:40:51,260::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1231::INFO::2010-12-20 17:41:35,688::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1239::INFO::2010-12-20 17:42:20,290::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1247::INFO::2010-12-20 17:43:04,717::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1255::INFO::2010-12-20 17:43:49,108::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1263::INFO::2010-12-20 17:44:33,506::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1271::INFO::2010-12-20 17:45:17,853::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1275::INFO::2010-12-20 17:45:52,257::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)                                                                                                           
Thread-1275::INFO::2010-12-20 17:45:52,258::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}                                                                                                                                                           
Thread-1279::INFO::2010-12-20 17:46:02,286::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=20)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1291::INFO::2010-12-20 17:46:46,683::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1299::INFO::2010-12-20 17:47:31,042::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20)                                                
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool                                                                                
Thread-1307::INFO::2010-12-20 17:48:15,424::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1315::INFO::2010-12-20 17:48:59,825::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1323::INFO::2010-12-20 17:49:44,205::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1331::INFO::2010-12-20 17:50:28,601::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1339::INFO::2010-12-20 17:51:13,011::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1343::INFO::2010-12-20 17:51:47,404::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745)
Thread-1343::INFO::2010-12-20 17:51:47,406::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}}
Thread-1347::INFO::2010-12-20 17:51:57,417::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=21)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1359::INFO::2010-12-20 17:52:41,762::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1367::INFO::2010-12-20 17:53:26,118::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1375::INFO::2010-12-20 17:54:10,504::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool
Thread-1383::INFO::2010-12-20 17:54:54,942::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21)
  File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool

Comment 1 Ayal Baron 2010-12-25 23:01:49 UTC

Attahced log doesn't contain the above excerpt.
Please attach correct log.

Comment 2 Ayal Baron 2010-12-29 17:59:59 UTC

(In reply to comment #1)
> Attahced log doesn't contain the above excerpt.
> Please attach correct log.

Haim, what about the correct logs?

Comment 4 Haim 2011-01-09 06:47:17 UTC

(In reply to comment #2)
> (In reply to comment #1)
> > Attahced log doesn't contain the above excerpt.
> > Please attach correct log.
> 
> Haim, what about the correct logs?

Have you looked at the attached file (vdsm.log) ?

Comment 5 Ayal Baron 2011-01-09 12:42:32 UTC

(In reply to comment #4)
> (In reply to comment #2)
> > (In reply to comment #1)
> > > Attahced log doesn't contain the above excerpt.
> > > Please attach correct log.
> > 
> > Haim, what about the correct logs?
> 
> Have you looked at the attached file (vdsm.log) ?

Of course I have, couldn't find the above excerpt in the log...

Comment 6 Ayal Baron 2011-01-17 08:02:01 UTC

Haim, please retest this with 2.2.6

Comment 7 Ayal Baron 2011-01-25 13:56:43 UTC

Paikov just reproduced.  Issue is that connectStoragePool exits with "Operation stuck on remote handler" and not on an error that means that master domain was not found.

Paikov, please attach logs.

Comment 8 Daniel Paikov 2011-01-25 14:08:01 UTC

Created attachment 475172 [details]
vdsm.log

Attaching log and moving bug back to Assigned.

Comment 9 Dan Kenigsberg 2011-02-03 21:06:39 UTC

Patches committed to DistCVS. Moving to MODIFIED to enabled Z-stream inclusion.

Comment 11 Dan Kenigsberg 2011-02-07 17:11:34 UTC

This bug has lived its purpose by being cloned into rhev 2.2.6. No one needs it now.

Note You need to log in before you can comment on or make changes to this bug.