Bug 664509
| Summary: | 2.2.6 connectStoragePool doesn't return with valid return response but with exception | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Haim <hateya> | ||||||
| Component: | vdsm22 | Assignee: | Eduardo Warszawski <ewarszaw> | ||||||
| Status: | CLOSED NEXTRELEASE | QA Contact: | yeylon <yeylon> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 5.6 | CC: | abaron, cpelland, danken, iheim, mgoldboi, smizrahi, srevivo, yeylon | ||||||
| Target Milestone: | rc | Keywords: | ZStream | ||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | vdsm22-4.5-63.13 | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2011-02-07 17:11:34 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 568128, 675133 | ||||||||
| Attachments: |
|
||||||||
Attahced log doesn't contain the above excerpt. Please attach correct log. (In reply to comment #1) > Attahced log doesn't contain the above excerpt. > Please attach correct log. Haim, what about the correct logs? (In reply to comment #2) > (In reply to comment #1) > > Attahced log doesn't contain the above excerpt. > > Please attach correct log. > > Haim, what about the correct logs? Have you looked at the attached file (vdsm.log) ? (In reply to comment #4) > (In reply to comment #2) > > (In reply to comment #1) > > > Attahced log doesn't contain the above excerpt. > > > Please attach correct log. > > > > Haim, what about the correct logs? > > Have you looked at the attached file (vdsm.log) ? Of course I have, couldn't find the above excerpt in the log... Haim, please retest this with 2.2.6 Paikov just reproduced. Issue is that connectStoragePool exits with "Operation stuck on remote handler" and not on an error that means that master domain was not found. Paikov, please attach logs. Created attachment 475172 [details]
vdsm.log
Attaching log and moving bug back to Assigned.
Patches committed to DistCVS. Moving to MODIFIED to enabled Z-stream inclusion. This bug has lived its purpose by being cloned into rhev 2.2.6. No one needs it now. |
Created attachment 469799 [details] vdsm.log Description of problem: on rhel5.6, with vdsm 2.2.6, which includes NFS time-out fix, it seems as connectStoragePool and reconstructMaster both fails on 'OSError: [Errno 62] Operation stuck on remote handler' and doesn't return valid return response to rhevm. guess that problem will occur when ever Run command will fail on the above, we should catch that exception, and return valid response. repro steps: 1) 1 host (spm) 2) 2 storage domains, 1 ISO (each storage domain on different storage server) 3) block communication to master storage domain on storage X 4) spm-protect will kill vdsm, and start it again 5) spmStop will be sent 6) connect storage server (will fail) 7) connect storage pool will fail, but no return response (see below) also see full log. [root@nott-vds4 ~]# cat /tmp/s Thread-731::INFO::2010-12-20 16:58:29,811::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-735::INFO::2010-12-20 16:58:39,854::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-747::INFO::2010-12-20 16:59:24,237::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-755::INFO::2010-12-20 17:00:08,643::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-763::INFO::2010-12-20 17:00:53,207::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-771::INFO::2010-12-20 17:01:37,611::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-779::INFO::2010-12-20 17:02:21,974::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-787::INFO::2010-12-20 17:03:06,335::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-795::INFO::2010-12-20 17:03:50,716::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=12) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-799::INFO::2010-12-20 17:04:25,064::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-799::INFO::2010-12-20 17:04:25,068::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-803::INFO::2010-12-20 17:04:35,110::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-815::INFO::2010-12-20 17:05:19,478::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-823::INFO::2010-12-20 17:06:03,863::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-831::INFO::2010-12-20 17:06:48,243::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-839::INFO::2010-12-20 17:07:32,625::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-847::INFO::2010-12-20 17:08:17,014::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-855::INFO::2010-12-20 17:09:01,414::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-863::INFO::2010-12-20 17:09:45,881::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=13) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-867::INFO::2010-12-20 17:10:20,438::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-867::INFO::2010-12-20 17:10:20,439::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-871::INFO::2010-12-20 17:10:30,452::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-883::INFO::2010-12-20 17:11:14,831::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-891::INFO::2010-12-20 17:11:59,207::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-899::INFO::2010-12-20 17:12:43,634::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-907::INFO::2010-12-20 17:13:28,022::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-915::INFO::2010-12-20 17:14:12,443::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-923::INFO::2010-12-20 17:14:56,844::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-931::INFO::2010-12-20 17:15:41,301::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=14) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-935::INFO::2010-12-20 17:16:15,649::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-935::INFO::2010-12-20 17:16:15,651::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-939::INFO::2010-12-20 17:16:25,705::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-951::INFO::2010-12-20 17:17:10,074::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-959::INFO::2010-12-20 17:17:54,509::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-967::INFO::2010-12-20 17:18:38,920::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-975::INFO::2010-12-20 17:19:23,294::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-983::INFO::2010-12-20 17:20:07,724::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-991::INFO::2010-12-20 17:20:52,094::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-999::INFO::2010-12-20 17:21:36,619::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=15) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1003::INFO::2010-12-20 17:22:10,985::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-1003::INFO::2010-12-20 17:22:10,987::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-1007::INFO::2010-12-20 17:22:21,021::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1019::INFO::2010-12-20 17:23:05,409::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1027::INFO::2010-12-20 17:23:49,802::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1035::INFO::2010-12-20 17:24:34,267::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1043::INFO::2010-12-20 17:25:18,654::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1051::INFO::2010-12-20 17:26:03,082::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1059::INFO::2010-12-20 17:26:47,466::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1067::INFO::2010-12-20 17:27:31,848::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=16) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1071::INFO::2010-12-20 17:28:06,264::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-1071::INFO::2010-12-20 17:28:06,265::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-1075::INFO::2010-12-20 17:28:16,290::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1087::INFO::2010-12-20 17:29:00,667::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1095::INFO::2010-12-20 17:29:45,023::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1103::INFO::2010-12-20 17:30:29,429::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1111::INFO::2010-12-20 17:31:13,788::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1119::INFO::2010-12-20 17:31:58,315::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1127::INFO::2010-12-20 17:32:42,710::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1135::INFO::2010-12-20 17:33:27,119::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=17) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1139::INFO::2010-12-20 17:34:01,483::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-1139::INFO::2010-12-20 17:34:01,485::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-1143::INFO::2010-12-20 17:34:11,519::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1155::INFO::2010-12-20 17:34:55,957::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1163::INFO::2010-12-20 17:35:40,379::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1171::INFO::2010-12-20 17:36:24,931::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1179::INFO::2010-12-20 17:37:09,311::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1187::INFO::2010-12-20 17:37:53,642::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1195::INFO::2010-12-20 17:38:38,022::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1203::INFO::2010-12-20 17:39:22,439::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=18) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1207::INFO::2010-12-20 17:39:56,803::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-1207::INFO::2010-12-20 17:39:56,805::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-1211::INFO::2010-12-20 17:40:06,828::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1223::INFO::2010-12-20 17:40:51,260::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1231::INFO::2010-12-20 17:41:35,688::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1239::INFO::2010-12-20 17:42:20,290::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1247::INFO::2010-12-20 17:43:04,717::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1255::INFO::2010-12-20 17:43:49,108::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1263::INFO::2010-12-20 17:44:33,506::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1271::INFO::2010-12-20 17:45:17,853::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=19) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1275::INFO::2010-12-20 17:45:52,257::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-1275::INFO::2010-12-20 17:45:52,258::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-1279::INFO::2010-12-20 17:46:02,286::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1291::INFO::2010-12-20 17:46:46,683::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1299::INFO::2010-12-20 17:47:31,042::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1307::INFO::2010-12-20 17:48:15,424::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1315::INFO::2010-12-20 17:48:59,825::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1323::INFO::2010-12-20 17:49:44,205::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1331::INFO::2010-12-20 17:50:28,601::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1339::INFO::2010-12-20 17:51:13,011::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=20) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1343::INFO::2010-12-20 17:51:47,404::dispatcher::95::irs::Run and protect: disconnectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745) Thread-1343::INFO::2010-12-20 17:51:47,406::dispatcher::101::irs::Run and protect: disconnectStoragePool, Return response: {'status': {'message': 'OK', 'code': 0}} Thread-1347::INFO::2010-12-20 17:51:57,417::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=caa3a3bc-177b-4445-a0e3-8cbb9dfac164 masterVersion=21) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1359::INFO::2010-12-20 17:52:41,762::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1367::INFO::2010-12-20 17:53:26,118::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1375::INFO::2010-12-20 17:54:10,504::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool Thread-1383::INFO::2010-12-20 17:54:54,942::dispatcher::95::irs::Run and protect: connectStoragePool, args: ( spUUID=4a358faa-803a-4930-8775-898bd0b8d745 hostID=2 scsiKey=4a358faa-803a-4930-8775-898bd0b8d745 msdUUID=0b0f006e-5aaa-4bbe-86ae-8d3bedef6097 masterVersion=21) File "/usr/share/vdsm/storage/hsm.py", line 535, in public_connectStoragePool