Attachment 891086 Details for Bug 1092914 – vdsm log

vdsm log

vdsm.log (text/plain), 1.31 MB, created by Tomas Dosek on 2014-04-30 09:03:04 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Tomas Dosek

Created: 2014-04-30 09:03:04 UTC

Size: 1.31 MB

patch

obsolete

>Thread-30004::ERROR::2014-04-30 07:01:24,822::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30004::ERROR::2014-04-30 07:01:24,832::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30004::DEBUG::2014-04-30 07:01:24,832::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30004::DEBUG::2014-04-30 07:01:24,838::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30005::DEBUG::2014-04-30 07:02:19,939::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30005::DEBUG::2014-04-30 07:02:19,944::task::579::TaskManager.Task::(_updateState) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::moving from state init -> state preparing
>Thread-30005::INFO::2014-04-30 07:02:19,945::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30005::DEBUG::2014-04-30 07:02:19,945::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`cbc6d67b-9da6-4194-ba6e-461bde080c01`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30005::DEBUG::2014-04-30 07:02:19,945::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30005::DEBUG::2014-04-30 07:02:19,946::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30005::DEBUG::2014-04-30 07:02:19,946::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`cbc6d67b-9da6-4194-ba6e-461bde080c01`::Granted request
>Thread-30005::DEBUG::2014-04-30 07:02:19,946::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`4a8bd313-c02b-4e51-a814-d4e4d4153e43`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30005::DEBUG::2014-04-30 07:02:19,947::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30005::DEBUG::2014-04-30 07:02:19,947::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30005::DEBUG::2014-04-30 07:02:19,947::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`4a8bd313-c02b-4e51-a814-d4e4d4153e43`::Granted request
>Thread-30005::INFO::2014-04-30 07:02:19,948::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30005::DEBUG::2014-04-30 07:02:19,948::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30005::DEBUG::2014-04-30 07:02:19,949::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30005::DEBUG::2014-04-30 07:02:19,949::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30005::DEBUG::2014-04-30 07:02:19,949::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30005::DEBUG::2014-04-30 07:02:19,949::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30005::DEBUG::2014-04-30 07:02:19,949::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30005::DEBUG::2014-04-30 07:02:19,950::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30005::DEBUG::2014-04-30 07:02:19,950::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30005::DEBUG::2014-04-30 07:02:19,950::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30005::DEBUG::2014-04-30 07:02:19,950::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30005::DEBUG::2014-04-30 07:02:19,950::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30005::DEBUG::2014-04-30 07:02:19,973::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30005::DEBUG::2014-04-30 07:02:19,973::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30005::DEBUG::2014-04-30 07:02:19,974::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30005::DEBUG::2014-04-30 07:02:19,975::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30005::DEBUG::2014-04-30 07:02:19,975::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30005::DEBUG::2014-04-30 07:02:19,975::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30005::DEBUG::2014-04-30 07:02:19,976::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30005::DEBUG::2014-04-30 07:02:19,976::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30005::DEBUG::2014-04-30 07:02:19,976::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30005::DEBUG::2014-04-30 07:02:19,976::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30005::ERROR::2014-04-30 07:02:19,977::task::850::TaskManager.Task::(_setError) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30005::DEBUG::2014-04-30 07:02:19,978::task::869::TaskManager.Task::(_run) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::Task._run: 607b79c7-b9ed-4916-8135-f8f659ae27af ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30005::DEBUG::2014-04-30 07:02:19,978::task::1194::TaskManager.Task::(stop) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::stopping in state preparing (force False)
>Thread-30005::DEBUG::2014-04-30 07:02:19,978::task::974::TaskManager.Task::(_decref) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::ref 1 aborting True
>Thread-30005::INFO::2014-04-30 07:02:19,979::task::1151::TaskManager.Task::(prepare) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30005::DEBUG::2014-04-30 07:02:19,979::task::1156::TaskManager.Task::(prepare) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30005::DEBUG::2014-04-30 07:02:19,979::task::974::TaskManager.Task::(_decref) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::ref 0 aborting True
>Thread-30005::DEBUG::2014-04-30 07:02:19,979::task::909::TaskManager.Task::(_doAbort) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::Task._doAbort: force False
>Thread-30005::DEBUG::2014-04-30 07:02:19,979::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30005::DEBUG::2014-04-30 07:02:19,980::task::579::TaskManager.Task::(_updateState) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::moving from state preparing -> state aborting
>Thread-30005::DEBUG::2014-04-30 07:02:19,980::task::534::TaskManager.Task::(__state_aborting) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::_aborting: recover policy none
>Thread-30005::DEBUG::2014-04-30 07:02:19,980::task::579::TaskManager.Task::(_updateState) Task=`607b79c7-b9ed-4916-8135-f8f659ae27af`::moving from state aborting -> state failed
>Thread-30005::DEBUG::2014-04-30 07:02:19,980::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30005::DEBUG::2014-04-30 07:02:19,980::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30005::ERROR::2014-04-30 07:02:19,981::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30004::ERROR::2014-04-30 07:03:29,890::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30004::ERROR::2014-04-30 07:03:29,891::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30004::DEBUG::2014-04-30 07:03:29,891::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30004::DEBUG::2014-04-30 07:03:29,895::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30004::DEBUG::2014-04-30 07:03:29,897::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30004::DEBUG::2014-04-30 07:03:29,898::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30004::INFO::2014-04-30 07:03:29,898::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30004::DEBUG::2014-04-30 07:03:29,898::task::1168::TaskManager.Task::(prepare) Task=`3129367a-e322-43b9-91d5-602ff6e28ff9`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30004::DEBUG::2014-04-30 07:03:29,899::task::579::TaskManager.Task::(_updateState) Task=`3129367a-e322-43b9-91d5-602ff6e28ff9`::moving from state preparing -> state finished
>Thread-30004::DEBUG::2014-04-30 07:03:29,899::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30004::DEBUG::2014-04-30 07:03:29,899::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30004::DEBUG::2014-04-30 07:03:29,899::task::974::TaskManager.Task::(_decref) Task=`3129367a-e322-43b9-91d5-602ff6e28ff9`::ref 0 aborting False
>Thread-30006::DEBUG::2014-04-30 07:04:17,553::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [29de148c]
>Thread-30006::DEBUG::2014-04-30 07:04:17,554::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30006::DEBUG::2014-04-30 07:04:17,571::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30006::DEBUG::2014-04-30 07:04:17,613::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30007::DEBUG::2014-04-30 07:04:17,845::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [29de148c]
>Thread-30007::ERROR::2014-04-30 07:04:17,845::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30007::DEBUG::2014-04-30 07:04:17,846::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30008::DEBUG::2014-04-30 07:04:20,330::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [52052a18]
>Thread-30008::DEBUG::2014-04-30 07:04:20,330::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30008::DEBUG::2014-04-30 07:04:20,344::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30008::DEBUG::2014-04-30 07:04:20,381::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30009::DEBUG::2014-04-30 07:04:20,612::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [52052a18]
>Thread-30009::ERROR::2014-04-30 07:04:20,613::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30009::DEBUG::2014-04-30 07:04:20,614::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30011::DEBUG::2014-04-30 07:04:21,088::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [30e733d0]
>Thread-30011::DEBUG::2014-04-30 07:04:21,089::task::579::TaskManager.Task::(_updateState) Task=`6ae7f98c-bde5-406c-a460-31c797231b64`::moving from state init -> state preparing
>Thread-30011::INFO::2014-04-30 07:04:21,089::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30011::DEBUG::2014-04-30 07:04:21,093::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30011::ERROR::2014-04-30 07:06:26,134::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30011::ERROR::2014-04-30 07:06:26,135::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30011::DEBUG::2014-04-30 07:06:26,135::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30011::DEBUG::2014-04-30 07:06:26,140::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30012::DEBUG::2014-04-30 07:07:21,271::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30012::DEBUG::2014-04-30 07:07:21,272::task::579::TaskManager.Task::(_updateState) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::moving from state init -> state preparing
>Thread-30012::INFO::2014-04-30 07:07:21,272::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30012::DEBUG::2014-04-30 07:07:21,273::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`b4a80582-6ab7-4f07-aa1f-bbe2f1616f76`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30012::DEBUG::2014-04-30 07:07:21,273::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30012::DEBUG::2014-04-30 07:07:21,273::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30012::DEBUG::2014-04-30 07:07:21,273::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`b4a80582-6ab7-4f07-aa1f-bbe2f1616f76`::Granted request
>Thread-30012::DEBUG::2014-04-30 07:07:21,274::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`9705ad23-cc7a-48d8-8e67-94dcfe2b21d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30012::DEBUG::2014-04-30 07:07:21,274::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30012::DEBUG::2014-04-30 07:07:21,274::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30012::DEBUG::2014-04-30 07:07:21,274::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`9705ad23-cc7a-48d8-8e67-94dcfe2b21d8`::Granted request
>Thread-30012::INFO::2014-04-30 07:07:21,275::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30012::DEBUG::2014-04-30 07:07:21,275::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30012::DEBUG::2014-04-30 07:07:21,276::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30012::DEBUG::2014-04-30 07:07:21,276::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30012::DEBUG::2014-04-30 07:07:21,276::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30012::DEBUG::2014-04-30 07:07:21,276::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30012::DEBUG::2014-04-30 07:07:21,277::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30012::DEBUG::2014-04-30 07:07:21,277::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30012::DEBUG::2014-04-30 07:07:21,277::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30012::DEBUG::2014-04-30 07:07:21,277::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30012::DEBUG::2014-04-30 07:07:21,277::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30012::DEBUG::2014-04-30 07:07:21,278::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30012::DEBUG::2014-04-30 07:07:21,300::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30012::DEBUG::2014-04-30 07:07:21,301::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30012::DEBUG::2014-04-30 07:07:21,301::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30012::DEBUG::2014-04-30 07:07:21,302::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30012::DEBUG::2014-04-30 07:07:21,302::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30012::DEBUG::2014-04-30 07:07:21,302::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30012::DEBUG::2014-04-30 07:07:21,303::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30012::DEBUG::2014-04-30 07:07:21,303::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30012::DEBUG::2014-04-30 07:07:21,303::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30012::DEBUG::2014-04-30 07:07:21,303::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30012::ERROR::2014-04-30 07:07:21,304::task::850::TaskManager.Task::(_setError) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30012::DEBUG::2014-04-30 07:07:21,305::task::869::TaskManager.Task::(_run) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::Task._run: 82347c01-87a1-4661-ac7e-afd3373666b4 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30012::DEBUG::2014-04-30 07:07:21,305::task::1194::TaskManager.Task::(stop) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::stopping in state preparing (force False)
>Thread-30012::DEBUG::2014-04-30 07:07:21,305::task::974::TaskManager.Task::(_decref) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::ref 1 aborting True
>Thread-30012::INFO::2014-04-30 07:07:21,306::task::1151::TaskManager.Task::(prepare) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30012::DEBUG::2014-04-30 07:07:21,306::task::1156::TaskManager.Task::(prepare) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30012::DEBUG::2014-04-30 07:07:21,306::task::974::TaskManager.Task::(_decref) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::ref 0 aborting True
>Thread-30012::DEBUG::2014-04-30 07:07:21,306::task::909::TaskManager.Task::(_doAbort) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::Task._doAbort: force False
>Thread-30012::DEBUG::2014-04-30 07:07:21,307::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30012::DEBUG::2014-04-30 07:07:21,307::task::579::TaskManager.Task::(_updateState) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::moving from state preparing -> state aborting
>Thread-30012::DEBUG::2014-04-30 07:07:21,307::task::534::TaskManager.Task::(__state_aborting) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::_aborting: recover policy none
>Thread-30012::DEBUG::2014-04-30 07:07:21,307::task::579::TaskManager.Task::(_updateState) Task=`82347c01-87a1-4661-ac7e-afd3373666b4`::moving from state aborting -> state failed
>Thread-30012::DEBUG::2014-04-30 07:07:21,307::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30012::DEBUG::2014-04-30 07:07:21,308::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30012::ERROR::2014-04-30 07:07:21,308::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30011::ERROR::2014-04-30 07:08:31,186::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30011::ERROR::2014-04-30 07:08:31,187::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30011::DEBUG::2014-04-30 07:08:31,188::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30011::DEBUG::2014-04-30 07:08:31,192::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30011::DEBUG::2014-04-30 07:08:31,193::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30011::DEBUG::2014-04-30 07:08:31,193::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30011::INFO::2014-04-30 07:08:31,194::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30011::DEBUG::2014-04-30 07:08:31,194::task::1168::TaskManager.Task::(prepare) Task=`6ae7f98c-bde5-406c-a460-31c797231b64`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30011::DEBUG::2014-04-30 07:08:31,194::task::579::TaskManager.Task::(_updateState) Task=`6ae7f98c-bde5-406c-a460-31c797231b64`::moving from state preparing -> state finished
>Thread-30011::DEBUG::2014-04-30 07:08:31,195::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30011::DEBUG::2014-04-30 07:08:31,195::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30011::DEBUG::2014-04-30 07:08:31,195::task::974::TaskManager.Task::(_decref) Task=`6ae7f98c-bde5-406c-a460-31c797231b64`::ref 0 aborting False
>Thread-30013::DEBUG::2014-04-30 07:09:17,539::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [9754ae1]
>Thread-30013::DEBUG::2014-04-30 07:09:17,540::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30013::DEBUG::2014-04-30 07:09:17,554::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30013::DEBUG::2014-04-30 07:09:17,596::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30014::DEBUG::2014-04-30 07:09:17,827::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [9754ae1]
>Thread-30014::ERROR::2014-04-30 07:09:17,828::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30014::DEBUG::2014-04-30 07:09:17,829::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30015::DEBUG::2014-04-30 07:09:18,644::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [28f8ef11]
>Thread-30015::DEBUG::2014-04-30 07:09:18,645::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30015::DEBUG::2014-04-30 07:09:18,659::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30015::DEBUG::2014-04-30 07:09:18,697::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30016::DEBUG::2014-04-30 07:09:18,929::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [28f8ef11]
>Thread-30016::ERROR::2014-04-30 07:09:18,930::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30016::DEBUG::2014-04-30 07:09:18,930::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30018::DEBUG::2014-04-30 07:09:19,408::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [37ff3ad0]
>Thread-30018::DEBUG::2014-04-30 07:09:19,408::task::579::TaskManager.Task::(_updateState) Task=`e95b7c68-444a-469e-a2fe-c43f3f3a0cfa`::moving from state init -> state preparing
>Thread-30018::INFO::2014-04-30 07:09:19,409::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30018::DEBUG::2014-04-30 07:09:19,413::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30018::ERROR::2014-04-30 07:11:24,465::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30018::ERROR::2014-04-30 07:11:24,467::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30018::DEBUG::2014-04-30 07:11:24,467::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30018::DEBUG::2014-04-30 07:11:24,474::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30019::DEBUG::2014-04-30 07:12:19,603::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30019::DEBUG::2014-04-30 07:12:19,604::task::579::TaskManager.Task::(_updateState) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::moving from state init -> state preparing
>Thread-30019::INFO::2014-04-30 07:12:19,604::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30019::DEBUG::2014-04-30 07:12:19,605::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4acda079-c2d2-4a7d-b829-9ba0eb17525d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30019::DEBUG::2014-04-30 07:12:19,605::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30019::DEBUG::2014-04-30 07:12:19,606::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30019::DEBUG::2014-04-30 07:12:19,606::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4acda079-c2d2-4a7d-b829-9ba0eb17525d`::Granted request
>Thread-30019::DEBUG::2014-04-30 07:12:19,607::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`5209510d-3ea0-4328-bb5f-a35413ce717c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30019::DEBUG::2014-04-30 07:12:19,607::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30019::DEBUG::2014-04-30 07:12:19,607::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30019::DEBUG::2014-04-30 07:12:19,607::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`5209510d-3ea0-4328-bb5f-a35413ce717c`::Granted request
>Thread-30019::INFO::2014-04-30 07:12:19,608::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30019::DEBUG::2014-04-30 07:12:19,609::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30019::DEBUG::2014-04-30 07:12:19,609::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30019::DEBUG::2014-04-30 07:12:19,609::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30019::DEBUG::2014-04-30 07:12:19,610::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30019::DEBUG::2014-04-30 07:12:19,610::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30019::DEBUG::2014-04-30 07:12:19,610::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30019::DEBUG::2014-04-30 07:12:19,610::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30019::DEBUG::2014-04-30 07:12:19,611::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30019::DEBUG::2014-04-30 07:12:19,611::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30019::DEBUG::2014-04-30 07:12:19,611::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30019::DEBUG::2014-04-30 07:12:19,611::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30019::DEBUG::2014-04-30 07:12:19,634::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30019::DEBUG::2014-04-30 07:12:19,635::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30019::DEBUG::2014-04-30 07:12:19,636::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30019::DEBUG::2014-04-30 07:12:19,636::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30019::DEBUG::2014-04-30 07:12:19,636::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30019::DEBUG::2014-04-30 07:12:19,637::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30019::DEBUG::2014-04-30 07:12:19,637::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30019::DEBUG::2014-04-30 07:12:19,637::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30019::DEBUG::2014-04-30 07:12:19,638::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30019::DEBUG::2014-04-30 07:12:19,638::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30019::ERROR::2014-04-30 07:12:19,638::task::850::TaskManager.Task::(_setError) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30019::DEBUG::2014-04-30 07:12:19,639::task::869::TaskManager.Task::(_run) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::Task._run: af3bb55a-ae5a-4276-8001-04b6ef59a9d6 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30019::DEBUG::2014-04-30 07:12:19,640::task::1194::TaskManager.Task::(stop) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::stopping in state preparing (force False)
>Thread-30019::DEBUG::2014-04-30 07:12:19,640::task::974::TaskManager.Task::(_decref) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::ref 1 aborting True
>Thread-30019::INFO::2014-04-30 07:12:19,640::task::1151::TaskManager.Task::(prepare) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30019::DEBUG::2014-04-30 07:12:19,640::task::1156::TaskManager.Task::(prepare) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30019::DEBUG::2014-04-30 07:12:19,641::task::974::TaskManager.Task::(_decref) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::ref 0 aborting True
>Thread-30019::DEBUG::2014-04-30 07:12:19,641::task::909::TaskManager.Task::(_doAbort) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::Task._doAbort: force False
>Thread-30019::DEBUG::2014-04-30 07:12:19,641::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30019::DEBUG::2014-04-30 07:12:19,641::task::579::TaskManager.Task::(_updateState) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::moving from state preparing -> state aborting
>Thread-30019::DEBUG::2014-04-30 07:12:19,641::task::534::TaskManager.Task::(__state_aborting) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::_aborting: recover policy none
>Thread-30019::DEBUG::2014-04-30 07:12:19,642::task::579::TaskManager.Task::(_updateState) Task=`af3bb55a-ae5a-4276-8001-04b6ef59a9d6`::moving from state aborting -> state failed
>Thread-30019::DEBUG::2014-04-30 07:12:19,642::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30019::DEBUG::2014-04-30 07:12:19,642::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30019::ERROR::2014-04-30 07:12:19,642::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30018::ERROR::2014-04-30 07:13:29,522::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30018::ERROR::2014-04-30 07:13:29,523::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30018::DEBUG::2014-04-30 07:13:29,523::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30018::DEBUG::2014-04-30 07:13:29,527::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30018::DEBUG::2014-04-30 07:13:29,529::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30018::DEBUG::2014-04-30 07:13:29,530::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30018::INFO::2014-04-30 07:13:29,530::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30018::DEBUG::2014-04-30 07:13:29,530::task::1168::TaskManager.Task::(prepare) Task=`e95b7c68-444a-469e-a2fe-c43f3f3a0cfa`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30018::DEBUG::2014-04-30 07:13:29,530::task::579::TaskManager.Task::(_updateState) Task=`e95b7c68-444a-469e-a2fe-c43f3f3a0cfa`::moving from state preparing -> state finished
>Thread-30018::DEBUG::2014-04-30 07:13:29,531::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30018::DEBUG::2014-04-30 07:13:29,531::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30018::DEBUG::2014-04-30 07:13:29,531::task::974::TaskManager.Task::(_decref) Task=`e95b7c68-444a-469e-a2fe-c43f3f3a0cfa`::ref 0 aborting False
>Thread-30020::DEBUG::2014-04-30 07:14:17,519::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [4bf35828]
>Thread-30020::DEBUG::2014-04-30 07:14:17,520::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30020::DEBUG::2014-04-30 07:14:17,536::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30020::DEBUG::2014-04-30 07:14:17,574::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30021::DEBUG::2014-04-30 07:14:17,801::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [4bf35828]
>Thread-30021::ERROR::2014-04-30 07:14:17,801::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30021::DEBUG::2014-04-30 07:14:17,802::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30022::DEBUG::2014-04-30 07:14:19,986::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [59b47c84]
>Thread-30022::DEBUG::2014-04-30 07:14:19,987::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30022::DEBUG::2014-04-30 07:14:20,001::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30022::DEBUG::2014-04-30 07:14:20,038::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30023::DEBUG::2014-04-30 07:14:20,263::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [59b47c84]
>Thread-30023::ERROR::2014-04-30 07:14:20,263::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30023::DEBUG::2014-04-30 07:14:20,264::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30025::DEBUG::2014-04-30 07:14:20,771::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [1ce24c7e]
>Thread-30025::DEBUG::2014-04-30 07:14:20,772::task::579::TaskManager.Task::(_updateState) Task=`82ac296b-27c4-49d3-9d6e-89fd204f7ac4`::moving from state init -> state preparing
>Thread-30025::INFO::2014-04-30 07:14:20,773::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30025::DEBUG::2014-04-30 07:14:20,778::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30025::ERROR::2014-04-30 07:16:25,823::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30025::ERROR::2014-04-30 07:16:25,824::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30025::DEBUG::2014-04-30 07:16:25,825::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30025::DEBUG::2014-04-30 07:16:25,829::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30026::DEBUG::2014-04-30 07:17:20,955::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30026::DEBUG::2014-04-30 07:17:20,956::task::579::TaskManager.Task::(_updateState) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::moving from state init -> state preparing
>Thread-30026::INFO::2014-04-30 07:17:20,956::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30026::DEBUG::2014-04-30 07:17:20,957::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`8c33776b-55d8-425a-bba2-19472fc81b45`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30026::DEBUG::2014-04-30 07:17:20,957::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30026::DEBUG::2014-04-30 07:17:20,958::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30026::DEBUG::2014-04-30 07:17:20,958::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`8c33776b-55d8-425a-bba2-19472fc81b45`::Granted request
>Thread-30026::DEBUG::2014-04-30 07:17:20,959::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`59c224ff-8c06-4884-b729-48568c45de28`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30026::DEBUG::2014-04-30 07:17:20,959::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30026::DEBUG::2014-04-30 07:17:20,959::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30026::DEBUG::2014-04-30 07:17:20,959::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`59c224ff-8c06-4884-b729-48568c45de28`::Granted request
>Thread-30026::INFO::2014-04-30 07:17:20,960::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30026::DEBUG::2014-04-30 07:17:20,961::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30026::DEBUG::2014-04-30 07:17:20,961::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30026::DEBUG::2014-04-30 07:17:20,961::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30026::DEBUG::2014-04-30 07:17:20,962::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30026::DEBUG::2014-04-30 07:17:20,962::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30026::DEBUG::2014-04-30 07:17:20,962::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30026::DEBUG::2014-04-30 07:17:20,962::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30026::DEBUG::2014-04-30 07:17:20,963::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30026::DEBUG::2014-04-30 07:17:20,963::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30026::DEBUG::2014-04-30 07:17:20,963::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30026::DEBUG::2014-04-30 07:17:20,963::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30026::DEBUG::2014-04-30 07:17:20,985::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30026::DEBUG::2014-04-30 07:17:20,986::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30026::DEBUG::2014-04-30 07:17:20,987::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30026::DEBUG::2014-04-30 07:17:20,987::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30026::DEBUG::2014-04-30 07:17:20,987::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30026::DEBUG::2014-04-30 07:17:20,987::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30026::DEBUG::2014-04-30 07:17:20,988::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30026::DEBUG::2014-04-30 07:17:20,988::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30026::DEBUG::2014-04-30 07:17:20,988::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30026::DEBUG::2014-04-30 07:17:20,989::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30026::ERROR::2014-04-30 07:17:20,989::task::850::TaskManager.Task::(_setError) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30026::DEBUG::2014-04-30 07:17:20,990::task::869::TaskManager.Task::(_run) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::Task._run: 4008c16c-3e90-4780-9a84-0b2eb3b0d5fb ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30026::DEBUG::2014-04-30 07:17:20,990::task::1194::TaskManager.Task::(stop) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::stopping in state preparing (force False)
>Thread-30026::DEBUG::2014-04-30 07:17:20,991::task::974::TaskManager.Task::(_decref) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::ref 1 aborting True
>Thread-30026::INFO::2014-04-30 07:17:20,991::task::1151::TaskManager.Task::(prepare) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30026::DEBUG::2014-04-30 07:17:20,991::task::1156::TaskManager.Task::(prepare) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30026::DEBUG::2014-04-30 07:17:20,991::task::974::TaskManager.Task::(_decref) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::ref 0 aborting True
>Thread-30026::DEBUG::2014-04-30 07:17:20,992::task::909::TaskManager.Task::(_doAbort) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::Task._doAbort: force False
>Thread-30026::DEBUG::2014-04-30 07:17:20,992::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30026::DEBUG::2014-04-30 07:17:20,992::task::579::TaskManager.Task::(_updateState) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::moving from state preparing -> state aborting
>Thread-30026::DEBUG::2014-04-30 07:17:20,992::task::534::TaskManager.Task::(__state_aborting) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::_aborting: recover policy none
>Thread-30026::DEBUG::2014-04-30 07:17:20,992::task::579::TaskManager.Task::(_updateState) Task=`4008c16c-3e90-4780-9a84-0b2eb3b0d5fb`::moving from state aborting -> state failed
>Thread-30026::DEBUG::2014-04-30 07:17:20,993::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30026::DEBUG::2014-04-30 07:17:20,993::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30026::ERROR::2014-04-30 07:17:20,993::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30025::ERROR::2014-04-30 07:18:30,870::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30025::ERROR::2014-04-30 07:18:30,871::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30025::DEBUG::2014-04-30 07:18:30,871::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30025::DEBUG::2014-04-30 07:18:30,875::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30025::DEBUG::2014-04-30 07:18:30,876::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30025::DEBUG::2014-04-30 07:18:30,877::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30025::INFO::2014-04-30 07:18:30,877::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30025::DEBUG::2014-04-30 07:18:30,877::task::1168::TaskManager.Task::(prepare) Task=`82ac296b-27c4-49d3-9d6e-89fd204f7ac4`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30025::DEBUG::2014-04-30 07:18:30,878::task::579::TaskManager.Task::(_updateState) Task=`82ac296b-27c4-49d3-9d6e-89fd204f7ac4`::moving from state preparing -> state finished
>Thread-30025::DEBUG::2014-04-30 07:18:30,878::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30025::DEBUG::2014-04-30 07:18:30,878::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30025::DEBUG::2014-04-30 07:18:30,878::task::974::TaskManager.Task::(_decref) Task=`82ac296b-27c4-49d3-9d6e-89fd204f7ac4`::ref 0 aborting False
>Thread-30027::DEBUG::2014-04-30 07:19:17,519::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2e36084d]
>Thread-30027::DEBUG::2014-04-30 07:19:17,520::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30027::DEBUG::2014-04-30 07:19:17,533::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30027::DEBUG::2014-04-30 07:19:17,571::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30028::DEBUG::2014-04-30 07:19:17,796::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2e36084d]
>Thread-30028::ERROR::2014-04-30 07:19:17,797::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30028::DEBUG::2014-04-30 07:19:17,797::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30029::DEBUG::2014-04-30 07:19:18,318::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [3b213f62]
>Thread-30029::DEBUG::2014-04-30 07:19:18,319::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30029::DEBUG::2014-04-30 07:19:18,332::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30029::DEBUG::2014-04-30 07:19:18,370::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30030::DEBUG::2014-04-30 07:19:18,594::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [3b213f62]
>Thread-30030::ERROR::2014-04-30 07:19:18,595::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30030::DEBUG::2014-04-30 07:19:18,595::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30032::DEBUG::2014-04-30 07:19:19,059::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [3c9e1d7d]
>Thread-30032::DEBUG::2014-04-30 07:19:19,059::task::579::TaskManager.Task::(_updateState) Task=`4699a201-19d0-4e74-a923-2ccb5f3b3a24`::moving from state init -> state preparing
>Thread-30032::INFO::2014-04-30 07:19:19,060::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30032::DEBUG::2014-04-30 07:19:19,063::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30032::ERROR::2014-04-30 07:21:24,104::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30032::ERROR::2014-04-30 07:21:24,105::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30032::DEBUG::2014-04-30 07:21:24,106::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30032::DEBUG::2014-04-30 07:21:24,111::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30033::DEBUG::2014-04-30 07:22:19,243::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30033::DEBUG::2014-04-30 07:22:19,244::task::579::TaskManager.Task::(_updateState) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::moving from state init -> state preparing
>Thread-30033::INFO::2014-04-30 07:22:19,244::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30033::DEBUG::2014-04-30 07:22:19,245::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f95e72e9-d563-44e2-aa1c-b3779ae85ae4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30033::DEBUG::2014-04-30 07:22:19,245::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30033::DEBUG::2014-04-30 07:22:19,245::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30033::DEBUG::2014-04-30 07:22:19,245::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f95e72e9-d563-44e2-aa1c-b3779ae85ae4`::Granted request
>Thread-30033::DEBUG::2014-04-30 07:22:19,246::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`82edcfb2-a37f-4549-a949-00484146de01`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30033::DEBUG::2014-04-30 07:22:19,246::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30033::DEBUG::2014-04-30 07:22:19,246::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30033::DEBUG::2014-04-30 07:22:19,246::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`82edcfb2-a37f-4549-a949-00484146de01`::Granted request
>Thread-30033::INFO::2014-04-30 07:22:19,247::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30033::DEBUG::2014-04-30 07:22:19,247::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30033::DEBUG::2014-04-30 07:22:19,248::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30033::DEBUG::2014-04-30 07:22:19,248::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30033::DEBUG::2014-04-30 07:22:19,248::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30033::DEBUG::2014-04-30 07:22:19,248::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30033::DEBUG::2014-04-30 07:22:19,249::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30033::DEBUG::2014-04-30 07:22:19,249::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30033::DEBUG::2014-04-30 07:22:19,249::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30033::DEBUG::2014-04-30 07:22:19,249::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30033::DEBUG::2014-04-30 07:22:19,249::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30033::DEBUG::2014-04-30 07:22:19,250::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30033::DEBUG::2014-04-30 07:22:19,271::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30033::DEBUG::2014-04-30 07:22:19,272::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30033::DEBUG::2014-04-30 07:22:19,273::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30033::DEBUG::2014-04-30 07:22:19,273::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30033::DEBUG::2014-04-30 07:22:19,273::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30033::DEBUG::2014-04-30 07:22:19,274::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30033::DEBUG::2014-04-30 07:22:19,274::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30033::DEBUG::2014-04-30 07:22:19,274::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30033::DEBUG::2014-04-30 07:22:19,275::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30033::DEBUG::2014-04-30 07:22:19,275::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30033::ERROR::2014-04-30 07:22:19,275::task::850::TaskManager.Task::(_setError) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30033::DEBUG::2014-04-30 07:22:19,276::task::869::TaskManager.Task::(_run) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::Task._run: cbf99998-0d14-41f9-b872-34d5c7c5567d ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30033::DEBUG::2014-04-30 07:22:19,276::task::1194::TaskManager.Task::(stop) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::stopping in state preparing (force False)
>Thread-30033::DEBUG::2014-04-30 07:22:19,277::task::974::TaskManager.Task::(_decref) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::ref 1 aborting True
>Thread-30033::INFO::2014-04-30 07:22:19,277::task::1151::TaskManager.Task::(prepare) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30033::DEBUG::2014-04-30 07:22:19,277::task::1156::TaskManager.Task::(prepare) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30033::DEBUG::2014-04-30 07:22:19,277::task::974::TaskManager.Task::(_decref) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::ref 0 aborting True
>Thread-30033::DEBUG::2014-04-30 07:22:19,278::task::909::TaskManager.Task::(_doAbort) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::Task._doAbort: force False
>Thread-30033::DEBUG::2014-04-30 07:22:19,278::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30033::DEBUG::2014-04-30 07:22:19,278::task::579::TaskManager.Task::(_updateState) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::moving from state preparing -> state aborting
>Thread-30033::DEBUG::2014-04-30 07:22:19,278::task::534::TaskManager.Task::(__state_aborting) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::_aborting: recover policy none
>Thread-30033::DEBUG::2014-04-30 07:22:19,279::task::579::TaskManager.Task::(_updateState) Task=`cbf99998-0d14-41f9-b872-34d5c7c5567d`::moving from state aborting -> state failed
>Thread-30033::DEBUG::2014-04-30 07:22:19,279::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30033::DEBUG::2014-04-30 07:22:19,279::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30033::ERROR::2014-04-30 07:22:19,279::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30032::ERROR::2014-04-30 07:23:29,154::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30032::ERROR::2014-04-30 07:23:29,155::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30032::DEBUG::2014-04-30 07:23:29,155::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30032::DEBUG::2014-04-30 07:23:29,159::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30032::DEBUG::2014-04-30 07:23:29,160::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30032::DEBUG::2014-04-30 07:23:29,161::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30032::INFO::2014-04-30 07:23:29,161::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30032::DEBUG::2014-04-30 07:23:29,161::task::1168::TaskManager.Task::(prepare) Task=`4699a201-19d0-4e74-a923-2ccb5f3b3a24`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30032::DEBUG::2014-04-30 07:23:29,162::task::579::TaskManager.Task::(_updateState) Task=`4699a201-19d0-4e74-a923-2ccb5f3b3a24`::moving from state preparing -> state finished
>Thread-30032::DEBUG::2014-04-30 07:23:29,162::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30032::DEBUG::2014-04-30 07:23:29,162::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30032::DEBUG::2014-04-30 07:23:29,162::task::974::TaskManager.Task::(_decref) Task=`4699a201-19d0-4e74-a923-2ccb5f3b3a24`::ref 0 aborting False
>Thread-30034::DEBUG::2014-04-30 07:24:18,161::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5fdcd707]
>Thread-30034::DEBUG::2014-04-30 07:24:18,169::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30034::DEBUG::2014-04-30 07:24:18,185::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30034::DEBUG::2014-04-30 07:24:18,228::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30035::DEBUG::2014-04-30 07:24:18,779::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5fdcd707]
>Thread-30035::ERROR::2014-04-30 07:24:18,785::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30035::DEBUG::2014-04-30 07:24:18,790::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30036::DEBUG::2014-04-30 07:24:19,977::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [17e66f10]
>Thread-30036::DEBUG::2014-04-30 07:24:19,978::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30036::DEBUG::2014-04-30 07:24:19,991::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30036::DEBUG::2014-04-30 07:24:20,029::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30037::DEBUG::2014-04-30 07:24:20,602::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [17e66f10]
>Thread-30037::ERROR::2014-04-30 07:24:20,602::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30037::DEBUG::2014-04-30 07:24:20,603::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30039::DEBUG::2014-04-30 07:24:21,806::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [4ca0e9a6]
>Thread-30039::DEBUG::2014-04-30 07:24:21,806::task::579::TaskManager.Task::(_updateState) Task=`b866e4ce-fe68-41a7-a0aa-3af5e59d5e2d`::moving from state init -> state preparing
>Thread-30039::INFO::2014-04-30 07:24:21,807::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30039::DEBUG::2014-04-30 07:24:21,810::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30039::ERROR::2014-04-30 07:26:26,853::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30039::ERROR::2014-04-30 07:26:26,854::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30039::DEBUG::2014-04-30 07:26:26,855::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30039::DEBUG::2014-04-30 07:26:26,860::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30040::DEBUG::2014-04-30 07:27:21,610::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30040::DEBUG::2014-04-30 07:27:21,611::task::579::TaskManager.Task::(_updateState) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::moving from state init -> state preparing
>Thread-30040::INFO::2014-04-30 07:27:21,611::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30040::DEBUG::2014-04-30 07:27:21,612::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`aad3b9c2-0c28-4e82-99be-4aaddc491419`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30040::DEBUG::2014-04-30 07:27:21,612::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30040::DEBUG::2014-04-30 07:27:21,612::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30040::DEBUG::2014-04-30 07:27:21,613::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`aad3b9c2-0c28-4e82-99be-4aaddc491419`::Granted request
>Thread-30040::DEBUG::2014-04-30 07:27:21,613::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`774aca1a-4311-4786-8c61-9a3689f978d7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30040::DEBUG::2014-04-30 07:27:21,614::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30040::DEBUG::2014-04-30 07:27:21,614::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30040::DEBUG::2014-04-30 07:27:21,614::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`774aca1a-4311-4786-8c61-9a3689f978d7`::Granted request
>Thread-30040::INFO::2014-04-30 07:27:21,615::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30040::DEBUG::2014-04-30 07:27:21,615::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30040::DEBUG::2014-04-30 07:27:21,616::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30040::DEBUG::2014-04-30 07:27:21,616::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30040::DEBUG::2014-04-30 07:27:21,616::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30040::DEBUG::2014-04-30 07:27:21,616::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30040::DEBUG::2014-04-30 07:27:21,617::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30040::DEBUG::2014-04-30 07:27:21,617::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30040::DEBUG::2014-04-30 07:27:21,617::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30040::DEBUG::2014-04-30 07:27:21,617::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30040::DEBUG::2014-04-30 07:27:21,618::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30040::DEBUG::2014-04-30 07:27:21,618::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30040::DEBUG::2014-04-30 07:27:21,640::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30040::DEBUG::2014-04-30 07:27:21,641::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30040::DEBUG::2014-04-30 07:27:21,642::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30040::DEBUG::2014-04-30 07:27:21,642::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30040::DEBUG::2014-04-30 07:27:21,642::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30040::DEBUG::2014-04-30 07:27:21,642::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30040::DEBUG::2014-04-30 07:27:21,643::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30040::DEBUG::2014-04-30 07:27:21,643::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30040::DEBUG::2014-04-30 07:27:21,643::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30040::DEBUG::2014-04-30 07:27:21,643::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30040::ERROR::2014-04-30 07:27:21,644::task::850::TaskManager.Task::(_setError) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30040::DEBUG::2014-04-30 07:27:21,645::task::869::TaskManager.Task::(_run) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::Task._run: 48d8e1ea-b1e7-4268-96bf-928cddb2b301 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30040::DEBUG::2014-04-30 07:27:21,645::task::1194::TaskManager.Task::(stop) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::stopping in state preparing (force False)
>Thread-30040::DEBUG::2014-04-30 07:27:21,645::task::974::TaskManager.Task::(_decref) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::ref 1 aborting True
>Thread-30040::INFO::2014-04-30 07:27:21,646::task::1151::TaskManager.Task::(prepare) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30040::DEBUG::2014-04-30 07:27:21,646::task::1156::TaskManager.Task::(prepare) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30040::DEBUG::2014-04-30 07:27:21,646::task::974::TaskManager.Task::(_decref) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::ref 0 aborting True
>Thread-30040::DEBUG::2014-04-30 07:27:21,646::task::909::TaskManager.Task::(_doAbort) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::Task._doAbort: force False
>Thread-30040::DEBUG::2014-04-30 07:27:21,647::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30040::DEBUG::2014-04-30 07:27:21,647::task::579::TaskManager.Task::(_updateState) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::moving from state preparing -> state aborting
>Thread-30040::DEBUG::2014-04-30 07:27:21,647::task::534::TaskManager.Task::(__state_aborting) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::_aborting: recover policy none
>Thread-30040::DEBUG::2014-04-30 07:27:21,647::task::579::TaskManager.Task::(_updateState) Task=`48d8e1ea-b1e7-4268-96bf-928cddb2b301`::moving from state aborting -> state failed
>Thread-30040::DEBUG::2014-04-30 07:27:21,648::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30040::DEBUG::2014-04-30 07:27:21,648::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30040::ERROR::2014-04-30 07:27:21,648::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30039::ERROR::2014-04-30 07:28:31,902::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30039::ERROR::2014-04-30 07:28:31,903::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30039::DEBUG::2014-04-30 07:28:31,903::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30039::DEBUG::2014-04-30 07:28:31,907::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30039::DEBUG::2014-04-30 07:28:31,909::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30039::DEBUG::2014-04-30 07:28:31,909::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30039::INFO::2014-04-30 07:28:31,909::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30039::DEBUG::2014-04-30 07:28:31,909::task::1168::TaskManager.Task::(prepare) Task=`b866e4ce-fe68-41a7-a0aa-3af5e59d5e2d`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30039::DEBUG::2014-04-30 07:28:31,910::task::579::TaskManager.Task::(_updateState) Task=`b866e4ce-fe68-41a7-a0aa-3af5e59d5e2d`::moving from state preparing -> state finished
>Thread-30039::DEBUG::2014-04-30 07:28:31,910::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30039::DEBUG::2014-04-30 07:28:31,910::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30039::DEBUG::2014-04-30 07:28:31,910::task::974::TaskManager.Task::(_decref) Task=`b866e4ce-fe68-41a7-a0aa-3af5e59d5e2d`::ref 0 aborting False
>Thread-30041::DEBUG::2014-04-30 07:29:17,501::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [c2cb076]
>Thread-30041::DEBUG::2014-04-30 07:29:17,502::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30041::DEBUG::2014-04-30 07:29:17,516::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30041::DEBUG::2014-04-30 07:29:17,553::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30042::DEBUG::2014-04-30 07:29:17,778::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [c2cb076]
>Thread-30042::ERROR::2014-04-30 07:29:17,779::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30042::DEBUG::2014-04-30 07:29:17,779::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30043::DEBUG::2014-04-30 07:29:18,974::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [407b01d9]
>Thread-30043::DEBUG::2014-04-30 07:29:18,974::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30043::DEBUG::2014-04-30 07:29:18,988::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30043::DEBUG::2014-04-30 07:29:19,026::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30044::DEBUG::2014-04-30 07:29:19,261::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [407b01d9]
>Thread-30044::ERROR::2014-04-30 07:29:19,261::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30044::DEBUG::2014-04-30 07:29:19,262::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30046::DEBUG::2014-04-30 07:29:19,723::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [67f7f3c2]
>Thread-30046::DEBUG::2014-04-30 07:29:19,724::task::579::TaskManager.Task::(_updateState) Task=`5f4cdde7-9070-45d5-9518-4209e1d421d1`::moving from state init -> state preparing
>Thread-30046::INFO::2014-04-30 07:29:19,724::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30046::DEBUG::2014-04-30 07:29:19,729::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30046::ERROR::2014-04-30 07:31:24,771::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30046::ERROR::2014-04-30 07:31:24,773::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30046::DEBUG::2014-04-30 07:31:24,773::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30046::DEBUG::2014-04-30 07:31:24,778::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30047::DEBUG::2014-04-30 07:32:19,900::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30047::DEBUG::2014-04-30 07:32:19,901::task::579::TaskManager.Task::(_updateState) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::moving from state init -> state preparing
>Thread-30047::INFO::2014-04-30 07:32:19,901::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30047::DEBUG::2014-04-30 07:32:19,902::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`43c49a85-14ef-41e4-a978-0f74de805a20`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30047::DEBUG::2014-04-30 07:32:19,902::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30047::DEBUG::2014-04-30 07:32:19,903::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30047::DEBUG::2014-04-30 07:32:19,903::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`43c49a85-14ef-41e4-a978-0f74de805a20`::Granted request
>Thread-30047::DEBUG::2014-04-30 07:32:19,903::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`398d362d-8b5b-4070-8e91-5a157756c4f0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30047::DEBUG::2014-04-30 07:32:19,904::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30047::DEBUG::2014-04-30 07:32:19,904::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30047::DEBUG::2014-04-30 07:32:19,904::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`398d362d-8b5b-4070-8e91-5a157756c4f0`::Granted request
>Thread-30047::INFO::2014-04-30 07:32:19,904::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30047::DEBUG::2014-04-30 07:32:19,905::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30047::DEBUG::2014-04-30 07:32:19,905::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30047::DEBUG::2014-04-30 07:32:19,905::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30047::DEBUG::2014-04-30 07:32:19,906::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30047::DEBUG::2014-04-30 07:32:19,906::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30047::DEBUG::2014-04-30 07:32:19,906::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30047::DEBUG::2014-04-30 07:32:19,906::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30047::DEBUG::2014-04-30 07:32:19,906::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30047::DEBUG::2014-04-30 07:32:19,907::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30047::DEBUG::2014-04-30 07:32:19,907::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30047::DEBUG::2014-04-30 07:32:19,907::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30047::DEBUG::2014-04-30 07:32:19,929::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30047::DEBUG::2014-04-30 07:32:19,930::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30047::DEBUG::2014-04-30 07:32:19,931::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30047::DEBUG::2014-04-30 07:32:19,931::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30047::DEBUG::2014-04-30 07:32:19,931::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30047::DEBUG::2014-04-30 07:32:19,931::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30047::DEBUG::2014-04-30 07:32:19,932::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30047::DEBUG::2014-04-30 07:32:19,932::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30047::DEBUG::2014-04-30 07:32:19,932::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30047::DEBUG::2014-04-30 07:32:19,932::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30047::ERROR::2014-04-30 07:32:19,933::task::850::TaskManager.Task::(_setError) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30047::DEBUG::2014-04-30 07:32:19,934::task::869::TaskManager.Task::(_run) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::Task._run: cfc7ebed-0931-45f0-bc60-f3cfea19766c ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30047::DEBUG::2014-04-30 07:32:19,934::task::1194::TaskManager.Task::(stop) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::stopping in state preparing (force False)
>Thread-30047::DEBUG::2014-04-30 07:32:19,934::task::974::TaskManager.Task::(_decref) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::ref 1 aborting True
>Thread-30047::INFO::2014-04-30 07:32:19,935::task::1151::TaskManager.Task::(prepare) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30047::DEBUG::2014-04-30 07:32:19,935::task::1156::TaskManager.Task::(prepare) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30047::DEBUG::2014-04-30 07:32:19,935::task::974::TaskManager.Task::(_decref) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::ref 0 aborting True
>Thread-30047::DEBUG::2014-04-30 07:32:19,935::task::909::TaskManager.Task::(_doAbort) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::Task._doAbort: force False
>Thread-30047::DEBUG::2014-04-30 07:32:19,936::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30047::DEBUG::2014-04-30 07:32:19,936::task::579::TaskManager.Task::(_updateState) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::moving from state preparing -> state aborting
>Thread-30047::DEBUG::2014-04-30 07:32:19,936::task::534::TaskManager.Task::(__state_aborting) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::_aborting: recover policy none
>Thread-30047::DEBUG::2014-04-30 07:32:19,936::task::579::TaskManager.Task::(_updateState) Task=`cfc7ebed-0931-45f0-bc60-f3cfea19766c`::moving from state aborting -> state failed
>Thread-30047::DEBUG::2014-04-30 07:32:19,936::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30047::DEBUG::2014-04-30 07:32:19,937::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30047::ERROR::2014-04-30 07:32:19,937::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30046::ERROR::2014-04-30 07:33:29,819::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30046::ERROR::2014-04-30 07:33:29,820::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30046::DEBUG::2014-04-30 07:33:29,820::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30046::DEBUG::2014-04-30 07:33:29,824::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30046::DEBUG::2014-04-30 07:33:29,825::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30046::DEBUG::2014-04-30 07:33:29,826::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30046::INFO::2014-04-30 07:33:29,826::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30046::DEBUG::2014-04-30 07:33:29,826::task::1168::TaskManager.Task::(prepare) Task=`5f4cdde7-9070-45d5-9518-4209e1d421d1`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30046::DEBUG::2014-04-30 07:33:29,826::task::579::TaskManager.Task::(_updateState) Task=`5f4cdde7-9070-45d5-9518-4209e1d421d1`::moving from state preparing -> state finished
>Thread-30046::DEBUG::2014-04-30 07:33:29,827::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30046::DEBUG::2014-04-30 07:33:29,827::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30046::DEBUG::2014-04-30 07:33:29,827::task::974::TaskManager.Task::(_decref) Task=`5f4cdde7-9070-45d5-9518-4209e1d421d1`::ref 0 aborting False
>Thread-30048::DEBUG::2014-04-30 07:34:17,490::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [3fa2372]
>Thread-30048::DEBUG::2014-04-30 07:34:17,491::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30048::DEBUG::2014-04-30 07:34:17,506::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30048::DEBUG::2014-04-30 07:34:17,543::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30049::DEBUG::2014-04-30 07:34:17,768::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [3fa2372]
>Thread-30049::ERROR::2014-04-30 07:34:17,769::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30049::DEBUG::2014-04-30 07:34:17,770::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30050::DEBUG::2014-04-30 07:34:20,290::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5c65b29f]
>Thread-30050::DEBUG::2014-04-30 07:34:20,290::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30050::DEBUG::2014-04-30 07:34:20,304::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30050::DEBUG::2014-04-30 07:34:20,342::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30051::DEBUG::2014-04-30 07:34:20,575::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5c65b29f]
>Thread-30051::ERROR::2014-04-30 07:34:20,575::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30051::DEBUG::2014-04-30 07:34:20,576::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30053::DEBUG::2014-04-30 07:34:21,046::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [666573bb]
>Thread-30053::DEBUG::2014-04-30 07:34:21,046::task::579::TaskManager.Task::(_updateState) Task=`e7e7216d-a283-48a9-af6f-087e45e5a3c6`::moving from state init -> state preparing
>Thread-30053::INFO::2014-04-30 07:34:21,047::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30053::DEBUG::2014-04-30 07:34:21,052::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30053::ERROR::2014-04-30 07:36:26,094::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30053::ERROR::2014-04-30 07:36:26,095::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30053::DEBUG::2014-04-30 07:36:26,096::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30053::DEBUG::2014-04-30 07:36:26,100::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30054::DEBUG::2014-04-30 07:37:21,222::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30054::DEBUG::2014-04-30 07:37:21,223::task::579::TaskManager.Task::(_updateState) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::moving from state init -> state preparing
>Thread-30054::INFO::2014-04-30 07:37:21,223::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30054::DEBUG::2014-04-30 07:37:21,224::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a8cd1718-1a94-470b-8ef8-0d1ffe141414`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30054::DEBUG::2014-04-30 07:37:21,224::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30054::DEBUG::2014-04-30 07:37:21,224::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30054::DEBUG::2014-04-30 07:37:21,225::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a8cd1718-1a94-470b-8ef8-0d1ffe141414`::Granted request
>Thread-30054::DEBUG::2014-04-30 07:37:21,225::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`9c595a2d-a621-40de-8a30-10c382cf3bca`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30054::DEBUG::2014-04-30 07:37:21,225::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30054::DEBUG::2014-04-30 07:37:21,226::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30054::DEBUG::2014-04-30 07:37:21,226::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`9c595a2d-a621-40de-8a30-10c382cf3bca`::Granted request
>Thread-30054::INFO::2014-04-30 07:37:21,226::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30054::DEBUG::2014-04-30 07:37:21,227::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30054::DEBUG::2014-04-30 07:37:21,227::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30054::DEBUG::2014-04-30 07:37:21,227::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30054::DEBUG::2014-04-30 07:37:21,227::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30054::DEBUG::2014-04-30 07:37:21,228::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30054::DEBUG::2014-04-30 07:37:21,228::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30054::DEBUG::2014-04-30 07:37:21,228::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30054::DEBUG::2014-04-30 07:37:21,228::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30054::DEBUG::2014-04-30 07:37:21,228::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30054::DEBUG::2014-04-30 07:37:21,229::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30054::DEBUG::2014-04-30 07:37:21,229::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30054::DEBUG::2014-04-30 07:37:21,251::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30054::DEBUG::2014-04-30 07:37:21,251::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30054::DEBUG::2014-04-30 07:37:21,252::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30054::DEBUG::2014-04-30 07:37:21,253::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30054::DEBUG::2014-04-30 07:37:21,253::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30054::DEBUG::2014-04-30 07:37:21,253::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30054::DEBUG::2014-04-30 07:37:21,253::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30054::DEBUG::2014-04-30 07:37:21,254::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30054::DEBUG::2014-04-30 07:37:21,254::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30054::DEBUG::2014-04-30 07:37:21,254::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30054::ERROR::2014-04-30 07:37:21,254::task::850::TaskManager.Task::(_setError) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30054::DEBUG::2014-04-30 07:37:21,255::task::869::TaskManager.Task::(_run) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::Task._run: 3663ca96-bf3d-435b-a475-1161f6aa0fc3 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30054::DEBUG::2014-04-30 07:37:21,256::task::1194::TaskManager.Task::(stop) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::stopping in state preparing (force False)
>Thread-30054::DEBUG::2014-04-30 07:37:21,256::task::974::TaskManager.Task::(_decref) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::ref 1 aborting True
>Thread-30054::INFO::2014-04-30 07:37:21,256::task::1151::TaskManager.Task::(prepare) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30054::DEBUG::2014-04-30 07:37:21,257::task::1156::TaskManager.Task::(prepare) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30054::DEBUG::2014-04-30 07:37:21,257::task::974::TaskManager.Task::(_decref) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::ref 0 aborting True
>Thread-30054::DEBUG::2014-04-30 07:37:21,257::task::909::TaskManager.Task::(_doAbort) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::Task._doAbort: force False
>Thread-30054::DEBUG::2014-04-30 07:37:21,257::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30054::DEBUG::2014-04-30 07:37:21,257::task::579::TaskManager.Task::(_updateState) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::moving from state preparing -> state aborting
>Thread-30054::DEBUG::2014-04-30 07:37:21,258::task::534::TaskManager.Task::(__state_aborting) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::_aborting: recover policy none
>Thread-30054::DEBUG::2014-04-30 07:37:21,258::task::579::TaskManager.Task::(_updateState) Task=`3663ca96-bf3d-435b-a475-1161f6aa0fc3`::moving from state aborting -> state failed
>Thread-30054::DEBUG::2014-04-30 07:37:21,258::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30054::DEBUG::2014-04-30 07:37:21,258::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30054::ERROR::2014-04-30 07:37:21,259::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30053::ERROR::2014-04-30 07:38:31,141::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30053::ERROR::2014-04-30 07:38:31,142::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30053::DEBUG::2014-04-30 07:38:31,142::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30053::DEBUG::2014-04-30 07:38:31,146::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30053::DEBUG::2014-04-30 07:38:31,147::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30053::DEBUG::2014-04-30 07:38:31,148::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30053::INFO::2014-04-30 07:38:31,148::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30053::DEBUG::2014-04-30 07:38:31,148::task::1168::TaskManager.Task::(prepare) Task=`e7e7216d-a283-48a9-af6f-087e45e5a3c6`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30053::DEBUG::2014-04-30 07:38:31,148::task::579::TaskManager.Task::(_updateState) Task=`e7e7216d-a283-48a9-af6f-087e45e5a3c6`::moving from state preparing -> state finished
>Thread-30053::DEBUG::2014-04-30 07:38:31,149::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30053::DEBUG::2014-04-30 07:38:31,149::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30053::DEBUG::2014-04-30 07:38:31,149::task::974::TaskManager.Task::(_decref) Task=`e7e7216d-a283-48a9-af6f-087e45e5a3c6`::ref 0 aborting False
>Thread-30055::DEBUG::2014-04-30 07:39:17,486::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [6bc5b380]
>Thread-30055::DEBUG::2014-04-30 07:39:17,487::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30055::DEBUG::2014-04-30 07:39:17,500::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30055::DEBUG::2014-04-30 07:39:17,538::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30056::DEBUG::2014-04-30 07:39:17,763::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [6bc5b380]
>Thread-30056::ERROR::2014-04-30 07:39:17,764::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30056::DEBUG::2014-04-30 07:39:17,764::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30057::DEBUG::2014-04-30 07:39:18,589::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [4f5a836]
>Thread-30057::DEBUG::2014-04-30 07:39:18,590::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30057::DEBUG::2014-04-30 07:39:18,604::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30057::DEBUG::2014-04-30 07:39:18,641::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30058::DEBUG::2014-04-30 07:39:18,866::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [4f5a836]
>Thread-30058::ERROR::2014-04-30 07:39:18,866::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30058::DEBUG::2014-04-30 07:39:18,867::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30060::DEBUG::2014-04-30 07:39:19,331::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [2a48947d]
>Thread-30060::DEBUG::2014-04-30 07:39:19,332::task::579::TaskManager.Task::(_updateState) Task=`687f2638-565e-445b-9c49-626ab5e88927`::moving from state init -> state preparing
>Thread-30060::INFO::2014-04-30 07:39:19,332::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30060::DEBUG::2014-04-30 07:39:19,338::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30060::ERROR::2014-04-30 07:41:24,392::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30060::ERROR::2014-04-30 07:41:24,394::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30060::DEBUG::2014-04-30 07:41:24,395::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30060::DEBUG::2014-04-30 07:41:24,400::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30061::DEBUG::2014-04-30 07:42:19,519::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30061::DEBUG::2014-04-30 07:42:19,520::task::579::TaskManager.Task::(_updateState) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::moving from state init -> state preparing
>Thread-30061::INFO::2014-04-30 07:42:19,520::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30061::DEBUG::2014-04-30 07:42:19,521::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`941694e0-6974-4e44-b80b-da6b9f47b5ac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30061::DEBUG::2014-04-30 07:42:19,521::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30061::DEBUG::2014-04-30 07:42:19,521::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30061::DEBUG::2014-04-30 07:42:19,522::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`941694e0-6974-4e44-b80b-da6b9f47b5ac`::Granted request
>Thread-30061::DEBUG::2014-04-30 07:42:19,522::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`ad17b2af-e34c-4215-b8c4-907c53508698`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30061::DEBUG::2014-04-30 07:42:19,523::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30061::DEBUG::2014-04-30 07:42:19,523::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30061::DEBUG::2014-04-30 07:42:19,523::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`ad17b2af-e34c-4215-b8c4-907c53508698`::Granted request
>Thread-30061::INFO::2014-04-30 07:42:19,524::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30061::DEBUG::2014-04-30 07:42:19,525::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30061::DEBUG::2014-04-30 07:42:19,525::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30061::DEBUG::2014-04-30 07:42:19,525::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30061::DEBUG::2014-04-30 07:42:19,525::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30061::DEBUG::2014-04-30 07:42:19,526::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30061::DEBUG::2014-04-30 07:42:19,526::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30061::DEBUG::2014-04-30 07:42:19,526::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30061::DEBUG::2014-04-30 07:42:19,527::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30061::DEBUG::2014-04-30 07:42:19,527::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30061::DEBUG::2014-04-30 07:42:19,527::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30061::DEBUG::2014-04-30 07:42:19,527::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30061::DEBUG::2014-04-30 07:42:19,550::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30061::DEBUG::2014-04-30 07:42:19,550::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30061::DEBUG::2014-04-30 07:42:19,553::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30061::DEBUG::2014-04-30 07:42:19,553::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30061::DEBUG::2014-04-30 07:42:19,553::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30061::DEBUG::2014-04-30 07:42:19,553::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30061::DEBUG::2014-04-30 07:42:19,554::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30061::DEBUG::2014-04-30 07:42:19,554::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30061::DEBUG::2014-04-30 07:42:19,554::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30061::DEBUG::2014-04-30 07:42:19,555::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30061::ERROR::2014-04-30 07:42:19,555::task::850::TaskManager.Task::(_setError) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30061::DEBUG::2014-04-30 07:42:19,556::task::869::TaskManager.Task::(_run) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::Task._run: 8687e4bb-9291-46a0-96b4-fd26abe23c42 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30061::DEBUG::2014-04-30 07:42:19,556::task::1194::TaskManager.Task::(stop) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::stopping in state preparing (force False)
>Thread-30061::DEBUG::2014-04-30 07:42:19,557::task::974::TaskManager.Task::(_decref) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::ref 1 aborting True
>Thread-30061::INFO::2014-04-30 07:42:19,557::task::1151::TaskManager.Task::(prepare) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30061::DEBUG::2014-04-30 07:42:19,557::task::1156::TaskManager.Task::(prepare) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30061::DEBUG::2014-04-30 07:42:19,557::task::974::TaskManager.Task::(_decref) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::ref 0 aborting True
>Thread-30061::DEBUG::2014-04-30 07:42:19,558::task::909::TaskManager.Task::(_doAbort) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::Task._doAbort: force False
>Thread-30061::DEBUG::2014-04-30 07:42:19,558::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30061::DEBUG::2014-04-30 07:42:19,558::task::579::TaskManager.Task::(_updateState) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::moving from state preparing -> state aborting
>Thread-30061::DEBUG::2014-04-30 07:42:19,558::task::534::TaskManager.Task::(__state_aborting) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::_aborting: recover policy none
>Thread-30061::DEBUG::2014-04-30 07:42:19,558::task::579::TaskManager.Task::(_updateState) Task=`8687e4bb-9291-46a0-96b4-fd26abe23c42`::moving from state aborting -> state failed
>Thread-30061::DEBUG::2014-04-30 07:42:19,559::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30061::DEBUG::2014-04-30 07:42:19,559::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30061::ERROR::2014-04-30 07:42:19,559::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30060::ERROR::2014-04-30 07:43:29,442::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30060::ERROR::2014-04-30 07:43:29,443::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30060::DEBUG::2014-04-30 07:43:29,444::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30060::DEBUG::2014-04-30 07:43:29,448::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30060::DEBUG::2014-04-30 07:43:29,449::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30060::DEBUG::2014-04-30 07:43:29,450::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30060::INFO::2014-04-30 07:43:29,450::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30060::DEBUG::2014-04-30 07:43:29,450::task::1168::TaskManager.Task::(prepare) Task=`687f2638-565e-445b-9c49-626ab5e88927`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30060::DEBUG::2014-04-30 07:43:29,450::task::579::TaskManager.Task::(_updateState) Task=`687f2638-565e-445b-9c49-626ab5e88927`::moving from state preparing -> state finished
>Thread-30060::DEBUG::2014-04-30 07:43:29,451::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30060::DEBUG::2014-04-30 07:43:29,451::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30060::DEBUG::2014-04-30 07:43:29,451::task::974::TaskManager.Task::(_decref) Task=`687f2638-565e-445b-9c49-626ab5e88927`::ref 0 aborting False
>Thread-30062::DEBUG::2014-04-30 07:44:17,480::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [4f9bec9f]
>Thread-30062::DEBUG::2014-04-30 07:44:17,481::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30062::DEBUG::2014-04-30 07:44:17,496::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30062::DEBUG::2014-04-30 07:44:17,533::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30063::DEBUG::2014-04-30 07:44:17,758::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [4f9bec9f]
>Thread-30063::ERROR::2014-04-30 07:44:17,759::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30063::DEBUG::2014-04-30 07:44:17,759::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30064::DEBUG::2014-04-30 07:44:19,900::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [4a50c78f]
>Thread-30064::DEBUG::2014-04-30 07:44:19,901::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30064::DEBUG::2014-04-30 07:44:19,915::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30064::DEBUG::2014-04-30 07:44:19,958::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30065::DEBUG::2014-04-30 07:44:20,183::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [4a50c78f]
>Thread-30065::ERROR::2014-04-30 07:44:20,183::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30065::DEBUG::2014-04-30 07:44:20,184::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30067::DEBUG::2014-04-30 07:44:20,669::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [7ad8334]
>Thread-30067::DEBUG::2014-04-30 07:44:20,670::task::579::TaskManager.Task::(_updateState) Task=`2a25b735-75c0-45c5-b910-5c4dfbc07ef7`::moving from state init -> state preparing
>Thread-30067::INFO::2014-04-30 07:44:20,670::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30067::DEBUG::2014-04-30 07:44:20,675::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30067::ERROR::2014-04-30 07:46:25,716::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30067::ERROR::2014-04-30 07:46:25,717::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30067::DEBUG::2014-04-30 07:46:25,717::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30067::DEBUG::2014-04-30 07:46:25,722::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30068::DEBUG::2014-04-30 07:47:20,867::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30068::DEBUG::2014-04-30 07:47:20,868::task::579::TaskManager.Task::(_updateState) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::moving from state init -> state preparing
>Thread-30068::INFO::2014-04-30 07:47:20,868::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30068::DEBUG::2014-04-30 07:47:20,869::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`be4d398e-7065-4270-b4a0-96e9ef6bd652`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30068::DEBUG::2014-04-30 07:47:20,869::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30068::DEBUG::2014-04-30 07:47:20,869::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30068::DEBUG::2014-04-30 07:47:20,870::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`be4d398e-7065-4270-b4a0-96e9ef6bd652`::Granted request
>Thread-30068::DEBUG::2014-04-30 07:47:20,870::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`45f79c53-f7d6-4428-918e-6791a94f53e5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30068::DEBUG::2014-04-30 07:47:20,871::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30068::DEBUG::2014-04-30 07:47:20,871::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30068::DEBUG::2014-04-30 07:47:20,871::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`45f79c53-f7d6-4428-918e-6791a94f53e5`::Granted request
>Thread-30068::INFO::2014-04-30 07:47:20,872::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30068::DEBUG::2014-04-30 07:47:20,872::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30068::DEBUG::2014-04-30 07:47:20,873::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30068::DEBUG::2014-04-30 07:47:20,873::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30068::DEBUG::2014-04-30 07:47:20,873::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30068::DEBUG::2014-04-30 07:47:20,873::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30068::DEBUG::2014-04-30 07:47:20,874::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30068::DEBUG::2014-04-30 07:47:20,874::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30068::DEBUG::2014-04-30 07:47:20,874::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30068::DEBUG::2014-04-30 07:47:20,874::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30068::DEBUG::2014-04-30 07:47:20,875::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30068::DEBUG::2014-04-30 07:47:20,875::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30068::DEBUG::2014-04-30 07:47:20,897::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30068::DEBUG::2014-04-30 07:47:20,897::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30068::DEBUG::2014-04-30 07:47:20,898::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30068::DEBUG::2014-04-30 07:47:20,899::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30068::DEBUG::2014-04-30 07:47:20,899::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30068::DEBUG::2014-04-30 07:47:20,899::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30068::DEBUG::2014-04-30 07:47:20,899::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30068::DEBUG::2014-04-30 07:47:20,900::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30068::DEBUG::2014-04-30 07:47:20,900::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30068::DEBUG::2014-04-30 07:47:20,900::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30068::ERROR::2014-04-30 07:47:20,900::task::850::TaskManager.Task::(_setError) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30068::DEBUG::2014-04-30 07:47:20,902::task::869::TaskManager.Task::(_run) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::Task._run: bf121925-968f-4e8d-a4b6-3754026dae6b ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30068::DEBUG::2014-04-30 07:47:20,902::task::1194::TaskManager.Task::(stop) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::stopping in state preparing (force False)
>Thread-30068::DEBUG::2014-04-30 07:47:20,902::task::974::TaskManager.Task::(_decref) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::ref 1 aborting True
>Thread-30068::INFO::2014-04-30 07:47:20,902::task::1151::TaskManager.Task::(prepare) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30068::DEBUG::2014-04-30 07:47:20,903::task::1156::TaskManager.Task::(prepare) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30068::DEBUG::2014-04-30 07:47:20,903::task::974::TaskManager.Task::(_decref) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::ref 0 aborting True
>Thread-30068::DEBUG::2014-04-30 07:47:20,903::task::909::TaskManager.Task::(_doAbort) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::Task._doAbort: force False
>Thread-30068::DEBUG::2014-04-30 07:47:20,903::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30068::DEBUG::2014-04-30 07:47:20,904::task::579::TaskManager.Task::(_updateState) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::moving from state preparing -> state aborting
>Thread-30068::DEBUG::2014-04-30 07:47:20,904::task::534::TaskManager.Task::(__state_aborting) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::_aborting: recover policy none
>Thread-30068::DEBUG::2014-04-30 07:47:20,904::task::579::TaskManager.Task::(_updateState) Task=`bf121925-968f-4e8d-a4b6-3754026dae6b`::moving from state aborting -> state failed
>Thread-30068::DEBUG::2014-04-30 07:47:20,904::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30068::DEBUG::2014-04-30 07:47:20,904::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30068::ERROR::2014-04-30 07:47:20,905::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30067::ERROR::2014-04-30 07:48:30,763::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30067::ERROR::2014-04-30 07:48:30,764::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30067::DEBUG::2014-04-30 07:48:30,764::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30067::DEBUG::2014-04-30 07:48:30,768::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30067::DEBUG::2014-04-30 07:48:30,769::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30067::DEBUG::2014-04-30 07:48:30,769::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30067::INFO::2014-04-30 07:48:30,770::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30067::DEBUG::2014-04-30 07:48:30,770::task::1168::TaskManager.Task::(prepare) Task=`2a25b735-75c0-45c5-b910-5c4dfbc07ef7`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30067::DEBUG::2014-04-30 07:48:30,770::task::579::TaskManager.Task::(_updateState) Task=`2a25b735-75c0-45c5-b910-5c4dfbc07ef7`::moving from state preparing -> state finished
>Thread-30067::DEBUG::2014-04-30 07:48:30,770::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30067::DEBUG::2014-04-30 07:48:30,771::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30067::DEBUG::2014-04-30 07:48:30,771::task::974::TaskManager.Task::(_decref) Task=`2a25b735-75c0-45c5-b910-5c4dfbc07ef7`::ref 0 aborting False
>Thread-30069::DEBUG::2014-04-30 07:49:17,470::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [b97c315]
>Thread-30069::DEBUG::2014-04-30 07:49:17,471::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30069::DEBUG::2014-04-30 07:49:17,484::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30069::DEBUG::2014-04-30 07:49:17,521::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30070::DEBUG::2014-04-30 07:49:17,746::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [b97c315]
>Thread-30070::ERROR::2014-04-30 07:49:17,747::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30070::DEBUG::2014-04-30 07:49:17,747::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30071::DEBUG::2014-04-30 07:49:18,236::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [10f6b248]
>Thread-30071::DEBUG::2014-04-30 07:49:18,237::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30071::DEBUG::2014-04-30 07:49:18,250::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30071::DEBUG::2014-04-30 07:49:18,287::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30072::DEBUG::2014-04-30 07:49:18,513::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [10f6b248]
>Thread-30072::ERROR::2014-04-30 07:49:18,514::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30072::DEBUG::2014-04-30 07:49:18,515::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30074::DEBUG::2014-04-30 07:49:18,979::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [5a19a9eb]
>Thread-30074::DEBUG::2014-04-30 07:49:18,979::task::579::TaskManager.Task::(_updateState) Task=`f6c81232-97f3-4ac9-b135-12b6bf70ac5c`::moving from state init -> state preparing
>Thread-30074::INFO::2014-04-30 07:49:18,980::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30074::DEBUG::2014-04-30 07:49:18,984::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30074::ERROR::2014-04-30 07:51:24,028::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30074::ERROR::2014-04-30 07:51:24,033::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30074::DEBUG::2014-04-30 07:51:24,033::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30074::DEBUG::2014-04-30 07:51:24,038::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30075::DEBUG::2014-04-30 07:52:19,161::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30075::DEBUG::2014-04-30 07:52:19,161::task::579::TaskManager.Task::(_updateState) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::moving from state init -> state preparing
>Thread-30075::INFO::2014-04-30 07:52:19,162::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30075::DEBUG::2014-04-30 07:52:19,162::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7d1d8e87-6d7e-4ed9-872f-4b443e2c7438`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30075::DEBUG::2014-04-30 07:52:19,163::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30075::DEBUG::2014-04-30 07:52:19,163::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30075::DEBUG::2014-04-30 07:52:19,163::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7d1d8e87-6d7e-4ed9-872f-4b443e2c7438`::Granted request
>Thread-30075::DEBUG::2014-04-30 07:52:19,164::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`167779d8-9312-4264-bcb9-1294936aa76f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30075::DEBUG::2014-04-30 07:52:19,164::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30075::DEBUG::2014-04-30 07:52:19,165::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30075::DEBUG::2014-04-30 07:52:19,165::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`167779d8-9312-4264-bcb9-1294936aa76f`::Granted request
>Thread-30075::INFO::2014-04-30 07:52:19,165::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30075::DEBUG::2014-04-30 07:52:19,166::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30075::DEBUG::2014-04-30 07:52:19,166::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30075::DEBUG::2014-04-30 07:52:19,167::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30075::DEBUG::2014-04-30 07:52:19,167::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30075::DEBUG::2014-04-30 07:52:19,167::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30075::DEBUG::2014-04-30 07:52:19,167::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30075::DEBUG::2014-04-30 07:52:19,168::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30075::DEBUG::2014-04-30 07:52:19,168::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30075::DEBUG::2014-04-30 07:52:19,168::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30075::DEBUG::2014-04-30 07:52:19,168::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30075::DEBUG::2014-04-30 07:52:19,169::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30075::DEBUG::2014-04-30 07:52:19,191::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30075::DEBUG::2014-04-30 07:52:19,191::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30075::DEBUG::2014-04-30 07:52:19,192::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30075::DEBUG::2014-04-30 07:52:19,192::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30075::DEBUG::2014-04-30 07:52:19,193::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30075::DEBUG::2014-04-30 07:52:19,193::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30075::DEBUG::2014-04-30 07:52:19,193::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30075::DEBUG::2014-04-30 07:52:19,194::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30075::DEBUG::2014-04-30 07:52:19,194::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30075::DEBUG::2014-04-30 07:52:19,194::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30075::ERROR::2014-04-30 07:52:19,194::task::850::TaskManager.Task::(_setError) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30075::DEBUG::2014-04-30 07:52:19,195::task::869::TaskManager.Task::(_run) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::Task._run: 0a792fe6-7753-40a0-8770-12e833982f34 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30075::DEBUG::2014-04-30 07:52:19,196::task::1194::TaskManager.Task::(stop) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::stopping in state preparing (force False)
>Thread-30075::DEBUG::2014-04-30 07:52:19,196::task::974::TaskManager.Task::(_decref) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::ref 1 aborting True
>Thread-30075::INFO::2014-04-30 07:52:19,196::task::1151::TaskManager.Task::(prepare) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30075::DEBUG::2014-04-30 07:52:19,196::task::1156::TaskManager.Task::(prepare) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30075::DEBUG::2014-04-30 07:52:19,197::task::974::TaskManager.Task::(_decref) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::ref 0 aborting True
>Thread-30075::DEBUG::2014-04-30 07:52:19,197::task::909::TaskManager.Task::(_doAbort) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::Task._doAbort: force False
>Thread-30075::DEBUG::2014-04-30 07:52:19,197::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30075::DEBUG::2014-04-30 07:52:19,197::task::579::TaskManager.Task::(_updateState) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::moving from state preparing -> state aborting
>Thread-30075::DEBUG::2014-04-30 07:52:19,198::task::534::TaskManager.Task::(__state_aborting) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::_aborting: recover policy none
>Thread-30075::DEBUG::2014-04-30 07:52:19,198::task::579::TaskManager.Task::(_updateState) Task=`0a792fe6-7753-40a0-8770-12e833982f34`::moving from state aborting -> state failed
>Thread-30075::DEBUG::2014-04-30 07:52:19,198::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30075::DEBUG::2014-04-30 07:52:19,198::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30075::ERROR::2014-04-30 07:52:19,198::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30074::ERROR::2014-04-30 07:53:29,079::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30074::ERROR::2014-04-30 07:53:29,080::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30074::DEBUG::2014-04-30 07:53:29,080::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30074::DEBUG::2014-04-30 07:53:29,084::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30074::DEBUG::2014-04-30 07:53:29,085::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30074::DEBUG::2014-04-30 07:53:29,086::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30074::INFO::2014-04-30 07:53:29,086::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30074::DEBUG::2014-04-30 07:53:29,086::task::1168::TaskManager.Task::(prepare) Task=`f6c81232-97f3-4ac9-b135-12b6bf70ac5c`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30074::DEBUG::2014-04-30 07:53:29,086::task::579::TaskManager.Task::(_updateState) Task=`f6c81232-97f3-4ac9-b135-12b6bf70ac5c`::moving from state preparing -> state finished
>Thread-30074::DEBUG::2014-04-30 07:53:29,087::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30074::DEBUG::2014-04-30 07:53:29,087::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30074::DEBUG::2014-04-30 07:53:29,087::task::974::TaskManager.Task::(_decref) Task=`f6c81232-97f3-4ac9-b135-12b6bf70ac5c`::ref 0 aborting False
>Thread-30076::DEBUG::2014-04-30 07:54:17,472::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [6ba7ee73]
>Thread-30076::DEBUG::2014-04-30 07:54:17,478::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30076::DEBUG::2014-04-30 07:54:17,491::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30076::DEBUG::2014-04-30 07:54:17,529::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30077::DEBUG::2014-04-30 07:54:17,761::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [6ba7ee73]
>Thread-30077::ERROR::2014-04-30 07:54:17,761::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30077::DEBUG::2014-04-30 07:54:17,762::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30078::DEBUG::2014-04-30 07:54:19,529::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [58a53c0f]
>Thread-30078::DEBUG::2014-04-30 07:54:19,531::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30078::DEBUG::2014-04-30 07:54:19,544::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30078::DEBUG::2014-04-30 07:54:19,582::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30079::DEBUG::2014-04-30 07:54:19,835::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [58a53c0f]
>Thread-30079::ERROR::2014-04-30 07:54:19,835::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30079::DEBUG::2014-04-30 07:54:19,836::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30081::DEBUG::2014-04-30 07:54:20,297::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [f71f218]
>Thread-30081::DEBUG::2014-04-30 07:54:20,298::task::579::TaskManager.Task::(_updateState) Task=`a351d088-2b5e-4f07-96be-134d8fd007fa`::moving from state init -> state preparing
>Thread-30081::INFO::2014-04-30 07:54:20,299::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30081::DEBUG::2014-04-30 07:54:20,303::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30081::ERROR::2014-04-30 07:56:25,346::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30081::ERROR::2014-04-30 07:56:25,347::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30081::DEBUG::2014-04-30 07:56:25,347::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30081::DEBUG::2014-04-30 07:56:25,352::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30082::DEBUG::2014-04-30 07:57:21,077::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30082::DEBUG::2014-04-30 07:57:21,078::task::579::TaskManager.Task::(_updateState) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::moving from state init -> state preparing
>Thread-30082::INFO::2014-04-30 07:57:21,078::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30082::DEBUG::2014-04-30 07:57:21,079::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e38d4839-01bf-4a64-9bec-ab73c1564774`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30082::DEBUG::2014-04-30 07:57:21,079::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30082::DEBUG::2014-04-30 07:57:21,079::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30082::DEBUG::2014-04-30 07:57:21,080::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e38d4839-01bf-4a64-9bec-ab73c1564774`::Granted request
>Thread-30082::DEBUG::2014-04-30 07:57:21,080::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`36b1eb61-8247-4563-b7ec-8519e1d3ae4c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30082::DEBUG::2014-04-30 07:57:21,080::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30082::DEBUG::2014-04-30 07:57:21,081::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30082::DEBUG::2014-04-30 07:57:21,081::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`36b1eb61-8247-4563-b7ec-8519e1d3ae4c`::Granted request
>Thread-30082::INFO::2014-04-30 07:57:21,081::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30082::DEBUG::2014-04-30 07:57:21,082::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30082::DEBUG::2014-04-30 07:57:21,082::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30082::DEBUG::2014-04-30 07:57:21,082::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30082::DEBUG::2014-04-30 07:57:21,082::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30082::DEBUG::2014-04-30 07:57:21,083::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30082::DEBUG::2014-04-30 07:57:21,083::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30082::DEBUG::2014-04-30 07:57:21,083::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30082::DEBUG::2014-04-30 07:57:21,083::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30082::DEBUG::2014-04-30 07:57:21,083::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30082::DEBUG::2014-04-30 07:57:21,084::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30082::DEBUG::2014-04-30 07:57:21,084::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30082::DEBUG::2014-04-30 07:57:21,106::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30082::DEBUG::2014-04-30 07:57:21,106::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30082::DEBUG::2014-04-30 07:57:21,107::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30082::DEBUG::2014-04-30 07:57:21,108::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30082::DEBUG::2014-04-30 07:57:21,108::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30082::DEBUG::2014-04-30 07:57:21,108::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30082::DEBUG::2014-04-30 07:57:21,108::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30082::DEBUG::2014-04-30 07:57:21,109::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30082::DEBUG::2014-04-30 07:57:21,109::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30082::DEBUG::2014-04-30 07:57:21,109::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30082::ERROR::2014-04-30 07:57:21,110::task::850::TaskManager.Task::(_setError) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30082::DEBUG::2014-04-30 07:57:21,111::task::869::TaskManager.Task::(_run) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::Task._run: 7f879efb-7fed-48a0-a57d-dcd5df97e5ac ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30082::DEBUG::2014-04-30 07:57:21,111::task::1194::TaskManager.Task::(stop) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::stopping in state preparing (force False)
>Thread-30082::DEBUG::2014-04-30 07:57:21,111::task::974::TaskManager.Task::(_decref) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::ref 1 aborting True
>Thread-30082::INFO::2014-04-30 07:57:21,111::task::1151::TaskManager.Task::(prepare) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30082::DEBUG::2014-04-30 07:57:21,112::task::1156::TaskManager.Task::(prepare) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30082::DEBUG::2014-04-30 07:57:21,112::task::974::TaskManager.Task::(_decref) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::ref 0 aborting True
>Thread-30082::DEBUG::2014-04-30 07:57:21,112::task::909::TaskManager.Task::(_doAbort) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::Task._doAbort: force False
>Thread-30082::DEBUG::2014-04-30 07:57:21,112::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30082::DEBUG::2014-04-30 07:57:21,113::task::579::TaskManager.Task::(_updateState) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::moving from state preparing -> state aborting
>Thread-30082::DEBUG::2014-04-30 07:57:21,113::task::534::TaskManager.Task::(__state_aborting) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::_aborting: recover policy none
>Thread-30082::DEBUG::2014-04-30 07:57:21,113::task::579::TaskManager.Task::(_updateState) Task=`7f879efb-7fed-48a0-a57d-dcd5df97e5ac`::moving from state aborting -> state failed
>Thread-30082::DEBUG::2014-04-30 07:57:21,113::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30082::DEBUG::2014-04-30 07:57:21,113::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30082::ERROR::2014-04-30 07:57:21,114::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30081::ERROR::2014-04-30 07:58:30,395::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30081::ERROR::2014-04-30 07:58:30,396::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30081::DEBUG::2014-04-30 07:58:30,396::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30081::DEBUG::2014-04-30 07:58:30,400::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30081::DEBUG::2014-04-30 07:58:30,401::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30081::DEBUG::2014-04-30 07:58:30,402::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30081::INFO::2014-04-30 07:58:30,402::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30081::DEBUG::2014-04-30 07:58:30,402::task::1168::TaskManager.Task::(prepare) Task=`a351d088-2b5e-4f07-96be-134d8fd007fa`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30081::DEBUG::2014-04-30 07:58:30,402::task::579::TaskManager.Task::(_updateState) Task=`a351d088-2b5e-4f07-96be-134d8fd007fa`::moving from state preparing -> state finished
>Thread-30081::DEBUG::2014-04-30 07:58:30,403::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30081::DEBUG::2014-04-30 07:58:30,403::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30081::DEBUG::2014-04-30 07:58:30,403::task::974::TaskManager.Task::(_decref) Task=`a351d088-2b5e-4f07-96be-134d8fd007fa`::ref 0 aborting False
>Thread-30083::DEBUG::2014-04-30 07:59:18,103::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [7b311fd8]
>Thread-30083::DEBUG::2014-04-30 07:59:18,104::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30083::DEBUG::2014-04-30 07:59:18,117::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30083::DEBUG::2014-04-30 07:59:18,154::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30084::DEBUG::2014-04-30 07:59:18,706::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [7b311fd8]
>Thread-30084::ERROR::2014-04-30 07:59:18,706::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30084::DEBUG::2014-04-30 07:59:18,707::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30085::DEBUG::2014-04-30 07:59:21,815::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [1e8d0255]
>Thread-30085::DEBUG::2014-04-30 07:59:21,817::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30085::DEBUG::2014-04-30 07:59:21,831::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30085::DEBUG::2014-04-30 07:59:21,870::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30086::DEBUG::2014-04-30 07:59:22,451::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [1e8d0255]
>Thread-30086::ERROR::2014-04-30 07:59:22,451::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30086::DEBUG::2014-04-30 07:59:22,452::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30088::DEBUG::2014-04-30 07:59:23,635::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [3ed6f6fb]
>Thread-30088::DEBUG::2014-04-30 07:59:23,636::task::579::TaskManager.Task::(_updateState) Task=`06dd1af4-4a92-4dae-af5b-58a74e8ae2fe`::moving from state init -> state preparing
>Thread-30088::INFO::2014-04-30 07:59:23,636::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30088::DEBUG::2014-04-30 07:59:23,640::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30088::ERROR::2014-04-30 08:01:28,689::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30088::ERROR::2014-04-30 08:01:28,691::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30088::DEBUG::2014-04-30 08:01:28,692::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30088::DEBUG::2014-04-30 08:01:28,698::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30089::DEBUG::2014-04-30 08:02:28,487::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30089::DEBUG::2014-04-30 08:02:28,488::task::579::TaskManager.Task::(_updateState) Task=`58101850-85c3-4483-a254-b39f19de25dc`::moving from state init -> state preparing
>Thread-30089::INFO::2014-04-30 08:02:28,489::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30089::DEBUG::2014-04-30 08:02:28,489::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`8bc3e970-6c86-4ba1-b2a9-93dd66494444`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30089::DEBUG::2014-04-30 08:02:28,489::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30089::DEBUG::2014-04-30 08:02:28,490::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30089::DEBUG::2014-04-30 08:02:28,490::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`8bc3e970-6c86-4ba1-b2a9-93dd66494444`::Granted request
>Thread-30089::DEBUG::2014-04-30 08:02:28,491::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`1c7436cf-04dd-48d1-a742-ed030d3744c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30089::DEBUG::2014-04-30 08:02:28,491::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30089::DEBUG::2014-04-30 08:02:28,491::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30089::DEBUG::2014-04-30 08:02:28,491::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`1c7436cf-04dd-48d1-a742-ed030d3744c2`::Granted request
>Thread-30089::INFO::2014-04-30 08:02:28,492::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30089::DEBUG::2014-04-30 08:02:28,492::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30089::DEBUG::2014-04-30 08:02:28,493::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30089::DEBUG::2014-04-30 08:02:28,493::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30089::DEBUG::2014-04-30 08:02:28,493::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30089::DEBUG::2014-04-30 08:02:28,493::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30089::DEBUG::2014-04-30 08:02:28,493::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30089::DEBUG::2014-04-30 08:02:28,494::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30089::DEBUG::2014-04-30 08:02:28,494::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30089::DEBUG::2014-04-30 08:02:28,494::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30089::DEBUG::2014-04-30 08:02:28,494::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30089::DEBUG::2014-04-30 08:02:28,494::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30089::DEBUG::2014-04-30 08:02:28,516::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30089::DEBUG::2014-04-30 08:02:28,517::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30089::DEBUG::2014-04-30 08:02:28,518::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30089::DEBUG::2014-04-30 08:02:28,518::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30089::DEBUG::2014-04-30 08:02:28,518::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30089::DEBUG::2014-04-30 08:02:28,519::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30089::DEBUG::2014-04-30 08:02:28,519::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30089::DEBUG::2014-04-30 08:02:28,519::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30089::DEBUG::2014-04-30 08:02:28,520::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30089::DEBUG::2014-04-30 08:02:28,520::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30089::ERROR::2014-04-30 08:02:28,520::task::850::TaskManager.Task::(_setError) Task=`58101850-85c3-4483-a254-b39f19de25dc`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30089::DEBUG::2014-04-30 08:02:28,521::task::869::TaskManager.Task::(_run) Task=`58101850-85c3-4483-a254-b39f19de25dc`::Task._run: 58101850-85c3-4483-a254-b39f19de25dc ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30089::DEBUG::2014-04-30 08:02:28,521::task::1194::TaskManager.Task::(stop) Task=`58101850-85c3-4483-a254-b39f19de25dc`::stopping in state preparing (force False)
>Thread-30089::DEBUG::2014-04-30 08:02:28,522::task::974::TaskManager.Task::(_decref) Task=`58101850-85c3-4483-a254-b39f19de25dc`::ref 1 aborting True
>Thread-30089::INFO::2014-04-30 08:02:28,522::task::1151::TaskManager.Task::(prepare) Task=`58101850-85c3-4483-a254-b39f19de25dc`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30089::DEBUG::2014-04-30 08:02:28,522::task::1156::TaskManager.Task::(prepare) Task=`58101850-85c3-4483-a254-b39f19de25dc`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30089::DEBUG::2014-04-30 08:02:28,523::task::974::TaskManager.Task::(_decref) Task=`58101850-85c3-4483-a254-b39f19de25dc`::ref 0 aborting True
>Thread-30089::DEBUG::2014-04-30 08:02:28,523::task::909::TaskManager.Task::(_doAbort) Task=`58101850-85c3-4483-a254-b39f19de25dc`::Task._doAbort: force False
>Thread-30089::DEBUG::2014-04-30 08:02:28,523::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30089::DEBUG::2014-04-30 08:02:28,523::task::579::TaskManager.Task::(_updateState) Task=`58101850-85c3-4483-a254-b39f19de25dc`::moving from state preparing -> state aborting
>Thread-30089::DEBUG::2014-04-30 08:02:28,523::task::534::TaskManager.Task::(__state_aborting) Task=`58101850-85c3-4483-a254-b39f19de25dc`::_aborting: recover policy none
>Thread-30089::DEBUG::2014-04-30 08:02:28,524::task::579::TaskManager.Task::(_updateState) Task=`58101850-85c3-4483-a254-b39f19de25dc`::moving from state aborting -> state failed
>Thread-30089::DEBUG::2014-04-30 08:02:28,524::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30089::DEBUG::2014-04-30 08:02:28,524::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30089::ERROR::2014-04-30 08:02:28,524::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30088::ERROR::2014-04-30 08:03:33,743::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30088::ERROR::2014-04-30 08:03:33,744::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30088::DEBUG::2014-04-30 08:03:33,745::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30088::DEBUG::2014-04-30 08:03:33,749::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30088::DEBUG::2014-04-30 08:03:33,751::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30088::DEBUG::2014-04-30 08:03:33,752::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30088::INFO::2014-04-30 08:03:33,752::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30088::DEBUG::2014-04-30 08:03:33,752::task::1168::TaskManager.Task::(prepare) Task=`06dd1af4-4a92-4dae-af5b-58a74e8ae2fe`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30088::DEBUG::2014-04-30 08:03:33,753::task::579::TaskManager.Task::(_updateState) Task=`06dd1af4-4a92-4dae-af5b-58a74e8ae2fe`::moving from state preparing -> state finished
>Thread-30088::DEBUG::2014-04-30 08:03:33,753::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30088::DEBUG::2014-04-30 08:03:33,753::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30088::DEBUG::2014-04-30 08:03:33,753::task::974::TaskManager.Task::(_decref) Task=`06dd1af4-4a92-4dae-af5b-58a74e8ae2fe`::ref 0 aborting False
>Thread-30090::DEBUG::2014-04-30 08:04:17,477::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [227b194]
>Thread-30090::DEBUG::2014-04-30 08:04:17,478::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30090::DEBUG::2014-04-30 08:04:17,494::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30090::DEBUG::2014-04-30 08:04:17,536::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30091::DEBUG::2014-04-30 08:04:17,776::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [227b194]
>Thread-30091::ERROR::2014-04-30 08:04:17,777::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30091::DEBUG::2014-04-30 08:04:17,777::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30092::DEBUG::2014-04-30 08:04:19,883::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [700b1a]
>Thread-30092::DEBUG::2014-04-30 08:04:19,884::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30092::DEBUG::2014-04-30 08:04:19,897::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30092::DEBUG::2014-04-30 08:04:19,935::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30093::DEBUG::2014-04-30 08:04:20,175::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [700b1a]
>Thread-30093::ERROR::2014-04-30 08:04:20,175::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30093::DEBUG::2014-04-30 08:04:20,176::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30095::DEBUG::2014-04-30 08:04:20,670::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [6dc9ecd9]
>Thread-30095::DEBUG::2014-04-30 08:04:20,670::task::579::TaskManager.Task::(_updateState) Task=`6a4b63aa-04b7-43ba-b50d-7c5ad427f9f5`::moving from state init -> state preparing
>Thread-30095::INFO::2014-04-30 08:04:20,671::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30095::DEBUG::2014-04-30 08:04:20,674::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30095::ERROR::2014-04-30 08:06:25,717::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30095::ERROR::2014-04-30 08:06:25,717::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30095::DEBUG::2014-04-30 08:06:25,718::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30095::DEBUG::2014-04-30 08:06:25,723::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30096::DEBUG::2014-04-30 08:07:20,832::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30096::DEBUG::2014-04-30 08:07:20,833::task::579::TaskManager.Task::(_updateState) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::moving from state init -> state preparing
>Thread-30096::INFO::2014-04-30 08:07:20,833::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30096::DEBUG::2014-04-30 08:07:20,834::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f64acccf-bf65-4cc8-b386-0b46bcc62af4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30096::DEBUG::2014-04-30 08:07:20,834::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30096::DEBUG::2014-04-30 08:07:20,834::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30096::DEBUG::2014-04-30 08:07:20,835::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f64acccf-bf65-4cc8-b386-0b46bcc62af4`::Granted request
>Thread-30096::DEBUG::2014-04-30 08:07:20,835::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`d8f3944e-3588-4341-acd7-785a1cc9b794`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30096::DEBUG::2014-04-30 08:07:20,835::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30096::DEBUG::2014-04-30 08:07:20,836::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30096::DEBUG::2014-04-30 08:07:20,836::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`d8f3944e-3588-4341-acd7-785a1cc9b794`::Granted request
>Thread-30096::INFO::2014-04-30 08:07:20,836::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30096::DEBUG::2014-04-30 08:07:20,837::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30096::DEBUG::2014-04-30 08:07:20,837::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30096::DEBUG::2014-04-30 08:07:20,837::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30096::DEBUG::2014-04-30 08:07:20,837::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30096::DEBUG::2014-04-30 08:07:20,838::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30096::DEBUG::2014-04-30 08:07:20,838::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30096::DEBUG::2014-04-30 08:07:20,838::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30096::DEBUG::2014-04-30 08:07:20,838::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30096::DEBUG::2014-04-30 08:07:20,838::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30096::DEBUG::2014-04-30 08:07:20,839::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30096::DEBUG::2014-04-30 08:07:20,839::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30096::DEBUG::2014-04-30 08:07:20,861::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30096::DEBUG::2014-04-30 08:07:20,862::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30096::DEBUG::2014-04-30 08:07:20,863::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30096::DEBUG::2014-04-30 08:07:20,863::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30096::DEBUG::2014-04-30 08:07:20,863::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30096::DEBUG::2014-04-30 08:07:20,864::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30096::DEBUG::2014-04-30 08:07:20,864::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30096::DEBUG::2014-04-30 08:07:20,864::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30096::DEBUG::2014-04-30 08:07:20,864::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30096::DEBUG::2014-04-30 08:07:20,865::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30096::ERROR::2014-04-30 08:07:20,865::task::850::TaskManager.Task::(_setError) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30096::DEBUG::2014-04-30 08:07:20,866::task::869::TaskManager.Task::(_run) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::Task._run: f915ba1b-47c4-4837-96a0-5ac9ff975e34 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30096::DEBUG::2014-04-30 08:07:20,866::task::1194::TaskManager.Task::(stop) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::stopping in state preparing (force False)
>Thread-30096::DEBUG::2014-04-30 08:07:20,867::task::974::TaskManager.Task::(_decref) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::ref 1 aborting True
>Thread-30096::INFO::2014-04-30 08:07:20,867::task::1151::TaskManager.Task::(prepare) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30096::DEBUG::2014-04-30 08:07:20,867::task::1156::TaskManager.Task::(prepare) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30096::DEBUG::2014-04-30 08:07:20,867::task::974::TaskManager.Task::(_decref) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::ref 0 aborting True
>Thread-30096::DEBUG::2014-04-30 08:07:20,868::task::909::TaskManager.Task::(_doAbort) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::Task._doAbort: force False
>Thread-30096::DEBUG::2014-04-30 08:07:20,868::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30096::DEBUG::2014-04-30 08:07:20,868::task::579::TaskManager.Task::(_updateState) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::moving from state preparing -> state aborting
>Thread-30096::DEBUG::2014-04-30 08:07:20,868::task::534::TaskManager.Task::(__state_aborting) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::_aborting: recover policy none
>Thread-30096::DEBUG::2014-04-30 08:07:20,868::task::579::TaskManager.Task::(_updateState) Task=`f915ba1b-47c4-4837-96a0-5ac9ff975e34`::moving from state aborting -> state failed
>Thread-30096::DEBUG::2014-04-30 08:07:20,869::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30096::DEBUG::2014-04-30 08:07:20,869::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30096::ERROR::2014-04-30 08:07:20,869::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30095::ERROR::2014-04-30 08:08:30,764::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30095::ERROR::2014-04-30 08:08:30,765::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30095::DEBUG::2014-04-30 08:08:30,765::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30095::DEBUG::2014-04-30 08:08:30,769::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30095::DEBUG::2014-04-30 08:08:30,770::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30095::DEBUG::2014-04-30 08:08:30,770::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30095::INFO::2014-04-30 08:08:30,771::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30095::DEBUG::2014-04-30 08:08:30,771::task::1168::TaskManager.Task::(prepare) Task=`6a4b63aa-04b7-43ba-b50d-7c5ad427f9f5`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30095::DEBUG::2014-04-30 08:08:30,771::task::579::TaskManager.Task::(_updateState) Task=`6a4b63aa-04b7-43ba-b50d-7c5ad427f9f5`::moving from state preparing -> state finished
>Thread-30095::DEBUG::2014-04-30 08:08:30,771::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30095::DEBUG::2014-04-30 08:08:30,772::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30095::DEBUG::2014-04-30 08:08:30,772::task::974::TaskManager.Task::(_decref) Task=`6a4b63aa-04b7-43ba-b50d-7c5ad427f9f5`::ref 0 aborting False
>Thread-30097::DEBUG::2014-04-30 08:09:17,444::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [211b7dd1]
>Thread-30097::DEBUG::2014-04-30 08:09:17,445::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30097::DEBUG::2014-04-30 08:09:17,458::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30097::DEBUG::2014-04-30 08:09:17,496::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30098::DEBUG::2014-04-30 08:09:17,721::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [211b7dd1]
>Thread-30098::ERROR::2014-04-30 08:09:17,721::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30098::DEBUG::2014-04-30 08:09:17,726::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30099::DEBUG::2014-04-30 08:09:18,237::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [408b3dcd]
>Thread-30099::DEBUG::2014-04-30 08:09:18,238::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30099::DEBUG::2014-04-30 08:09:18,251::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30099::DEBUG::2014-04-30 08:09:18,288::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30100::DEBUG::2014-04-30 08:09:18,513::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [408b3dcd]
>Thread-30100::ERROR::2014-04-30 08:09:18,513::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30100::DEBUG::2014-04-30 08:09:18,514::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30102::DEBUG::2014-04-30 08:09:18,976::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [3728854a]
>Thread-30102::DEBUG::2014-04-30 08:09:18,977::task::579::TaskManager.Task::(_updateState) Task=`d002ac97-9716-49f5-a864-f6d3a0129de0`::moving from state init -> state preparing
>Thread-30102::INFO::2014-04-30 08:09:18,977::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30102::DEBUG::2014-04-30 08:09:18,982::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30102::ERROR::2014-04-30 08:11:24,025::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30102::ERROR::2014-04-30 08:11:24,031::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30102::DEBUG::2014-04-30 08:11:24,031::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30102::DEBUG::2014-04-30 08:11:24,036::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30103::DEBUG::2014-04-30 08:12:19,163::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30103::DEBUG::2014-04-30 08:12:19,164::task::579::TaskManager.Task::(_updateState) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::moving from state init -> state preparing
>Thread-30103::INFO::2014-04-30 08:12:19,165::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30103::DEBUG::2014-04-30 08:12:19,167::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`37b06657-4999-452a-ab45-4766719b193a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30103::DEBUG::2014-04-30 08:12:19,167::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30103::DEBUG::2014-04-30 08:12:19,167::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30103::DEBUG::2014-04-30 08:12:19,167::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`37b06657-4999-452a-ab45-4766719b193a`::Granted request
>Thread-30103::DEBUG::2014-04-30 08:12:19,168::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`239dbeca-1cc4-4d40-93f6-caf00c3a7fbe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30103::DEBUG::2014-04-30 08:12:19,168::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30103::DEBUG::2014-04-30 08:12:19,169::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30103::DEBUG::2014-04-30 08:12:19,169::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`239dbeca-1cc4-4d40-93f6-caf00c3a7fbe`::Granted request
>Thread-30103::INFO::2014-04-30 08:12:19,170::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30103::DEBUG::2014-04-30 08:12:19,170::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30103::DEBUG::2014-04-30 08:12:19,171::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30103::DEBUG::2014-04-30 08:12:19,171::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30103::DEBUG::2014-04-30 08:12:19,171::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30103::DEBUG::2014-04-30 08:12:19,171::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30103::DEBUG::2014-04-30 08:12:19,172::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30103::DEBUG::2014-04-30 08:12:19,172::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30103::DEBUG::2014-04-30 08:12:19,172::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30103::DEBUG::2014-04-30 08:12:19,172::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30103::DEBUG::2014-04-30 08:12:19,173::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30103::DEBUG::2014-04-30 08:12:19,173::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30103::DEBUG::2014-04-30 08:12:19,195::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30103::DEBUG::2014-04-30 08:12:19,196::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30103::DEBUG::2014-04-30 08:12:19,196::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30103::DEBUG::2014-04-30 08:12:19,197::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30103::DEBUG::2014-04-30 08:12:19,197::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30103::DEBUG::2014-04-30 08:12:19,197::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30103::DEBUG::2014-04-30 08:12:19,198::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30103::DEBUG::2014-04-30 08:12:19,198::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30103::DEBUG::2014-04-30 08:12:19,198::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30103::DEBUG::2014-04-30 08:12:19,198::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30103::ERROR::2014-04-30 08:12:19,199::task::850::TaskManager.Task::(_setError) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30103::DEBUG::2014-04-30 08:12:19,200::task::869::TaskManager.Task::(_run) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::Task._run: 68281660-716c-4ea4-bedf-038c1e8ded06 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30103::DEBUG::2014-04-30 08:12:19,200::task::1194::TaskManager.Task::(stop) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::stopping in state preparing (force False)
>Thread-30103::DEBUG::2014-04-30 08:12:19,200::task::974::TaskManager.Task::(_decref) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::ref 1 aborting True
>Thread-30103::INFO::2014-04-30 08:12:19,201::task::1151::TaskManager.Task::(prepare) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30103::DEBUG::2014-04-30 08:12:19,201::task::1156::TaskManager.Task::(prepare) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30103::DEBUG::2014-04-30 08:12:19,201::task::974::TaskManager.Task::(_decref) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::ref 0 aborting True
>Thread-30103::DEBUG::2014-04-30 08:12:19,201::task::909::TaskManager.Task::(_doAbort) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::Task._doAbort: force False
>Thread-30103::DEBUG::2014-04-30 08:12:19,202::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30103::DEBUG::2014-04-30 08:12:19,202::task::579::TaskManager.Task::(_updateState) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::moving from state preparing -> state aborting
>Thread-30103::DEBUG::2014-04-30 08:12:19,202::task::534::TaskManager.Task::(__state_aborting) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::_aborting: recover policy none
>Thread-30103::DEBUG::2014-04-30 08:12:19,202::task::579::TaskManager.Task::(_updateState) Task=`68281660-716c-4ea4-bedf-038c1e8ded06`::moving from state aborting -> state failed
>Thread-30103::DEBUG::2014-04-30 08:12:19,203::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30103::DEBUG::2014-04-30 08:12:19,203::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30103::ERROR::2014-04-30 08:12:19,203::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30102::ERROR::2014-04-30 08:13:29,083::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30102::ERROR::2014-04-30 08:13:29,084::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30102::DEBUG::2014-04-30 08:13:29,085::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30102::DEBUG::2014-04-30 08:13:29,089::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30102::DEBUG::2014-04-30 08:13:29,090::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30102::DEBUG::2014-04-30 08:13:29,090::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30102::INFO::2014-04-30 08:13:29,090::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30102::DEBUG::2014-04-30 08:13:29,091::task::1168::TaskManager.Task::(prepare) Task=`d002ac97-9716-49f5-a864-f6d3a0129de0`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30102::DEBUG::2014-04-30 08:13:29,091::task::579::TaskManager.Task::(_updateState) Task=`d002ac97-9716-49f5-a864-f6d3a0129de0`::moving from state preparing -> state finished
>Thread-30102::DEBUG::2014-04-30 08:13:29,091::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30102::DEBUG::2014-04-30 08:13:29,091::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30102::DEBUG::2014-04-30 08:13:29,092::task::974::TaskManager.Task::(_decref) Task=`d002ac97-9716-49f5-a864-f6d3a0129de0`::ref 0 aborting False
>Thread-30104::DEBUG::2014-04-30 08:14:17,436::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5534c01c]
>Thread-30104::DEBUG::2014-04-30 08:14:17,437::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30104::DEBUG::2014-04-30 08:14:17,451::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30104::DEBUG::2014-04-30 08:14:17,488::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30105::DEBUG::2014-04-30 08:14:17,713::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5534c01c]
>Thread-30105::ERROR::2014-04-30 08:14:17,714::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30105::DEBUG::2014-04-30 08:14:17,714::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30106::DEBUG::2014-04-30 08:14:19,540::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [61de2f6e]
>Thread-30106::DEBUG::2014-04-30 08:14:19,540::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30106::DEBUG::2014-04-30 08:14:19,554::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30106::DEBUG::2014-04-30 08:14:19,591::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30107::DEBUG::2014-04-30 08:14:19,817::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [61de2f6e]
>Thread-30107::ERROR::2014-04-30 08:14:19,817::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30107::DEBUG::2014-04-30 08:14:19,818::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30109::DEBUG::2014-04-30 08:14:20,281::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [5d67cb4d]
>Thread-30109::DEBUG::2014-04-30 08:14:20,282::task::579::TaskManager.Task::(_updateState) Task=`91502332-f53d-4c48-8e94-5266457ae68c`::moving from state init -> state preparing
>Thread-30109::INFO::2014-04-30 08:14:20,282::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30109::DEBUG::2014-04-30 08:14:20,287::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30109::ERROR::2014-04-30 08:16:25,329::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30109::ERROR::2014-04-30 08:16:25,330::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30109::DEBUG::2014-04-30 08:16:25,330::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30109::DEBUG::2014-04-30 08:16:25,335::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30110::DEBUG::2014-04-30 08:17:20,469::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30110::DEBUG::2014-04-30 08:17:20,470::task::579::TaskManager.Task::(_updateState) Task=`20605b96-32e8-4f67-b477-913f29987e51`::moving from state init -> state preparing
>Thread-30110::INFO::2014-04-30 08:17:20,471::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30110::DEBUG::2014-04-30 08:17:20,471::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7aed5329-65a1-4faf-a9c6-ee1b453f6950`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30110::DEBUG::2014-04-30 08:17:20,471::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30110::DEBUG::2014-04-30 08:17:20,472::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30110::DEBUG::2014-04-30 08:17:20,472::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7aed5329-65a1-4faf-a9c6-ee1b453f6950`::Granted request
>Thread-30110::DEBUG::2014-04-30 08:17:20,472::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`20e815ee-ca99-45ce-927a-1247ec060db4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30110::DEBUG::2014-04-30 08:17:20,473::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30110::DEBUG::2014-04-30 08:17:20,473::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30110::DEBUG::2014-04-30 08:17:20,473::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`20e815ee-ca99-45ce-927a-1247ec060db4`::Granted request
>Thread-30110::INFO::2014-04-30 08:17:20,473::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30110::DEBUG::2014-04-30 08:17:20,474::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30110::DEBUG::2014-04-30 08:17:20,474::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30110::DEBUG::2014-04-30 08:17:20,474::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30110::DEBUG::2014-04-30 08:17:20,475::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30110::DEBUG::2014-04-30 08:17:20,475::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30110::DEBUG::2014-04-30 08:17:20,475::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30110::DEBUG::2014-04-30 08:17:20,475::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30110::DEBUG::2014-04-30 08:17:20,475::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30110::DEBUG::2014-04-30 08:17:20,476::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30110::DEBUG::2014-04-30 08:17:20,476::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30110::DEBUG::2014-04-30 08:17:20,476::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30110::DEBUG::2014-04-30 08:17:20,498::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30110::DEBUG::2014-04-30 08:17:20,499::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30110::DEBUG::2014-04-30 08:17:20,500::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30110::DEBUG::2014-04-30 08:17:20,500::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30110::DEBUG::2014-04-30 08:17:20,500::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30110::DEBUG::2014-04-30 08:17:20,500::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30110::DEBUG::2014-04-30 08:17:20,501::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30110::DEBUG::2014-04-30 08:17:20,501::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30110::DEBUG::2014-04-30 08:17:20,501::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30110::DEBUG::2014-04-30 08:17:20,502::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30110::ERROR::2014-04-30 08:17:20,502::task::850::TaskManager.Task::(_setError) Task=`20605b96-32e8-4f67-b477-913f29987e51`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30110::DEBUG::2014-04-30 08:17:20,504::task::869::TaskManager.Task::(_run) Task=`20605b96-32e8-4f67-b477-913f29987e51`::Task._run: 20605b96-32e8-4f67-b477-913f29987e51 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30110::DEBUG::2014-04-30 08:17:20,504::task::1194::TaskManager.Task::(stop) Task=`20605b96-32e8-4f67-b477-913f29987e51`::stopping in state preparing (force False)
>Thread-30110::DEBUG::2014-04-30 08:17:20,504::task::974::TaskManager.Task::(_decref) Task=`20605b96-32e8-4f67-b477-913f29987e51`::ref 1 aborting True
>Thread-30110::INFO::2014-04-30 08:17:20,505::task::1151::TaskManager.Task::(prepare) Task=`20605b96-32e8-4f67-b477-913f29987e51`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30110::DEBUG::2014-04-30 08:17:20,505::task::1156::TaskManager.Task::(prepare) Task=`20605b96-32e8-4f67-b477-913f29987e51`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30110::DEBUG::2014-04-30 08:17:20,505::task::974::TaskManager.Task::(_decref) Task=`20605b96-32e8-4f67-b477-913f29987e51`::ref 0 aborting True
>Thread-30110::DEBUG::2014-04-30 08:17:20,505::task::909::TaskManager.Task::(_doAbort) Task=`20605b96-32e8-4f67-b477-913f29987e51`::Task._doAbort: force False
>Thread-30110::DEBUG::2014-04-30 08:17:20,506::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30110::DEBUG::2014-04-30 08:17:20,506::task::579::TaskManager.Task::(_updateState) Task=`20605b96-32e8-4f67-b477-913f29987e51`::moving from state preparing -> state aborting
>Thread-30110::DEBUG::2014-04-30 08:17:20,506::task::534::TaskManager.Task::(__state_aborting) Task=`20605b96-32e8-4f67-b477-913f29987e51`::_aborting: recover policy none
>Thread-30110::DEBUG::2014-04-30 08:17:20,506::task::579::TaskManager.Task::(_updateState) Task=`20605b96-32e8-4f67-b477-913f29987e51`::moving from state aborting -> state failed
>Thread-30110::DEBUG::2014-04-30 08:17:20,506::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30110::DEBUG::2014-04-30 08:17:20,507::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30110::ERROR::2014-04-30 08:17:20,507::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30109::ERROR::2014-04-30 08:18:30,386::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30109::ERROR::2014-04-30 08:18:30,387::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30109::DEBUG::2014-04-30 08:18:30,387::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30109::DEBUG::2014-04-30 08:18:30,393::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30109::DEBUG::2014-04-30 08:18:30,394::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30109::DEBUG::2014-04-30 08:18:30,394::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30109::INFO::2014-04-30 08:18:30,394::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30109::DEBUG::2014-04-30 08:18:30,395::task::1168::TaskManager.Task::(prepare) Task=`91502332-f53d-4c48-8e94-5266457ae68c`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30109::DEBUG::2014-04-30 08:18:30,395::task::579::TaskManager.Task::(_updateState) Task=`91502332-f53d-4c48-8e94-5266457ae68c`::moving from state preparing -> state finished
>Thread-30109::DEBUG::2014-04-30 08:18:30,395::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30109::DEBUG::2014-04-30 08:18:30,395::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30109::DEBUG::2014-04-30 08:18:30,396::task::974::TaskManager.Task::(_decref) Task=`91502332-f53d-4c48-8e94-5266457ae68c`::ref 0 aborting False
>Thread-30111::DEBUG::2014-04-30 08:19:17,433::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [705cb3dd]
>Thread-30111::DEBUG::2014-04-30 08:19:17,435::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30111::DEBUG::2014-04-30 08:19:17,449::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30111::DEBUG::2014-04-30 08:19:17,487::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30112::DEBUG::2014-04-30 08:19:17,711::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [705cb3dd]
>Thread-30112::ERROR::2014-04-30 08:19:17,712::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30112::DEBUG::2014-04-30 08:19:17,713::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30113::DEBUG::2014-04-30 08:19:20,842::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [45ddd7f3]
>Thread-30113::DEBUG::2014-04-30 08:19:20,843::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30113::DEBUG::2014-04-30 08:19:20,856::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30113::DEBUG::2014-04-30 08:19:20,893::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30114::DEBUG::2014-04-30 08:19:21,118::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [45ddd7f3]
>Thread-30114::ERROR::2014-04-30 08:19:21,118::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30114::DEBUG::2014-04-30 08:19:21,119::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30116::DEBUG::2014-04-30 08:19:21,581::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [b950413]
>Thread-30116::DEBUG::2014-04-30 08:19:21,582::task::579::TaskManager.Task::(_updateState) Task=`7b41acb2-1c2c-4386-9eb8-9f8cec0de1d6`::moving from state init -> state preparing
>Thread-30116::INFO::2014-04-30 08:19:21,583::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30116::DEBUG::2014-04-30 08:19:21,586::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30116::ERROR::2014-04-30 08:21:26,628::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30116::ERROR::2014-04-30 08:21:26,629::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30116::DEBUG::2014-04-30 08:21:26,630::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30116::DEBUG::2014-04-30 08:21:26,634::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30117::DEBUG::2014-04-30 08:22:21,770::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30117::DEBUG::2014-04-30 08:22:21,771::task::579::TaskManager.Task::(_updateState) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::moving from state init -> state preparing
>Thread-30117::INFO::2014-04-30 08:22:21,771::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30117::DEBUG::2014-04-30 08:22:21,771::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a22a57de-1b26-409f-9b9d-9e851062a602`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30117::DEBUG::2014-04-30 08:22:21,772::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30117::DEBUG::2014-04-30 08:22:21,772::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30117::DEBUG::2014-04-30 08:22:21,772::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a22a57de-1b26-409f-9b9d-9e851062a602`::Granted request
>Thread-30117::DEBUG::2014-04-30 08:22:21,773::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`ed2ab776-1136-4ace-8f77-82e631580b1e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30117::DEBUG::2014-04-30 08:22:21,773::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30117::DEBUG::2014-04-30 08:22:21,773::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30117::DEBUG::2014-04-30 08:22:21,773::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`ed2ab776-1136-4ace-8f77-82e631580b1e`::Granted request
>Thread-30117::INFO::2014-04-30 08:22:21,774::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30117::DEBUG::2014-04-30 08:22:21,774::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30117::DEBUG::2014-04-30 08:22:21,775::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30117::DEBUG::2014-04-30 08:22:21,775::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30117::DEBUG::2014-04-30 08:22:21,775::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30117::DEBUG::2014-04-30 08:22:21,775::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30117::DEBUG::2014-04-30 08:22:21,775::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30117::DEBUG::2014-04-30 08:22:21,776::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30117::DEBUG::2014-04-30 08:22:21,776::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30117::DEBUG::2014-04-30 08:22:21,776::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30117::DEBUG::2014-04-30 08:22:21,776::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30117::DEBUG::2014-04-30 08:22:21,776::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30117::DEBUG::2014-04-30 08:22:21,798::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30117::DEBUG::2014-04-30 08:22:21,799::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30117::DEBUG::2014-04-30 08:22:21,800::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30117::DEBUG::2014-04-30 08:22:21,800::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30117::DEBUG::2014-04-30 08:22:21,800::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30117::DEBUG::2014-04-30 08:22:21,801::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30117::DEBUG::2014-04-30 08:22:21,801::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30117::DEBUG::2014-04-30 08:22:21,801::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30117::DEBUG::2014-04-30 08:22:21,802::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30117::DEBUG::2014-04-30 08:22:21,802::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30117::ERROR::2014-04-30 08:22:21,802::task::850::TaskManager.Task::(_setError) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30117::DEBUG::2014-04-30 08:22:21,803::task::869::TaskManager.Task::(_run) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::Task._run: 1168b683-cc5f-43eb-8b95-98c4aa812701 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30117::DEBUG::2014-04-30 08:22:21,803::task::1194::TaskManager.Task::(stop) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::stopping in state preparing (force False)
>Thread-30117::DEBUG::2014-04-30 08:22:21,804::task::974::TaskManager.Task::(_decref) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::ref 1 aborting True
>Thread-30117::INFO::2014-04-30 08:22:21,804::task::1151::TaskManager.Task::(prepare) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30117::DEBUG::2014-04-30 08:22:21,804::task::1156::TaskManager.Task::(prepare) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30117::DEBUG::2014-04-30 08:22:21,804::task::974::TaskManager.Task::(_decref) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::ref 0 aborting True
>Thread-30117::DEBUG::2014-04-30 08:22:21,805::task::909::TaskManager.Task::(_doAbort) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::Task._doAbort: force False
>Thread-30117::DEBUG::2014-04-30 08:22:21,805::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30117::DEBUG::2014-04-30 08:22:21,805::task::579::TaskManager.Task::(_updateState) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::moving from state preparing -> state aborting
>Thread-30117::DEBUG::2014-04-30 08:22:21,805::task::534::TaskManager.Task::(__state_aborting) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::_aborting: recover policy none
>Thread-30117::DEBUG::2014-04-30 08:22:21,806::task::579::TaskManager.Task::(_updateState) Task=`1168b683-cc5f-43eb-8b95-98c4aa812701`::moving from state aborting -> state failed
>Thread-30117::DEBUG::2014-04-30 08:22:21,806::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30117::DEBUG::2014-04-30 08:22:21,806::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30117::ERROR::2014-04-30 08:22:21,806::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30116::ERROR::2014-04-30 08:23:31,677::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30116::ERROR::2014-04-30 08:23:31,679::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30116::DEBUG::2014-04-30 08:23:31,679::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30116::DEBUG::2014-04-30 08:23:31,683::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30116::DEBUG::2014-04-30 08:23:31,685::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30116::DEBUG::2014-04-30 08:23:31,686::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30116::INFO::2014-04-30 08:23:31,686::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30116::DEBUG::2014-04-30 08:23:31,686::task::1168::TaskManager.Task::(prepare) Task=`7b41acb2-1c2c-4386-9eb8-9f8cec0de1d6`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30116::DEBUG::2014-04-30 08:23:31,686::task::579::TaskManager.Task::(_updateState) Task=`7b41acb2-1c2c-4386-9eb8-9f8cec0de1d6`::moving from state preparing -> state finished
>Thread-30116::DEBUG::2014-04-30 08:23:31,687::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30116::DEBUG::2014-04-30 08:23:31,687::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30116::DEBUG::2014-04-30 08:23:31,687::task::974::TaskManager.Task::(_decref) Task=`7b41acb2-1c2c-4386-9eb8-9f8cec0de1d6`::ref 0 aborting False
>Thread-30118::DEBUG::2014-04-30 08:24:17,421::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [58393dfa]
>Thread-30118::DEBUG::2014-04-30 08:24:17,423::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30118::DEBUG::2014-04-30 08:24:17,436::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30118::DEBUG::2014-04-30 08:24:17,473::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30119::DEBUG::2014-04-30 08:24:17,699::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [58393dfa]
>Thread-30119::ERROR::2014-04-30 08:24:17,699::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30119::DEBUG::2014-04-30 08:24:17,700::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30120::DEBUG::2014-04-30 08:24:19,143::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [7ff065d4]
>Thread-30120::DEBUG::2014-04-30 08:24:19,144::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30120::DEBUG::2014-04-30 08:24:19,157::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30120::DEBUG::2014-04-30 08:24:19,194::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30121::DEBUG::2014-04-30 08:24:19,420::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [7ff065d4]
>Thread-30121::ERROR::2014-04-30 08:24:19,421::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30121::DEBUG::2014-04-30 08:24:19,421::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30123::DEBUG::2014-04-30 08:24:19,888::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [7becd6c2]
>Thread-30123::DEBUG::2014-04-30 08:24:19,889::task::579::TaskManager.Task::(_updateState) Task=`255a0ef2-050e-43eb-88bb-934ea6fc23c9`::moving from state init -> state preparing
>Thread-30123::INFO::2014-04-30 08:24:19,889::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30123::DEBUG::2014-04-30 08:24:19,895::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30123::ERROR::2014-04-30 08:26:24,937::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30123::ERROR::2014-04-30 08:26:24,946::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30123::DEBUG::2014-04-30 08:26:24,946::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30123::DEBUG::2014-04-30 08:26:24,951::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30124::DEBUG::2014-04-30 08:27:20,064::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30124::DEBUG::2014-04-30 08:27:20,065::task::579::TaskManager.Task::(_updateState) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::moving from state init -> state preparing
>Thread-30124::INFO::2014-04-30 08:27:20,065::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30124::DEBUG::2014-04-30 08:27:20,066::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`0bd00f0b-bcce-464d-bef8-e9872dd7388c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30124::DEBUG::2014-04-30 08:27:20,066::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30124::DEBUG::2014-04-30 08:27:20,066::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30124::DEBUG::2014-04-30 08:27:20,066::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`0bd00f0b-bcce-464d-bef8-e9872dd7388c`::Granted request
>Thread-30124::DEBUG::2014-04-30 08:27:20,067::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`410a3385-15f1-45ea-aa49-b10ac51e35d6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30124::DEBUG::2014-04-30 08:27:20,067::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30124::DEBUG::2014-04-30 08:27:20,067::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30124::DEBUG::2014-04-30 08:27:20,068::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`410a3385-15f1-45ea-aa49-b10ac51e35d6`::Granted request
>Thread-30124::INFO::2014-04-30 08:27:20,068::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30124::DEBUG::2014-04-30 08:27:20,069::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30124::DEBUG::2014-04-30 08:27:20,069::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30124::DEBUG::2014-04-30 08:27:20,069::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30124::DEBUG::2014-04-30 08:27:20,069::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30124::DEBUG::2014-04-30 08:27:20,070::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30124::DEBUG::2014-04-30 08:27:20,070::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30124::DEBUG::2014-04-30 08:27:20,070::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30124::DEBUG::2014-04-30 08:27:20,070::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30124::DEBUG::2014-04-30 08:27:20,070::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30124::DEBUG::2014-04-30 08:27:20,070::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30124::DEBUG::2014-04-30 08:27:20,071::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30124::DEBUG::2014-04-30 08:27:20,098::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30124::DEBUG::2014-04-30 08:27:20,099::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30124::DEBUG::2014-04-30 08:27:20,100::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30124::DEBUG::2014-04-30 08:27:20,100::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30124::DEBUG::2014-04-30 08:27:20,100::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30124::DEBUG::2014-04-30 08:27:20,101::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30124::DEBUG::2014-04-30 08:27:20,101::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30124::DEBUG::2014-04-30 08:27:20,101::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30124::DEBUG::2014-04-30 08:27:20,101::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30124::DEBUG::2014-04-30 08:27:20,102::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30124::ERROR::2014-04-30 08:27:20,102::task::850::TaskManager.Task::(_setError) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30124::DEBUG::2014-04-30 08:27:20,103::task::869::TaskManager.Task::(_run) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::Task._run: c35e35e2-0b04-4c6b-9412-9c4f5ebf2077 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30124::DEBUG::2014-04-30 08:27:20,103::task::1194::TaskManager.Task::(stop) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::stopping in state preparing (force False)
>Thread-30124::DEBUG::2014-04-30 08:27:20,104::task::974::TaskManager.Task::(_decref) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::ref 1 aborting True
>Thread-30124::INFO::2014-04-30 08:27:20,104::task::1151::TaskManager.Task::(prepare) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30124::DEBUG::2014-04-30 08:27:20,104::task::1156::TaskManager.Task::(prepare) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30124::DEBUG::2014-04-30 08:27:20,104::task::974::TaskManager.Task::(_decref) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::ref 0 aborting True
>Thread-30124::DEBUG::2014-04-30 08:27:20,105::task::909::TaskManager.Task::(_doAbort) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::Task._doAbort: force False
>Thread-30124::DEBUG::2014-04-30 08:27:20,105::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30124::DEBUG::2014-04-30 08:27:20,105::task::579::TaskManager.Task::(_updateState) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::moving from state preparing -> state aborting
>Thread-30124::DEBUG::2014-04-30 08:27:20,105::task::534::TaskManager.Task::(__state_aborting) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::_aborting: recover policy none
>Thread-30124::DEBUG::2014-04-30 08:27:20,106::task::579::TaskManager.Task::(_updateState) Task=`c35e35e2-0b04-4c6b-9412-9c4f5ebf2077`::moving from state aborting -> state failed
>Thread-30124::DEBUG::2014-04-30 08:27:20,106::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30124::DEBUG::2014-04-30 08:27:20,106::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30124::ERROR::2014-04-30 08:27:20,106::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30123::ERROR::2014-04-30 08:28:29,993::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30123::ERROR::2014-04-30 08:28:29,995::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30123::DEBUG::2014-04-30 08:28:29,995::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30123::DEBUG::2014-04-30 08:28:29,999::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30123::DEBUG::2014-04-30 08:28:30,000::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30123::DEBUG::2014-04-30 08:28:30,001::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30123::INFO::2014-04-30 08:28:30,001::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30123::DEBUG::2014-04-30 08:28:30,001::task::1168::TaskManager.Task::(prepare) Task=`255a0ef2-050e-43eb-88bb-934ea6fc23c9`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30123::DEBUG::2014-04-30 08:28:30,001::task::579::TaskManager.Task::(_updateState) Task=`255a0ef2-050e-43eb-88bb-934ea6fc23c9`::moving from state preparing -> state finished
>Thread-30123::DEBUG::2014-04-30 08:28:30,002::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30123::DEBUG::2014-04-30 08:28:30,002::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30123::DEBUG::2014-04-30 08:28:30,002::task::974::TaskManager.Task::(_decref) Task=`255a0ef2-050e-43eb-88bb-934ea6fc23c9`::ref 0 aborting False
>Thread-30125::DEBUG::2014-04-30 08:29:17,429::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [10c7d1d0]
>Thread-30125::DEBUG::2014-04-30 08:29:17,430::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30125::DEBUG::2014-04-30 08:29:17,444::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30125::DEBUG::2014-04-30 08:29:17,482::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30126::DEBUG::2014-04-30 08:29:17,707::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [10c7d1d0]
>Thread-30126::ERROR::2014-04-30 08:29:17,707::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30126::DEBUG::2014-04-30 08:29:17,708::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30127::DEBUG::2014-04-30 08:29:20,507::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [42b0efd0]
>Thread-30127::DEBUG::2014-04-30 08:29:20,508::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30127::DEBUG::2014-04-30 08:29:20,522::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30127::DEBUG::2014-04-30 08:29:20,559::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30128::DEBUG::2014-04-30 08:29:20,783::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [42b0efd0]
>Thread-30128::ERROR::2014-04-30 08:29:20,784::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30128::DEBUG::2014-04-30 08:29:20,785::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30130::DEBUG::2014-04-30 08:29:21,246::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [28c2c865]
>Thread-30130::DEBUG::2014-04-30 08:29:21,247::task::579::TaskManager.Task::(_updateState) Task=`406c1a7b-4173-4448-a5e2-2e3c6c338bc7`::moving from state init -> state preparing
>Thread-30130::INFO::2014-04-30 08:29:21,247::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30130::DEBUG::2014-04-30 08:29:21,252::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30130::ERROR::2014-04-30 08:31:26,294::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30130::ERROR::2014-04-30 08:31:26,295::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30130::DEBUG::2014-04-30 08:31:26,296::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30130::DEBUG::2014-04-30 08:31:26,301::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30131::DEBUG::2014-04-30 08:32:21,425::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30131::DEBUG::2014-04-30 08:32:21,426::task::579::TaskManager.Task::(_updateState) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::moving from state init -> state preparing
>Thread-30131::INFO::2014-04-30 08:32:21,427::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30131::DEBUG::2014-04-30 08:32:21,427::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7a7ddf20-df9b-423b-8c23-e813a6d6e2af`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30131::DEBUG::2014-04-30 08:32:21,427::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30131::DEBUG::2014-04-30 08:32:21,428::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30131::DEBUG::2014-04-30 08:32:21,428::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7a7ddf20-df9b-423b-8c23-e813a6d6e2af`::Granted request
>Thread-30131::DEBUG::2014-04-30 08:32:21,429::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`15ab4e81-e4f4-4ab3-989c-df83e1e47491`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30131::DEBUG::2014-04-30 08:32:21,429::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30131::DEBUG::2014-04-30 08:32:21,429::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30131::DEBUG::2014-04-30 08:32:21,430::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`15ab4e81-e4f4-4ab3-989c-df83e1e47491`::Granted request
>Thread-30131::INFO::2014-04-30 08:32:21,430::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30131::DEBUG::2014-04-30 08:32:21,431::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30131::DEBUG::2014-04-30 08:32:21,431::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30131::DEBUG::2014-04-30 08:32:21,431::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30131::DEBUG::2014-04-30 08:32:21,432::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30131::DEBUG::2014-04-30 08:32:21,432::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30131::DEBUG::2014-04-30 08:32:21,432::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30131::DEBUG::2014-04-30 08:32:21,433::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30131::DEBUG::2014-04-30 08:32:21,433::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30131::DEBUG::2014-04-30 08:32:21,433::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30131::DEBUG::2014-04-30 08:32:21,433::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30131::DEBUG::2014-04-30 08:32:21,434::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30131::DEBUG::2014-04-30 08:32:21,456::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30131::DEBUG::2014-04-30 08:32:21,456::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30131::DEBUG::2014-04-30 08:32:21,457::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30131::DEBUG::2014-04-30 08:32:21,457::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30131::DEBUG::2014-04-30 08:32:21,458::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30131::DEBUG::2014-04-30 08:32:21,458::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30131::DEBUG::2014-04-30 08:32:21,458::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30131::DEBUG::2014-04-30 08:32:21,459::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30131::DEBUG::2014-04-30 08:32:21,459::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30131::DEBUG::2014-04-30 08:32:21,459::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30131::ERROR::2014-04-30 08:32:21,459::task::850::TaskManager.Task::(_setError) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30131::DEBUG::2014-04-30 08:32:21,460::task::869::TaskManager.Task::(_run) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::Task._run: 783b0ce2-76ea-466c-8a84-3df3e44add6e ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30131::DEBUG::2014-04-30 08:32:21,461::task::1194::TaskManager.Task::(stop) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::stopping in state preparing (force False)
>Thread-30131::DEBUG::2014-04-30 08:32:21,461::task::974::TaskManager.Task::(_decref) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::ref 1 aborting True
>Thread-30131::INFO::2014-04-30 08:32:21,461::task::1151::TaskManager.Task::(prepare) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30131::DEBUG::2014-04-30 08:32:21,461::task::1156::TaskManager.Task::(prepare) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30131::DEBUG::2014-04-30 08:32:21,462::task::974::TaskManager.Task::(_decref) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::ref 0 aborting True
>Thread-30131::DEBUG::2014-04-30 08:32:21,462::task::909::TaskManager.Task::(_doAbort) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::Task._doAbort: force False
>Thread-30131::DEBUG::2014-04-30 08:32:21,462::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30131::DEBUG::2014-04-30 08:32:21,462::task::579::TaskManager.Task::(_updateState) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::moving from state preparing -> state aborting
>Thread-30131::DEBUG::2014-04-30 08:32:21,463::task::534::TaskManager.Task::(__state_aborting) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::_aborting: recover policy none
>Thread-30131::DEBUG::2014-04-30 08:32:21,463::task::579::TaskManager.Task::(_updateState) Task=`783b0ce2-76ea-466c-8a84-3df3e44add6e`::moving from state aborting -> state failed
>Thread-30131::DEBUG::2014-04-30 08:32:21,463::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30131::DEBUG::2014-04-30 08:32:21,463::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30131::ERROR::2014-04-30 08:32:21,463::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30130::ERROR::2014-04-30 08:33:31,342::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30130::ERROR::2014-04-30 08:33:31,343::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30130::DEBUG::2014-04-30 08:33:31,344::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30130::DEBUG::2014-04-30 08:33:31,348::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30130::DEBUG::2014-04-30 08:33:31,349::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30130::DEBUG::2014-04-30 08:33:31,349::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30130::INFO::2014-04-30 08:33:31,349::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30130::DEBUG::2014-04-30 08:33:31,350::task::1168::TaskManager.Task::(prepare) Task=`406c1a7b-4173-4448-a5e2-2e3c6c338bc7`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30130::DEBUG::2014-04-30 08:33:31,350::task::579::TaskManager.Task::(_updateState) Task=`406c1a7b-4173-4448-a5e2-2e3c6c338bc7`::moving from state preparing -> state finished
>Thread-30130::DEBUG::2014-04-30 08:33:31,350::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30130::DEBUG::2014-04-30 08:33:31,350::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30130::DEBUG::2014-04-30 08:33:31,351::task::974::TaskManager.Task::(_decref) Task=`406c1a7b-4173-4448-a5e2-2e3c6c338bc7`::ref 0 aborting False
>Thread-30132::DEBUG::2014-04-30 08:34:17,406::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [6b85123e]
>Thread-30132::DEBUG::2014-04-30 08:34:17,407::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30132::DEBUG::2014-04-30 08:34:17,420::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30132::DEBUG::2014-04-30 08:34:17,458::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30133::DEBUG::2014-04-30 08:34:17,682::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [6b85123e]
>Thread-30133::ERROR::2014-04-30 08:34:17,683::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30133::DEBUG::2014-04-30 08:34:17,683::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30134::DEBUG::2014-04-30 08:34:18,800::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5e1db2c8]
>Thread-30134::DEBUG::2014-04-30 08:34:18,801::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30134::DEBUG::2014-04-30 08:34:18,815::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30134::DEBUG::2014-04-30 08:34:18,852::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30135::DEBUG::2014-04-30 08:34:19,077::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5e1db2c8]
>Thread-30135::ERROR::2014-04-30 08:34:19,077::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30135::DEBUG::2014-04-30 08:34:19,078::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30137::DEBUG::2014-04-30 08:34:19,540::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [77928d3d]
>Thread-30137::DEBUG::2014-04-30 08:34:19,541::task::579::TaskManager.Task::(_updateState) Task=`4d1dec77-3dd6-4e23-b50a-86163c31e640`::moving from state init -> state preparing
>Thread-30137::INFO::2014-04-30 08:34:19,541::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30137::DEBUG::2014-04-30 08:34:19,545::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30137::ERROR::2014-04-30 08:36:24,585::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30137::ERROR::2014-04-30 08:36:24,586::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30137::DEBUG::2014-04-30 08:36:24,587::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30137::DEBUG::2014-04-30 08:36:24,592::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30138::DEBUG::2014-04-30 08:37:19,717::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30138::DEBUG::2014-04-30 08:37:19,718::task::579::TaskManager.Task::(_updateState) Task=`440945de-c492-45e7-9df1-d811795ae75c`::moving from state init -> state preparing
>Thread-30138::INFO::2014-04-30 08:37:19,718::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30138::DEBUG::2014-04-30 08:37:19,718::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7b180f77-894f-46d7-b16e-42d5853b6b5f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30138::DEBUG::2014-04-30 08:37:19,719::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30138::DEBUG::2014-04-30 08:37:19,719::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30138::DEBUG::2014-04-30 08:37:19,719::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7b180f77-894f-46d7-b16e-42d5853b6b5f`::Granted request
>Thread-30138::DEBUG::2014-04-30 08:37:19,720::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`f3dbc744-495b-437b-bc59-00ce926ef372`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30138::DEBUG::2014-04-30 08:37:19,720::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30138::DEBUG::2014-04-30 08:37:19,720::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30138::DEBUG::2014-04-30 08:37:19,720::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`f3dbc744-495b-437b-bc59-00ce926ef372`::Granted request
>Thread-30138::INFO::2014-04-30 08:37:19,721::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30138::DEBUG::2014-04-30 08:37:19,721::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30138::DEBUG::2014-04-30 08:37:19,722::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30138::DEBUG::2014-04-30 08:37:19,722::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30138::DEBUG::2014-04-30 08:37:19,722::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30138::DEBUG::2014-04-30 08:37:19,722::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30138::DEBUG::2014-04-30 08:37:19,722::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30138::DEBUG::2014-04-30 08:37:19,723::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30138::DEBUG::2014-04-30 08:37:19,723::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30138::DEBUG::2014-04-30 08:37:19,723::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30138::DEBUG::2014-04-30 08:37:19,723::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30138::DEBUG::2014-04-30 08:37:19,723::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30138::DEBUG::2014-04-30 08:37:19,745::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30138::DEBUG::2014-04-30 08:37:19,746::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30138::DEBUG::2014-04-30 08:37:19,747::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30138::DEBUG::2014-04-30 08:37:19,747::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30138::DEBUG::2014-04-30 08:37:19,748::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30138::DEBUG::2014-04-30 08:37:19,748::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30138::DEBUG::2014-04-30 08:37:19,748::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30138::DEBUG::2014-04-30 08:37:19,748::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30138::DEBUG::2014-04-30 08:37:19,749::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30138::DEBUG::2014-04-30 08:37:19,749::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30138::ERROR::2014-04-30 08:37:19,749::task::850::TaskManager.Task::(_setError) Task=`440945de-c492-45e7-9df1-d811795ae75c`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30138::DEBUG::2014-04-30 08:37:19,750::task::869::TaskManager.Task::(_run) Task=`440945de-c492-45e7-9df1-d811795ae75c`::Task._run: 440945de-c492-45e7-9df1-d811795ae75c ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30138::DEBUG::2014-04-30 08:37:19,751::task::1194::TaskManager.Task::(stop) Task=`440945de-c492-45e7-9df1-d811795ae75c`::stopping in state preparing (force False)
>Thread-30138::DEBUG::2014-04-30 08:37:19,751::task::974::TaskManager.Task::(_decref) Task=`440945de-c492-45e7-9df1-d811795ae75c`::ref 1 aborting True
>Thread-30138::INFO::2014-04-30 08:37:19,751::task::1151::TaskManager.Task::(prepare) Task=`440945de-c492-45e7-9df1-d811795ae75c`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30138::DEBUG::2014-04-30 08:37:19,751::task::1156::TaskManager.Task::(prepare) Task=`440945de-c492-45e7-9df1-d811795ae75c`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30138::DEBUG::2014-04-30 08:37:19,752::task::974::TaskManager.Task::(_decref) Task=`440945de-c492-45e7-9df1-d811795ae75c`::ref 0 aborting True
>Thread-30138::DEBUG::2014-04-30 08:37:19,752::task::909::TaskManager.Task::(_doAbort) Task=`440945de-c492-45e7-9df1-d811795ae75c`::Task._doAbort: force False
>Thread-30138::DEBUG::2014-04-30 08:37:19,752::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30138::DEBUG::2014-04-30 08:37:19,752::task::579::TaskManager.Task::(_updateState) Task=`440945de-c492-45e7-9df1-d811795ae75c`::moving from state preparing -> state aborting
>Thread-30138::DEBUG::2014-04-30 08:37:19,752::task::534::TaskManager.Task::(__state_aborting) Task=`440945de-c492-45e7-9df1-d811795ae75c`::_aborting: recover policy none
>Thread-30138::DEBUG::2014-04-30 08:37:19,753::task::579::TaskManager.Task::(_updateState) Task=`440945de-c492-45e7-9df1-d811795ae75c`::moving from state aborting -> state failed
>Thread-30138::DEBUG::2014-04-30 08:37:19,753::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30138::DEBUG::2014-04-30 08:37:19,753::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30138::ERROR::2014-04-30 08:37:19,753::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30137::ERROR::2014-04-30 08:38:29,632::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30137::ERROR::2014-04-30 08:38:29,633::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30137::DEBUG::2014-04-30 08:38:29,633::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30137::DEBUG::2014-04-30 08:38:29,637::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30137::DEBUG::2014-04-30 08:38:29,638::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30137::DEBUG::2014-04-30 08:38:29,639::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30137::INFO::2014-04-30 08:38:29,639::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30137::DEBUG::2014-04-30 08:38:29,639::task::1168::TaskManager.Task::(prepare) Task=`4d1dec77-3dd6-4e23-b50a-86163c31e640`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30137::DEBUG::2014-04-30 08:38:29,639::task::579::TaskManager.Task::(_updateState) Task=`4d1dec77-3dd6-4e23-b50a-86163c31e640`::moving from state preparing -> state finished
>Thread-30137::DEBUG::2014-04-30 08:38:29,640::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30137::DEBUG::2014-04-30 08:38:29,640::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30137::DEBUG::2014-04-30 08:38:29,640::task::974::TaskManager.Task::(_decref) Task=`4d1dec77-3dd6-4e23-b50a-86163c31e640`::ref 0 aborting False
>Thread-30139::DEBUG::2014-04-30 08:39:17,406::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [751da8b2]
>Thread-30139::DEBUG::2014-04-30 08:39:17,407::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30139::DEBUG::2014-04-30 08:39:17,420::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30139::DEBUG::2014-04-30 08:39:17,459::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30140::DEBUG::2014-04-30 08:39:17,684::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [751da8b2]
>Thread-30140::ERROR::2014-04-30 08:39:17,685::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30140::DEBUG::2014-04-30 08:39:17,685::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30141::DEBUG::2014-04-30 08:39:20,098::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5b883a5a]
>Thread-30141::DEBUG::2014-04-30 08:39:20,099::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30141::DEBUG::2014-04-30 08:39:20,112::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30141::DEBUG::2014-04-30 08:39:20,149::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30142::DEBUG::2014-04-30 08:39:20,375::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5b883a5a]
>Thread-30142::ERROR::2014-04-30 08:39:20,375::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30142::DEBUG::2014-04-30 08:39:20,376::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30144::DEBUG::2014-04-30 08:39:20,841::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [621c492a]
>Thread-30144::DEBUG::2014-04-30 08:39:20,842::task::579::TaskManager.Task::(_updateState) Task=`656ca6ba-19da-4e7f-8b84-ac5fa9e9ddc2`::moving from state init -> state preparing
>Thread-30144::INFO::2014-04-30 08:39:20,842::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30144::DEBUG::2014-04-30 08:39:20,847::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30144::ERROR::2014-04-30 08:41:25,897::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30144::ERROR::2014-04-30 08:41:25,899::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30144::DEBUG::2014-04-30 08:41:25,899::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30144::DEBUG::2014-04-30 08:41:25,906::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30145::DEBUG::2014-04-30 08:42:21,019::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30145::DEBUG::2014-04-30 08:42:21,020::task::579::TaskManager.Task::(_updateState) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::moving from state init -> state preparing
>Thread-30145::INFO::2014-04-30 08:42:21,020::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30145::DEBUG::2014-04-30 08:42:21,021::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d6f5e1e0-8966-4ebc-99cc-5e7d1ab16063`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30145::DEBUG::2014-04-30 08:42:21,021::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30145::DEBUG::2014-04-30 08:42:21,021::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30145::DEBUG::2014-04-30 08:42:21,021::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d6f5e1e0-8966-4ebc-99cc-5e7d1ab16063`::Granted request
>Thread-30145::DEBUG::2014-04-30 08:42:21,022::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`fcbd373f-0497-4b70-8727-ee028d872692`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30145::DEBUG::2014-04-30 08:42:21,022::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30145::DEBUG::2014-04-30 08:42:21,022::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30145::DEBUG::2014-04-30 08:42:21,022::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`fcbd373f-0497-4b70-8727-ee028d872692`::Granted request
>Thread-30145::INFO::2014-04-30 08:42:21,023::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30145::DEBUG::2014-04-30 08:42:21,023::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30145::DEBUG::2014-04-30 08:42:21,024::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30145::DEBUG::2014-04-30 08:42:21,024::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30145::DEBUG::2014-04-30 08:42:21,024::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30145::DEBUG::2014-04-30 08:42:21,024::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30145::DEBUG::2014-04-30 08:42:21,025::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30145::DEBUG::2014-04-30 08:42:21,025::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30145::DEBUG::2014-04-30 08:42:21,025::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30145::DEBUG::2014-04-30 08:42:21,025::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30145::DEBUG::2014-04-30 08:42:21,025::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30145::DEBUG::2014-04-30 08:42:21,025::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30145::DEBUG::2014-04-30 08:42:21,048::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30145::DEBUG::2014-04-30 08:42:21,048::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30145::DEBUG::2014-04-30 08:42:21,049::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30145::DEBUG::2014-04-30 08:42:21,050::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30145::DEBUG::2014-04-30 08:42:21,050::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30145::DEBUG::2014-04-30 08:42:21,050::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30145::DEBUG::2014-04-30 08:42:21,051::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30145::DEBUG::2014-04-30 08:42:21,051::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30145::DEBUG::2014-04-30 08:42:21,051::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30145::DEBUG::2014-04-30 08:42:21,051::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30145::ERROR::2014-04-30 08:42:21,052::task::850::TaskManager.Task::(_setError) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30145::DEBUG::2014-04-30 08:42:21,053::task::869::TaskManager.Task::(_run) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::Task._run: 6d342084-db9c-41eb-9e93-389781dd06b4 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30145::DEBUG::2014-04-30 08:42:21,053::task::1194::TaskManager.Task::(stop) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::stopping in state preparing (force False)
>Thread-30145::DEBUG::2014-04-30 08:42:21,053::task::974::TaskManager.Task::(_decref) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::ref 1 aborting True
>Thread-30145::INFO::2014-04-30 08:42:21,054::task::1151::TaskManager.Task::(prepare) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30145::DEBUG::2014-04-30 08:42:21,054::task::1156::TaskManager.Task::(prepare) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30145::DEBUG::2014-04-30 08:42:21,054::task::974::TaskManager.Task::(_decref) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::ref 0 aborting True
>Thread-30145::DEBUG::2014-04-30 08:42:21,054::task::909::TaskManager.Task::(_doAbort) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::Task._doAbort: force False
>Thread-30145::DEBUG::2014-04-30 08:42:21,055::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30145::DEBUG::2014-04-30 08:42:21,055::task::579::TaskManager.Task::(_updateState) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::moving from state preparing -> state aborting
>Thread-30145::DEBUG::2014-04-30 08:42:21,055::task::534::TaskManager.Task::(__state_aborting) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::_aborting: recover policy none
>Thread-30145::DEBUG::2014-04-30 08:42:21,055::task::579::TaskManager.Task::(_updateState) Task=`6d342084-db9c-41eb-9e93-389781dd06b4`::moving from state aborting -> state failed
>Thread-30145::DEBUG::2014-04-30 08:42:21,055::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30145::DEBUG::2014-04-30 08:42:21,056::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30145::ERROR::2014-04-30 08:42:21,056::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30144::ERROR::2014-04-30 08:43:30,953::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30144::ERROR::2014-04-30 08:43:30,955::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30144::DEBUG::2014-04-30 08:43:30,955::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30144::DEBUG::2014-04-30 08:43:30,959::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30144::DEBUG::2014-04-30 08:43:30,961::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30144::DEBUG::2014-04-30 08:43:30,961::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30144::INFO::2014-04-30 08:43:30,962::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30144::DEBUG::2014-04-30 08:43:30,962::task::1168::TaskManager.Task::(prepare) Task=`656ca6ba-19da-4e7f-8b84-ac5fa9e9ddc2`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30144::DEBUG::2014-04-30 08:43:30,962::task::579::TaskManager.Task::(_updateState) Task=`656ca6ba-19da-4e7f-8b84-ac5fa9e9ddc2`::moving from state preparing -> state finished
>Thread-30144::DEBUG::2014-04-30 08:43:30,963::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30144::DEBUG::2014-04-30 08:43:30,963::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30144::DEBUG::2014-04-30 08:43:30,963::task::974::TaskManager.Task::(_decref) Task=`656ca6ba-19da-4e7f-8b84-ac5fa9e9ddc2`::ref 0 aborting False
>Thread-30146::DEBUG::2014-04-30 08:44:17,393::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2fdc2d15]
>Thread-30146::DEBUG::2014-04-30 08:44:17,394::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30146::DEBUG::2014-04-30 08:44:17,410::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30146::DEBUG::2014-04-30 08:44:17,448::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30147::DEBUG::2014-04-30 08:44:17,673::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2fdc2d15]
>Thread-30147::ERROR::2014-04-30 08:44:17,673::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30147::DEBUG::2014-04-30 08:44:17,674::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30148::DEBUG::2014-04-30 08:44:18,384::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [482631c]
>Thread-30148::DEBUG::2014-04-30 08:44:18,385::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30148::DEBUG::2014-04-30 08:44:18,399::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30148::DEBUG::2014-04-30 08:44:18,436::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30149::DEBUG::2014-04-30 08:44:18,662::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [482631c]
>Thread-30149::ERROR::2014-04-30 08:44:18,662::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30149::DEBUG::2014-04-30 08:44:18,663::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30151::DEBUG::2014-04-30 08:44:19,125::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [81ef91c]
>Thread-30151::DEBUG::2014-04-30 08:44:19,125::task::579::TaskManager.Task::(_updateState) Task=`327c433b-a913-4090-bdd9-39b88c724f1b`::moving from state init -> state preparing
>Thread-30151::INFO::2014-04-30 08:44:19,126::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30151::DEBUG::2014-04-30 08:44:19,131::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30151::ERROR::2014-04-30 08:46:24,174::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30151::ERROR::2014-04-30 08:46:24,175::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30151::DEBUG::2014-04-30 08:46:24,176::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30151::DEBUG::2014-04-30 08:46:24,180::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30152::DEBUG::2014-04-30 08:47:19,317::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30152::DEBUG::2014-04-30 08:47:19,318::task::579::TaskManager.Task::(_updateState) Task=`99c56526-edb8-493c-9ee0-68440486c846`::moving from state init -> state preparing
>Thread-30152::INFO::2014-04-30 08:47:19,318::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30152::DEBUG::2014-04-30 08:47:19,319::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f931186b-4df4-4e77-8eda-eb0321555b3a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30152::DEBUG::2014-04-30 08:47:19,319::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30152::DEBUG::2014-04-30 08:47:19,320::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30152::DEBUG::2014-04-30 08:47:19,320::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f931186b-4df4-4e77-8eda-eb0321555b3a`::Granted request
>Thread-30152::DEBUG::2014-04-30 08:47:19,320::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`f57bc5ba-c677-48ae-a9f0-9fc599c2aebb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30152::DEBUG::2014-04-30 08:47:19,320::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30152::DEBUG::2014-04-30 08:47:19,321::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30152::DEBUG::2014-04-30 08:47:19,321::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`f57bc5ba-c677-48ae-a9f0-9fc599c2aebb`::Granted request
>Thread-30152::INFO::2014-04-30 08:47:19,321::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30152::DEBUG::2014-04-30 08:47:19,322::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30152::DEBUG::2014-04-30 08:47:19,322::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30152::DEBUG::2014-04-30 08:47:19,323::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30152::DEBUG::2014-04-30 08:47:19,323::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30152::DEBUG::2014-04-30 08:47:19,323::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30152::DEBUG::2014-04-30 08:47:19,323::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30152::DEBUG::2014-04-30 08:47:19,323::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30152::DEBUG::2014-04-30 08:47:19,324::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30152::DEBUG::2014-04-30 08:47:19,324::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30152::DEBUG::2014-04-30 08:47:19,324::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30152::DEBUG::2014-04-30 08:47:19,324::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30152::DEBUG::2014-04-30 08:47:19,346::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30152::DEBUG::2014-04-30 08:47:19,347::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30152::DEBUG::2014-04-30 08:47:19,348::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30152::DEBUG::2014-04-30 08:47:19,348::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30152::DEBUG::2014-04-30 08:47:19,348::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30152::DEBUG::2014-04-30 08:47:19,349::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30152::DEBUG::2014-04-30 08:47:19,349::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30152::DEBUG::2014-04-30 08:47:19,349::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30152::DEBUG::2014-04-30 08:47:19,349::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30152::DEBUG::2014-04-30 08:47:19,350::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30152::ERROR::2014-04-30 08:47:19,350::task::850::TaskManager.Task::(_setError) Task=`99c56526-edb8-493c-9ee0-68440486c846`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30152::DEBUG::2014-04-30 08:47:19,351::task::869::TaskManager.Task::(_run) Task=`99c56526-edb8-493c-9ee0-68440486c846`::Task._run: 99c56526-edb8-493c-9ee0-68440486c846 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30152::DEBUG::2014-04-30 08:47:19,351::task::1194::TaskManager.Task::(stop) Task=`99c56526-edb8-493c-9ee0-68440486c846`::stopping in state preparing (force False)
>Thread-30152::DEBUG::2014-04-30 08:47:19,352::task::974::TaskManager.Task::(_decref) Task=`99c56526-edb8-493c-9ee0-68440486c846`::ref 1 aborting True
>Thread-30152::INFO::2014-04-30 08:47:19,352::task::1151::TaskManager.Task::(prepare) Task=`99c56526-edb8-493c-9ee0-68440486c846`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30152::DEBUG::2014-04-30 08:47:19,352::task::1156::TaskManager.Task::(prepare) Task=`99c56526-edb8-493c-9ee0-68440486c846`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30152::DEBUG::2014-04-30 08:47:19,352::task::974::TaskManager.Task::(_decref) Task=`99c56526-edb8-493c-9ee0-68440486c846`::ref 0 aborting True
>Thread-30152::DEBUG::2014-04-30 08:47:19,353::task::909::TaskManager.Task::(_doAbort) Task=`99c56526-edb8-493c-9ee0-68440486c846`::Task._doAbort: force False
>Thread-30152::DEBUG::2014-04-30 08:47:19,353::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30152::DEBUG::2014-04-30 08:47:19,353::task::579::TaskManager.Task::(_updateState) Task=`99c56526-edb8-493c-9ee0-68440486c846`::moving from state preparing -> state aborting
>Thread-30152::DEBUG::2014-04-30 08:47:19,353::task::534::TaskManager.Task::(__state_aborting) Task=`99c56526-edb8-493c-9ee0-68440486c846`::_aborting: recover policy none
>Thread-30152::DEBUG::2014-04-30 08:47:19,353::task::579::TaskManager.Task::(_updateState) Task=`99c56526-edb8-493c-9ee0-68440486c846`::moving from state aborting -> state failed
>Thread-30152::DEBUG::2014-04-30 08:47:19,354::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30152::DEBUG::2014-04-30 08:47:19,354::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30152::ERROR::2014-04-30 08:47:19,354::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30151::ERROR::2014-04-30 08:48:29,221::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30151::ERROR::2014-04-30 08:48:29,222::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30151::DEBUG::2014-04-30 08:48:29,223::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30151::DEBUG::2014-04-30 08:48:29,227::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30151::DEBUG::2014-04-30 08:48:29,229::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30151::DEBUG::2014-04-30 08:48:29,229::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30151::INFO::2014-04-30 08:48:29,229::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30151::DEBUG::2014-04-30 08:48:29,229::task::1168::TaskManager.Task::(prepare) Task=`327c433b-a913-4090-bdd9-39b88c724f1b`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30151::DEBUG::2014-04-30 08:48:29,230::task::579::TaskManager.Task::(_updateState) Task=`327c433b-a913-4090-bdd9-39b88c724f1b`::moving from state preparing -> state finished
>Thread-30151::DEBUG::2014-04-30 08:48:29,230::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30151::DEBUG::2014-04-30 08:48:29,230::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30151::DEBUG::2014-04-30 08:48:29,230::task::974::TaskManager.Task::(_decref) Task=`327c433b-a913-4090-bdd9-39b88c724f1b`::ref 0 aborting False
>Thread-30153::DEBUG::2014-04-30 08:49:17,391::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [73ec1dca]
>Thread-30153::DEBUG::2014-04-30 08:49:17,392::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30153::DEBUG::2014-04-30 08:49:17,406::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30153::DEBUG::2014-04-30 08:49:17,443::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30154::DEBUG::2014-04-30 08:49:17,670::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [73ec1dca]
>Thread-30154::ERROR::2014-04-30 08:49:17,670::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30154::DEBUG::2014-04-30 08:49:17,671::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30155::DEBUG::2014-04-30 08:49:19,680::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2ad53db9]
>Thread-30155::DEBUG::2014-04-30 08:49:19,680::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30155::DEBUG::2014-04-30 08:49:19,694::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30155::DEBUG::2014-04-30 08:49:19,731::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30156::DEBUG::2014-04-30 08:49:19,957::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2ad53db9]
>Thread-30156::ERROR::2014-04-30 08:49:19,957::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30156::DEBUG::2014-04-30 08:49:19,958::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30158::DEBUG::2014-04-30 08:49:20,421::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [2f594e77]
>Thread-30158::DEBUG::2014-04-30 08:49:20,421::task::579::TaskManager.Task::(_updateState) Task=`5df384b8-5a0d-4958-9364-757927d574c2`::moving from state init -> state preparing
>Thread-30158::INFO::2014-04-30 08:49:20,422::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30158::DEBUG::2014-04-30 08:49:20,425::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30158::ERROR::2014-04-30 08:51:25,467::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30158::ERROR::2014-04-30 08:51:25,468::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30158::DEBUG::2014-04-30 08:51:25,469::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30158::DEBUG::2014-04-30 08:51:25,473::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30159::DEBUG::2014-04-30 08:52:20,608::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30159::DEBUG::2014-04-30 08:52:20,608::task::579::TaskManager.Task::(_updateState) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::moving from state init -> state preparing
>Thread-30159::INFO::2014-04-30 08:52:20,609::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30159::DEBUG::2014-04-30 08:52:20,609::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`1248417a-c5a2-48a0-ab0a-6c3163f9eff5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30159::DEBUG::2014-04-30 08:52:20,609::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30159::DEBUG::2014-04-30 08:52:20,610::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30159::DEBUG::2014-04-30 08:52:20,610::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`1248417a-c5a2-48a0-ab0a-6c3163f9eff5`::Granted request
>Thread-30159::DEBUG::2014-04-30 08:52:20,611::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`5990513e-1b5d-426b-abe8-4559bb847cb1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30159::DEBUG::2014-04-30 08:52:20,611::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30159::DEBUG::2014-04-30 08:52:20,611::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30159::DEBUG::2014-04-30 08:52:20,611::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`5990513e-1b5d-426b-abe8-4559bb847cb1`::Granted request
>Thread-30159::INFO::2014-04-30 08:52:20,612::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30159::DEBUG::2014-04-30 08:52:20,612::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30159::DEBUG::2014-04-30 08:52:20,612::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30159::DEBUG::2014-04-30 08:52:20,613::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30159::DEBUG::2014-04-30 08:52:20,613::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30159::DEBUG::2014-04-30 08:52:20,613::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30159::DEBUG::2014-04-30 08:52:20,613::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30159::DEBUG::2014-04-30 08:52:20,614::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30159::DEBUG::2014-04-30 08:52:20,614::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30159::DEBUG::2014-04-30 08:52:20,614::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30159::DEBUG::2014-04-30 08:52:20,614::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30159::DEBUG::2014-04-30 08:52:20,614::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30159::DEBUG::2014-04-30 08:52:20,637::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30159::DEBUG::2014-04-30 08:52:20,638::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30159::DEBUG::2014-04-30 08:52:20,639::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30159::DEBUG::2014-04-30 08:52:20,639::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30159::DEBUG::2014-04-30 08:52:20,639::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30159::DEBUG::2014-04-30 08:52:20,640::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30159::DEBUG::2014-04-30 08:52:20,640::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30159::DEBUG::2014-04-30 08:52:20,640::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30159::DEBUG::2014-04-30 08:52:20,640::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30159::DEBUG::2014-04-30 08:52:20,641::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30159::ERROR::2014-04-30 08:52:20,641::task::850::TaskManager.Task::(_setError) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30159::DEBUG::2014-04-30 08:52:20,642::task::869::TaskManager.Task::(_run) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::Task._run: 40005e59-0bf7-4279-adbb-78beb8a32476 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30159::DEBUG::2014-04-30 08:52:20,642::task::1194::TaskManager.Task::(stop) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::stopping in state preparing (force False)
>Thread-30159::DEBUG::2014-04-30 08:52:20,643::task::974::TaskManager.Task::(_decref) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::ref 1 aborting True
>Thread-30159::INFO::2014-04-30 08:52:20,643::task::1151::TaskManager.Task::(prepare) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30159::DEBUG::2014-04-30 08:52:20,643::task::1156::TaskManager.Task::(prepare) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30159::DEBUG::2014-04-30 08:52:20,643::task::974::TaskManager.Task::(_decref) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::ref 0 aborting True
>Thread-30159::DEBUG::2014-04-30 08:52:20,644::task::909::TaskManager.Task::(_doAbort) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::Task._doAbort: force False
>Thread-30159::DEBUG::2014-04-30 08:52:20,644::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30159::DEBUG::2014-04-30 08:52:20,644::task::579::TaskManager.Task::(_updateState) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::moving from state preparing -> state aborting
>Thread-30159::DEBUG::2014-04-30 08:52:20,644::task::534::TaskManager.Task::(__state_aborting) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::_aborting: recover policy none
>Thread-30159::DEBUG::2014-04-30 08:52:20,645::task::579::TaskManager.Task::(_updateState) Task=`40005e59-0bf7-4279-adbb-78beb8a32476`::moving from state aborting -> state failed
>Thread-30159::DEBUG::2014-04-30 08:52:20,645::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30159::DEBUG::2014-04-30 08:52:20,645::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30159::ERROR::2014-04-30 08:52:20,645::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30158::ERROR::2014-04-30 08:53:30,514::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30158::ERROR::2014-04-30 08:53:30,515::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30158::DEBUG::2014-04-30 08:53:30,516::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30158::DEBUG::2014-04-30 08:53:30,520::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30158::DEBUG::2014-04-30 08:53:30,521::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30158::DEBUG::2014-04-30 08:53:30,521::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30158::INFO::2014-04-30 08:53:30,521::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30158::DEBUG::2014-04-30 08:53:30,522::task::1168::TaskManager.Task::(prepare) Task=`5df384b8-5a0d-4958-9364-757927d574c2`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30158::DEBUG::2014-04-30 08:53:30,522::task::579::TaskManager.Task::(_updateState) Task=`5df384b8-5a0d-4958-9364-757927d574c2`::moving from state preparing -> state finished
>Thread-30158::DEBUG::2014-04-30 08:53:30,522::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30158::DEBUG::2014-04-30 08:53:30,522::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30158::DEBUG::2014-04-30 08:53:30,523::task::974::TaskManager.Task::(_decref) Task=`5df384b8-5a0d-4958-9364-757927d574c2`::ref 0 aborting False
>Thread-30160::DEBUG::2014-04-30 08:54:17,381::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [426ac5e7]
>Thread-30160::DEBUG::2014-04-30 08:54:17,382::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30160::DEBUG::2014-04-30 08:54:17,396::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30160::DEBUG::2014-04-30 08:54:17,433::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30161::DEBUG::2014-04-30 08:54:17,658::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [426ac5e7]
>Thread-30161::ERROR::2014-04-30 08:54:17,658::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30161::DEBUG::2014-04-30 08:54:17,662::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30162::DEBUG::2014-04-30 08:54:17,979::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [6ca36e68]
>Thread-30162::DEBUG::2014-04-30 08:54:17,981::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30162::DEBUG::2014-04-30 08:54:17,994::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30162::DEBUG::2014-04-30 08:54:18,032::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30163::DEBUG::2014-04-30 08:54:18,259::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [6ca36e68]
>Thread-30163::ERROR::2014-04-30 08:54:18,259::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30163::DEBUG::2014-04-30 08:54:18,260::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30165::DEBUG::2014-04-30 08:54:18,720::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [60b504fc]
>Thread-30165::DEBUG::2014-04-30 08:54:18,721::task::579::TaskManager.Task::(_updateState) Task=`c94fefaf-c353-4817-91b1-ca746ae18412`::moving from state init -> state preparing
>Thread-30165::INFO::2014-04-30 08:54:18,721::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30165::DEBUG::2014-04-30 08:54:18,726::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30165::ERROR::2014-04-30 08:56:23,768::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30165::ERROR::2014-04-30 08:56:23,773::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30165::DEBUG::2014-04-30 08:56:23,774::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30165::DEBUG::2014-04-30 08:56:23,779::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30166::DEBUG::2014-04-30 08:57:18,917::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30166::DEBUG::2014-04-30 08:57:18,918::task::579::TaskManager.Task::(_updateState) Task=`eff82c9a-267b-443c-8268-224503685fc0`::moving from state init -> state preparing
>Thread-30166::INFO::2014-04-30 08:57:18,918::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30166::DEBUG::2014-04-30 08:57:18,919::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`99fd10b8-f1fb-4111-8b93-ebdc25d8596b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30166::DEBUG::2014-04-30 08:57:18,919::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30166::DEBUG::2014-04-30 08:57:18,919::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30166::DEBUG::2014-04-30 08:57:18,920::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`99fd10b8-f1fb-4111-8b93-ebdc25d8596b`::Granted request
>Thread-30166::DEBUG::2014-04-30 08:57:18,921::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`a01766a7-ec47-4b2f-8f7e-6c44a6afd7da`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30166::DEBUG::2014-04-30 08:57:18,921::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30166::DEBUG::2014-04-30 08:57:18,921::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30166::DEBUG::2014-04-30 08:57:18,921::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`a01766a7-ec47-4b2f-8f7e-6c44a6afd7da`::Granted request
>Thread-30166::INFO::2014-04-30 08:57:18,922::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30166::DEBUG::2014-04-30 08:57:18,923::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30166::DEBUG::2014-04-30 08:57:18,923::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30166::DEBUG::2014-04-30 08:57:18,923::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30166::DEBUG::2014-04-30 08:57:18,924::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30166::DEBUG::2014-04-30 08:57:18,924::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30166::DEBUG::2014-04-30 08:57:18,924::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30166::DEBUG::2014-04-30 08:57:18,925::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30166::DEBUG::2014-04-30 08:57:18,925::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30166::DEBUG::2014-04-30 08:57:18,925::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30166::DEBUG::2014-04-30 08:57:18,925::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30166::DEBUG::2014-04-30 08:57:18,926::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30166::DEBUG::2014-04-30 08:57:18,980::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30166::DEBUG::2014-04-30 08:57:18,981::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30166::DEBUG::2014-04-30 08:57:18,982::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30166::DEBUG::2014-04-30 08:57:18,982::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30166::DEBUG::2014-04-30 08:57:18,982::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30166::DEBUG::2014-04-30 08:57:18,982::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30166::DEBUG::2014-04-30 08:57:18,983::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30166::DEBUG::2014-04-30 08:57:18,983::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30166::DEBUG::2014-04-30 08:57:18,983::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30166::DEBUG::2014-04-30 08:57:18,983::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30166::ERROR::2014-04-30 08:57:18,984::task::850::TaskManager.Task::(_setError) Task=`eff82c9a-267b-443c-8268-224503685fc0`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30166::DEBUG::2014-04-30 08:57:18,985::task::869::TaskManager.Task::(_run) Task=`eff82c9a-267b-443c-8268-224503685fc0`::Task._run: eff82c9a-267b-443c-8268-224503685fc0 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30166::DEBUG::2014-04-30 08:57:18,985::task::1194::TaskManager.Task::(stop) Task=`eff82c9a-267b-443c-8268-224503685fc0`::stopping in state preparing (force False)
>Thread-30166::DEBUG::2014-04-30 08:57:18,985::task::974::TaskManager.Task::(_decref) Task=`eff82c9a-267b-443c-8268-224503685fc0`::ref 1 aborting True
>Thread-30166::INFO::2014-04-30 08:57:18,986::task::1151::TaskManager.Task::(prepare) Task=`eff82c9a-267b-443c-8268-224503685fc0`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30166::DEBUG::2014-04-30 08:57:18,986::task::1156::TaskManager.Task::(prepare) Task=`eff82c9a-267b-443c-8268-224503685fc0`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30166::DEBUG::2014-04-30 08:57:18,986::task::974::TaskManager.Task::(_decref) Task=`eff82c9a-267b-443c-8268-224503685fc0`::ref 0 aborting True
>Thread-30166::DEBUG::2014-04-30 08:57:18,986::task::909::TaskManager.Task::(_doAbort) Task=`eff82c9a-267b-443c-8268-224503685fc0`::Task._doAbort: force False
>Thread-30166::DEBUG::2014-04-30 08:57:18,987::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30166::DEBUG::2014-04-30 08:57:18,987::task::579::TaskManager.Task::(_updateState) Task=`eff82c9a-267b-443c-8268-224503685fc0`::moving from state preparing -> state aborting
>Thread-30166::DEBUG::2014-04-30 08:57:18,987::task::534::TaskManager.Task::(__state_aborting) Task=`eff82c9a-267b-443c-8268-224503685fc0`::_aborting: recover policy none
>Thread-30166::DEBUG::2014-04-30 08:57:18,987::task::579::TaskManager.Task::(_updateState) Task=`eff82c9a-267b-443c-8268-224503685fc0`::moving from state aborting -> state failed
>Thread-30166::DEBUG::2014-04-30 08:57:18,987::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30166::DEBUG::2014-04-30 08:57:18,988::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30166::ERROR::2014-04-30 08:57:18,988::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30165::ERROR::2014-04-30 08:58:28,821::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30165::ERROR::2014-04-30 08:58:28,822::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30165::DEBUG::2014-04-30 08:58:28,822::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30165::DEBUG::2014-04-30 08:58:28,827::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30165::DEBUG::2014-04-30 08:58:28,828::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30165::DEBUG::2014-04-30 08:58:28,828::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30165::INFO::2014-04-30 08:58:28,828::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30165::DEBUG::2014-04-30 08:58:28,829::task::1168::TaskManager.Task::(prepare) Task=`c94fefaf-c353-4817-91b1-ca746ae18412`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30165::DEBUG::2014-04-30 08:58:28,829::task::579::TaskManager.Task::(_updateState) Task=`c94fefaf-c353-4817-91b1-ca746ae18412`::moving from state preparing -> state finished
>Thread-30165::DEBUG::2014-04-30 08:58:28,829::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30165::DEBUG::2014-04-30 08:58:28,829::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30165::DEBUG::2014-04-30 08:58:28,829::task::974::TaskManager.Task::(_decref) Task=`c94fefaf-c353-4817-91b1-ca746ae18412`::ref 0 aborting False
>Thread-30167::DEBUG::2014-04-30 08:59:17,429::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5443982e]
>Thread-30167::DEBUG::2014-04-30 08:59:17,430::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30167::DEBUG::2014-04-30 08:59:17,444::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30167::DEBUG::2014-04-30 08:59:17,483::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30168::DEBUG::2014-04-30 08:59:17,708::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5443982e]
>Thread-30168::ERROR::2014-04-30 08:59:17,709::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30168::DEBUG::2014-04-30 08:59:17,710::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30169::DEBUG::2014-04-30 08:59:19,381::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5cc06a25]
>Thread-30169::DEBUG::2014-04-30 08:59:19,381::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30169::DEBUG::2014-04-30 08:59:19,395::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30169::DEBUG::2014-04-30 08:59:19,434::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30170::DEBUG::2014-04-30 08:59:19,672::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5cc06a25]
>Thread-30170::ERROR::2014-04-30 08:59:19,673::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30170::DEBUG::2014-04-30 08:59:19,674::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30172::DEBUG::2014-04-30 08:59:20,183::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [4d0621ba]
>Thread-30172::DEBUG::2014-04-30 08:59:20,184::task::579::TaskManager.Task::(_updateState) Task=`83910dbd-3269-4825-aec0-404cce5fc03a`::moving from state init -> state preparing
>Thread-30172::INFO::2014-04-30 08:59:20,184::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30172::DEBUG::2014-04-30 08:59:20,189::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30172::ERROR::2014-04-30 09:01:25,241::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30172::ERROR::2014-04-30 09:01:25,242::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30172::DEBUG::2014-04-30 09:01:25,243::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30172::DEBUG::2014-04-30 09:01:25,249::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30173::DEBUG::2014-04-30 09:02:20,342::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30173::DEBUG::2014-04-30 09:02:20,343::task::579::TaskManager.Task::(_updateState) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::moving from state init -> state preparing
>Thread-30173::INFO::2014-04-30 09:02:20,343::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30173::DEBUG::2014-04-30 09:02:20,344::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d082a3de-e8ef-4437-b1f9-a67297ab5be8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30173::DEBUG::2014-04-30 09:02:20,344::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30173::DEBUG::2014-04-30 09:02:20,344::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30173::DEBUG::2014-04-30 09:02:20,344::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d082a3de-e8ef-4437-b1f9-a67297ab5be8`::Granted request
>Thread-30173::DEBUG::2014-04-30 09:02:20,345::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`8fcad797-56f6-43cf-8d68-4816db7a485b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30173::DEBUG::2014-04-30 09:02:20,345::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30173::DEBUG::2014-04-30 09:02:20,345::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30173::DEBUG::2014-04-30 09:02:20,345::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`8fcad797-56f6-43cf-8d68-4816db7a485b`::Granted request
>Thread-30173::INFO::2014-04-30 09:02:20,346::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30173::DEBUG::2014-04-30 09:02:20,347::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30173::DEBUG::2014-04-30 09:02:20,347::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30173::DEBUG::2014-04-30 09:02:20,347::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30173::DEBUG::2014-04-30 09:02:20,347::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30173::DEBUG::2014-04-30 09:02:20,347::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30173::DEBUG::2014-04-30 09:02:20,348::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30173::DEBUG::2014-04-30 09:02:20,348::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30173::DEBUG::2014-04-30 09:02:20,348::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30173::DEBUG::2014-04-30 09:02:20,348::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30173::DEBUG::2014-04-30 09:02:20,348::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30173::DEBUG::2014-04-30 09:02:20,349::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30173::DEBUG::2014-04-30 09:02:20,373::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30173::DEBUG::2014-04-30 09:02:20,373::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30173::DEBUG::2014-04-30 09:02:20,374::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30173::DEBUG::2014-04-30 09:02:20,374::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30173::DEBUG::2014-04-30 09:02:20,375::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30173::DEBUG::2014-04-30 09:02:20,375::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30173::DEBUG::2014-04-30 09:02:20,375::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30173::DEBUG::2014-04-30 09:02:20,376::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30173::DEBUG::2014-04-30 09:02:20,376::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30173::DEBUG::2014-04-30 09:02:20,376::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30173::ERROR::2014-04-30 09:02:20,376::task::850::TaskManager.Task::(_setError) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30173::DEBUG::2014-04-30 09:02:20,377::task::869::TaskManager.Task::(_run) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::Task._run: 4251217f-b336-4ce1-aa82-efc38ebbb328 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30173::DEBUG::2014-04-30 09:02:20,378::task::1194::TaskManager.Task::(stop) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::stopping in state preparing (force False)
>Thread-30173::DEBUG::2014-04-30 09:02:20,378::task::974::TaskManager.Task::(_decref) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::ref 1 aborting True
>Thread-30173::INFO::2014-04-30 09:02:20,378::task::1151::TaskManager.Task::(prepare) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30173::DEBUG::2014-04-30 09:02:20,379::task::1156::TaskManager.Task::(prepare) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30173::DEBUG::2014-04-30 09:02:20,379::task::974::TaskManager.Task::(_decref) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::ref 0 aborting True
>Thread-30173::DEBUG::2014-04-30 09:02:20,379::task::909::TaskManager.Task::(_doAbort) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::Task._doAbort: force False
>Thread-30173::DEBUG::2014-04-30 09:02:20,379::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30173::DEBUG::2014-04-30 09:02:20,379::task::579::TaskManager.Task::(_updateState) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::moving from state preparing -> state aborting
>Thread-30173::DEBUG::2014-04-30 09:02:20,380::task::534::TaskManager.Task::(__state_aborting) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::_aborting: recover policy none
>Thread-30173::DEBUG::2014-04-30 09:02:20,380::task::579::TaskManager.Task::(_updateState) Task=`4251217f-b336-4ce1-aa82-efc38ebbb328`::moving from state aborting -> state failed
>Thread-30173::DEBUG::2014-04-30 09:02:20,380::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30173::DEBUG::2014-04-30 09:02:20,380::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30173::ERROR::2014-04-30 09:02:20,381::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30172::ERROR::2014-04-30 09:03:30,294::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30172::ERROR::2014-04-30 09:03:30,296::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30172::DEBUG::2014-04-30 09:03:30,296::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30172::DEBUG::2014-04-30 09:03:30,301::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30172::DEBUG::2014-04-30 09:03:30,303::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30172::DEBUG::2014-04-30 09:03:30,303::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30172::INFO::2014-04-30 09:03:30,303::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30172::DEBUG::2014-04-30 09:03:30,303::task::1168::TaskManager.Task::(prepare) Task=`83910dbd-3269-4825-aec0-404cce5fc03a`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30172::DEBUG::2014-04-30 09:03:30,304::task::579::TaskManager.Task::(_updateState) Task=`83910dbd-3269-4825-aec0-404cce5fc03a`::moving from state preparing -> state finished
>Thread-30172::DEBUG::2014-04-30 09:03:30,304::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30172::DEBUG::2014-04-30 09:03:30,304::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30172::DEBUG::2014-04-30 09:03:30,304::task::974::TaskManager.Task::(_decref) Task=`83910dbd-3269-4825-aec0-404cce5fc03a`::ref 0 aborting False
>Thread-30174::DEBUG::2014-04-30 09:04:17,389::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [14921077]
>Thread-30174::DEBUG::2014-04-30 09:04:17,390::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30174::DEBUG::2014-04-30 09:04:17,407::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30174::DEBUG::2014-04-30 09:04:17,444::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30175::DEBUG::2014-04-30 09:04:17,671::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [14921077]
>Thread-30175::ERROR::2014-04-30 09:04:17,671::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30175::DEBUG::2014-04-30 09:04:17,672::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30176::DEBUG::2014-04-30 09:04:20,763::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2cb2df4d]
>Thread-30176::DEBUG::2014-04-30 09:04:20,764::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30176::DEBUG::2014-04-30 09:04:20,778::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30176::DEBUG::2014-04-30 09:04:20,815::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30177::DEBUG::2014-04-30 09:04:21,108::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2cb2df4d]
>Thread-30177::ERROR::2014-04-30 09:04:21,108::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30177::DEBUG::2014-04-30 09:04:21,109::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30179::DEBUG::2014-04-30 09:04:21,597::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [6787e0ee]
>Thread-30179::DEBUG::2014-04-30 09:04:21,598::task::579::TaskManager.Task::(_updateState) Task=`abcfa4e9-47ba-499d-b087-a3adbdbc2b3a`::moving from state init -> state preparing
>Thread-30179::INFO::2014-04-30 09:04:21,598::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30179::DEBUG::2014-04-30 09:04:21,603::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30179::ERROR::2014-04-30 09:06:26,646::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30179::ERROR::2014-04-30 09:06:26,647::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30179::DEBUG::2014-04-30 09:06:26,647::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30179::DEBUG::2014-04-30 09:06:26,652::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30180::DEBUG::2014-04-30 09:07:21,769::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30180::DEBUG::2014-04-30 09:07:21,769::task::579::TaskManager.Task::(_updateState) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::moving from state init -> state preparing
>Thread-30180::INFO::2014-04-30 09:07:21,770::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30180::DEBUG::2014-04-30 09:07:21,770::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`72b1c666-bfa7-4361-8579-f24ab274d29d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30180::DEBUG::2014-04-30 09:07:21,770::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30180::DEBUG::2014-04-30 09:07:21,771::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30180::DEBUG::2014-04-30 09:07:21,771::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`72b1c666-bfa7-4361-8579-f24ab274d29d`::Granted request
>Thread-30180::DEBUG::2014-04-30 09:07:21,772::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`707a0dfc-27f4-4634-bfbd-d1d0d5055dd1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30180::DEBUG::2014-04-30 09:07:21,772::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30180::DEBUG::2014-04-30 09:07:21,772::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30180::DEBUG::2014-04-30 09:07:21,772::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`707a0dfc-27f4-4634-bfbd-d1d0d5055dd1`::Granted request
>Thread-30180::INFO::2014-04-30 09:07:21,773::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30180::DEBUG::2014-04-30 09:07:21,773::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30180::DEBUG::2014-04-30 09:07:21,773::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30180::DEBUG::2014-04-30 09:07:21,774::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30180::DEBUG::2014-04-30 09:07:21,774::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30180::DEBUG::2014-04-30 09:07:21,774::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30180::DEBUG::2014-04-30 09:07:21,774::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30180::DEBUG::2014-04-30 09:07:21,774::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30180::DEBUG::2014-04-30 09:07:21,775::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30180::DEBUG::2014-04-30 09:07:21,775::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30180::DEBUG::2014-04-30 09:07:21,775::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30180::DEBUG::2014-04-30 09:07:21,775::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30180::DEBUG::2014-04-30 09:07:21,798::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30180::DEBUG::2014-04-30 09:07:21,799::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30180::DEBUG::2014-04-30 09:07:21,801::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30180::DEBUG::2014-04-30 09:07:21,801::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30180::DEBUG::2014-04-30 09:07:21,802::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30180::DEBUG::2014-04-30 09:07:21,802::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30180::DEBUG::2014-04-30 09:07:21,802::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30180::DEBUG::2014-04-30 09:07:21,803::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30180::DEBUG::2014-04-30 09:07:21,803::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30180::DEBUG::2014-04-30 09:07:21,803::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30180::ERROR::2014-04-30 09:07:21,803::task::850::TaskManager.Task::(_setError) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30180::DEBUG::2014-04-30 09:07:21,804::task::869::TaskManager.Task::(_run) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::Task._run: f4d6dd31-4942-492f-a83d-3bd6df635d83 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30180::DEBUG::2014-04-30 09:07:21,805::task::1194::TaskManager.Task::(stop) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::stopping in state preparing (force False)
>Thread-30180::DEBUG::2014-04-30 09:07:21,805::task::974::TaskManager.Task::(_decref) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::ref 1 aborting True
>Thread-30180::INFO::2014-04-30 09:07:21,805::task::1151::TaskManager.Task::(prepare) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30180::DEBUG::2014-04-30 09:07:21,805::task::1156::TaskManager.Task::(prepare) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30180::DEBUG::2014-04-30 09:07:21,806::task::974::TaskManager.Task::(_decref) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::ref 0 aborting True
>Thread-30180::DEBUG::2014-04-30 09:07:21,806::task::909::TaskManager.Task::(_doAbort) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::Task._doAbort: force False
>Thread-30180::DEBUG::2014-04-30 09:07:21,806::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30180::DEBUG::2014-04-30 09:07:21,806::task::579::TaskManager.Task::(_updateState) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::moving from state preparing -> state aborting
>Thread-30180::DEBUG::2014-04-30 09:07:21,807::task::534::TaskManager.Task::(__state_aborting) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::_aborting: recover policy none
>Thread-30180::DEBUG::2014-04-30 09:07:21,807::task::579::TaskManager.Task::(_updateState) Task=`f4d6dd31-4942-492f-a83d-3bd6df635d83`::moving from state aborting -> state failed
>Thread-30180::DEBUG::2014-04-30 09:07:21,807::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30180::DEBUG::2014-04-30 09:07:21,807::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30180::ERROR::2014-04-30 09:07:21,807::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30179::ERROR::2014-04-30 09:08:31,693::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30179::ERROR::2014-04-30 09:08:31,694::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30179::DEBUG::2014-04-30 09:08:31,694::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30179::DEBUG::2014-04-30 09:08:31,698::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30179::DEBUG::2014-04-30 09:08:31,699::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30179::DEBUG::2014-04-30 09:08:31,699::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30179::INFO::2014-04-30 09:08:31,700::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30179::DEBUG::2014-04-30 09:08:31,700::task::1168::TaskManager.Task::(prepare) Task=`abcfa4e9-47ba-499d-b087-a3adbdbc2b3a`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30179::DEBUG::2014-04-30 09:08:31,700::task::579::TaskManager.Task::(_updateState) Task=`abcfa4e9-47ba-499d-b087-a3adbdbc2b3a`::moving from state preparing -> state finished
>Thread-30179::DEBUG::2014-04-30 09:08:31,700::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30179::DEBUG::2014-04-30 09:08:31,701::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30179::DEBUG::2014-04-30 09:08:31,701::task::974::TaskManager.Task::(_decref) Task=`abcfa4e9-47ba-499d-b087-a3adbdbc2b3a`::ref 0 aborting False
>Thread-30181::DEBUG::2014-04-30 09:09:17,360::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [520bb3a3]
>Thread-30181::DEBUG::2014-04-30 09:09:17,361::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30181::DEBUG::2014-04-30 09:09:17,375::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30181::DEBUG::2014-04-30 09:09:17,412::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30182::DEBUG::2014-04-30 09:09:17,638::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [520bb3a3]
>Thread-30182::ERROR::2014-04-30 09:09:17,638::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30182::DEBUG::2014-04-30 09:09:17,639::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30183::DEBUG::2014-04-30 09:09:19,143::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [111f2de]
>Thread-30183::DEBUG::2014-04-30 09:09:19,144::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30183::DEBUG::2014-04-30 09:09:19,158::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30183::DEBUG::2014-04-30 09:09:19,195::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30184::DEBUG::2014-04-30 09:09:19,420::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [111f2de]
>Thread-30184::ERROR::2014-04-30 09:09:19,421::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30184::DEBUG::2014-04-30 09:09:19,422::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30186::DEBUG::2014-04-30 09:09:19,882::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [19a9006b]
>Thread-30186::DEBUG::2014-04-30 09:09:19,883::task::579::TaskManager.Task::(_updateState) Task=`28fcd921-3c32-4387-a44c-db22a096a541`::moving from state init -> state preparing
>Thread-30186::INFO::2014-04-30 09:09:19,883::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30186::DEBUG::2014-04-30 09:09:19,888::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30186::ERROR::2014-04-30 09:11:24,929::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30186::ERROR::2014-04-30 09:11:24,934::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30186::DEBUG::2014-04-30 09:11:24,935::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30186::DEBUG::2014-04-30 09:11:24,940::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30187::DEBUG::2014-04-30 09:12:20,072::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30187::DEBUG::2014-04-30 09:12:20,073::task::579::TaskManager.Task::(_updateState) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::moving from state init -> state preparing
>Thread-30187::INFO::2014-04-30 09:12:20,073::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30187::DEBUG::2014-04-30 09:12:20,074::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`8c39ac3f-aba3-453c-bd3b-c738873ac8e9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30187::DEBUG::2014-04-30 09:12:20,074::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30187::DEBUG::2014-04-30 09:12:20,074::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30187::DEBUG::2014-04-30 09:12:20,075::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`8c39ac3f-aba3-453c-bd3b-c738873ac8e9`::Granted request
>Thread-30187::DEBUG::2014-04-30 09:12:20,075::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`81969700-4afa-4825-9a2d-f984398bede1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30187::DEBUG::2014-04-30 09:12:20,076::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30187::DEBUG::2014-04-30 09:12:20,076::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30187::DEBUG::2014-04-30 09:12:20,076::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`81969700-4afa-4825-9a2d-f984398bede1`::Granted request
>Thread-30187::INFO::2014-04-30 09:12:20,077::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30187::DEBUG::2014-04-30 09:12:20,077::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30187::DEBUG::2014-04-30 09:12:20,078::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30187::DEBUG::2014-04-30 09:12:20,078::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30187::DEBUG::2014-04-30 09:12:20,078::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30187::DEBUG::2014-04-30 09:12:20,079::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30187::DEBUG::2014-04-30 09:12:20,079::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30187::DEBUG::2014-04-30 09:12:20,079::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30187::DEBUG::2014-04-30 09:12:20,079::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30187::DEBUG::2014-04-30 09:12:20,080::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30187::DEBUG::2014-04-30 09:12:20,080::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30187::DEBUG::2014-04-30 09:12:20,080::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30187::DEBUG::2014-04-30 09:12:20,102::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30187::DEBUG::2014-04-30 09:12:20,103::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30187::DEBUG::2014-04-30 09:12:20,104::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30187::DEBUG::2014-04-30 09:12:20,104::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30187::DEBUG::2014-04-30 09:12:20,105::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30187::DEBUG::2014-04-30 09:12:20,105::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30187::DEBUG::2014-04-30 09:12:20,105::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30187::DEBUG::2014-04-30 09:12:20,106::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30187::DEBUG::2014-04-30 09:12:20,106::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30187::DEBUG::2014-04-30 09:12:20,106::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30187::ERROR::2014-04-30 09:12:20,106::task::850::TaskManager.Task::(_setError) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30187::DEBUG::2014-04-30 09:12:20,107::task::869::TaskManager.Task::(_run) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::Task._run: 4ac2a00a-a2db-401a-8de5-6e08c254eebf ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30187::DEBUG::2014-04-30 09:12:20,108::task::1194::TaskManager.Task::(stop) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::stopping in state preparing (force False)
>Thread-30187::DEBUG::2014-04-30 09:12:20,108::task::974::TaskManager.Task::(_decref) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::ref 1 aborting True
>Thread-30187::INFO::2014-04-30 09:12:20,108::task::1151::TaskManager.Task::(prepare) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30187::DEBUG::2014-04-30 09:12:20,108::task::1156::TaskManager.Task::(prepare) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30187::DEBUG::2014-04-30 09:12:20,109::task::974::TaskManager.Task::(_decref) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::ref 0 aborting True
>Thread-30187::DEBUG::2014-04-30 09:12:20,109::task::909::TaskManager.Task::(_doAbort) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::Task._doAbort: force False
>Thread-30187::DEBUG::2014-04-30 09:12:20,109::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30187::DEBUG::2014-04-30 09:12:20,109::task::579::TaskManager.Task::(_updateState) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::moving from state preparing -> state aborting
>Thread-30187::DEBUG::2014-04-30 09:12:20,110::task::534::TaskManager.Task::(__state_aborting) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::_aborting: recover policy none
>Thread-30187::DEBUG::2014-04-30 09:12:20,110::task::579::TaskManager.Task::(_updateState) Task=`4ac2a00a-a2db-401a-8de5-6e08c254eebf`::moving from state aborting -> state failed
>Thread-30187::DEBUG::2014-04-30 09:12:20,110::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30187::DEBUG::2014-04-30 09:12:20,110::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30187::ERROR::2014-04-30 09:12:20,111::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30186::ERROR::2014-04-30 09:13:29,983::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30186::ERROR::2014-04-30 09:13:29,984::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30186::DEBUG::2014-04-30 09:13:29,984::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30186::DEBUG::2014-04-30 09:13:29,989::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30186::DEBUG::2014-04-30 09:13:29,990::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30186::DEBUG::2014-04-30 09:13:29,990::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30186::INFO::2014-04-30 09:13:29,990::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30186::DEBUG::2014-04-30 09:13:29,991::task::1168::TaskManager.Task::(prepare) Task=`28fcd921-3c32-4387-a44c-db22a096a541`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30186::DEBUG::2014-04-30 09:13:29,991::task::579::TaskManager.Task::(_updateState) Task=`28fcd921-3c32-4387-a44c-db22a096a541`::moving from state preparing -> state finished
>Thread-30186::DEBUG::2014-04-30 09:13:29,991::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30186::DEBUG::2014-04-30 09:13:29,991::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30186::DEBUG::2014-04-30 09:13:29,992::task::974::TaskManager.Task::(_decref) Task=`28fcd921-3c32-4387-a44c-db22a096a541`::ref 0 aborting False
>Thread-30188::DEBUG::2014-04-30 09:14:17,352::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [361df663]
>Thread-30188::DEBUG::2014-04-30 09:14:17,353::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30188::DEBUG::2014-04-30 09:14:17,367::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30188::DEBUG::2014-04-30 09:14:17,404::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30189::DEBUG::2014-04-30 09:14:22,753::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [361df663]
>Thread-30189::ERROR::2014-04-30 09:14:22,753::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30189::DEBUG::2014-04-30 09:14:22,754::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30190::DEBUG::2014-04-30 09:14:23,443::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [1c82b147]
>Thread-30190::DEBUG::2014-04-30 09:14:23,443::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30190::DEBUG::2014-04-30 09:14:23,457::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30190::DEBUG::2014-04-30 09:14:23,494::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30191::DEBUG::2014-04-30 09:14:23,719::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [1c82b147]
>Thread-30191::ERROR::2014-04-30 09:14:23,720::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30191::DEBUG::2014-04-30 09:14:23,720::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30193::DEBUG::2014-04-30 09:14:24,188::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [3f17578]
>Thread-30193::DEBUG::2014-04-30 09:14:24,189::task::579::TaskManager.Task::(_updateState) Task=`f1f31d75-54b5-4397-9bc7-9ee19d9cc614`::moving from state init -> state preparing
>Thread-30193::INFO::2014-04-30 09:14:24,189::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30193::DEBUG::2014-04-30 09:14:24,194::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30193::ERROR::2014-04-30 09:16:29,237::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30193::ERROR::2014-04-30 09:16:29,238::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30193::DEBUG::2014-04-30 09:16:29,238::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30193::DEBUG::2014-04-30 09:16:29,243::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30194::DEBUG::2014-04-30 09:17:24,381::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30194::DEBUG::2014-04-30 09:17:24,381::task::579::TaskManager.Task::(_updateState) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::moving from state init -> state preparing
>Thread-30194::INFO::2014-04-30 09:17:24,382::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30194::DEBUG::2014-04-30 09:17:24,382::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ec9b3f5c-4dd3-49a3-96b6-a4bda6a6372f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30194::DEBUG::2014-04-30 09:17:24,382::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30194::DEBUG::2014-04-30 09:17:24,383::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30194::DEBUG::2014-04-30 09:17:24,383::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ec9b3f5c-4dd3-49a3-96b6-a4bda6a6372f`::Granted request
>Thread-30194::DEBUG::2014-04-30 09:17:24,383::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`991a3648-1490-4ad4-afd4-e90f78e706d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30194::DEBUG::2014-04-30 09:17:24,384::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30194::DEBUG::2014-04-30 09:17:24,384::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30194::DEBUG::2014-04-30 09:17:24,384::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`991a3648-1490-4ad4-afd4-e90f78e706d2`::Granted request
>Thread-30194::INFO::2014-04-30 09:17:24,384::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30194::DEBUG::2014-04-30 09:17:24,385::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30194::DEBUG::2014-04-30 09:17:24,385::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30194::DEBUG::2014-04-30 09:17:24,386::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30194::DEBUG::2014-04-30 09:17:24,386::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30194::DEBUG::2014-04-30 09:17:24,386::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30194::DEBUG::2014-04-30 09:17:24,386::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30194::DEBUG::2014-04-30 09:17:24,386::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30194::DEBUG::2014-04-30 09:17:24,387::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30194::DEBUG::2014-04-30 09:17:24,387::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30194::DEBUG::2014-04-30 09:17:24,387::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30194::DEBUG::2014-04-30 09:17:24,387::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30194::DEBUG::2014-04-30 09:17:24,409::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30194::DEBUG::2014-04-30 09:17:24,410::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30194::DEBUG::2014-04-30 09:17:24,410::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30194::DEBUG::2014-04-30 09:17:24,411::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30194::DEBUG::2014-04-30 09:17:24,411::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30194::DEBUG::2014-04-30 09:17:24,411::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30194::DEBUG::2014-04-30 09:17:24,412::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30194::DEBUG::2014-04-30 09:17:24,412::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30194::DEBUG::2014-04-30 09:17:24,412::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30194::DEBUG::2014-04-30 09:17:24,412::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30194::ERROR::2014-04-30 09:17:24,413::task::850::TaskManager.Task::(_setError) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30194::DEBUG::2014-04-30 09:17:24,414::task::869::TaskManager.Task::(_run) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::Task._run: 3bc1189c-9589-4c4c-bb82-bfbbbcb58568 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30194::DEBUG::2014-04-30 09:17:24,414::task::1194::TaskManager.Task::(stop) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::stopping in state preparing (force False)
>Thread-30194::DEBUG::2014-04-30 09:17:24,414::task::974::TaskManager.Task::(_decref) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::ref 1 aborting True
>Thread-30194::INFO::2014-04-30 09:17:24,415::task::1151::TaskManager.Task::(prepare) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30194::DEBUG::2014-04-30 09:17:24,415::task::1156::TaskManager.Task::(prepare) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30194::DEBUG::2014-04-30 09:17:24,415::task::974::TaskManager.Task::(_decref) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::ref 0 aborting True
>Thread-30194::DEBUG::2014-04-30 09:17:24,415::task::909::TaskManager.Task::(_doAbort) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::Task._doAbort: force False
>Thread-30194::DEBUG::2014-04-30 09:17:24,415::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30194::DEBUG::2014-04-30 09:17:24,416::task::579::TaskManager.Task::(_updateState) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::moving from state preparing -> state aborting
>Thread-30194::DEBUG::2014-04-30 09:17:24,416::task::534::TaskManager.Task::(__state_aborting) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::_aborting: recover policy none
>Thread-30194::DEBUG::2014-04-30 09:17:24,416::task::579::TaskManager.Task::(_updateState) Task=`3bc1189c-9589-4c4c-bb82-bfbbbcb58568`::moving from state aborting -> state failed
>Thread-30194::DEBUG::2014-04-30 09:17:24,416::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30194::DEBUG::2014-04-30 09:17:24,417::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30194::ERROR::2014-04-30 09:17:24,417::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30193::ERROR::2014-04-30 09:18:34,285::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30193::ERROR::2014-04-30 09:18:34,286::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30193::DEBUG::2014-04-30 09:18:34,286::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30193::DEBUG::2014-04-30 09:18:34,290::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30193::DEBUG::2014-04-30 09:18:34,291::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30193::DEBUG::2014-04-30 09:18:34,292::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30193::INFO::2014-04-30 09:18:34,292::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30193::DEBUG::2014-04-30 09:18:34,292::task::1168::TaskManager.Task::(prepare) Task=`f1f31d75-54b5-4397-9bc7-9ee19d9cc614`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30193::DEBUG::2014-04-30 09:18:34,292::task::579::TaskManager.Task::(_updateState) Task=`f1f31d75-54b5-4397-9bc7-9ee19d9cc614`::moving from state preparing -> state finished
>Thread-30193::DEBUG::2014-04-30 09:18:34,293::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30193::DEBUG::2014-04-30 09:18:34,293::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30193::DEBUG::2014-04-30 09:18:34,293::task::974::TaskManager.Task::(_decref) Task=`f1f31d75-54b5-4397-9bc7-9ee19d9cc614`::ref 0 aborting False
>Thread-30195::DEBUG::2014-04-30 09:19:17,356::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [264b137a]
>Thread-30195::DEBUG::2014-04-30 09:19:17,357::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30195::DEBUG::2014-04-30 09:19:17,371::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30195::DEBUG::2014-04-30 09:19:17,408::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30196::DEBUG::2014-04-30 09:19:17,639::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [264b137a]
>Thread-30196::ERROR::2014-04-30 09:19:17,640::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30196::DEBUG::2014-04-30 09:19:17,640::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30197::DEBUG::2014-04-30 09:19:18,738::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [764adf7a]
>Thread-30197::DEBUG::2014-04-30 09:19:18,739::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30197::DEBUG::2014-04-30 09:19:18,753::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30197::DEBUG::2014-04-30 09:19:18,792::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30198::DEBUG::2014-04-30 09:19:19,016::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [764adf7a]
>Thread-30198::ERROR::2014-04-30 09:19:19,017::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30198::DEBUG::2014-04-30 09:19:19,017::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30200::DEBUG::2014-04-30 09:19:19,483::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [69e0ef3f]
>Thread-30200::DEBUG::2014-04-30 09:19:19,483::task::579::TaskManager.Task::(_updateState) Task=`a489630f-5286-4dd7-a891-179fd92e91b2`::moving from state init -> state preparing
>Thread-30200::INFO::2014-04-30 09:19:19,484::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30200::DEBUG::2014-04-30 09:19:19,487::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30200::ERROR::2014-04-30 09:21:24,527::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30200::ERROR::2014-04-30 09:21:24,529::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30200::DEBUG::2014-04-30 09:21:24,530::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30200::DEBUG::2014-04-30 09:21:24,535::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30201::DEBUG::2014-04-30 09:22:19,668::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30201::DEBUG::2014-04-30 09:22:19,669::task::579::TaskManager.Task::(_updateState) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::moving from state init -> state preparing
>Thread-30201::INFO::2014-04-30 09:22:19,669::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30201::DEBUG::2014-04-30 09:22:19,670::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bfedcda2-8725-420d-b74e-2f594fb7668c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30201::DEBUG::2014-04-30 09:22:19,670::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30201::DEBUG::2014-04-30 09:22:19,670::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30201::DEBUG::2014-04-30 09:22:19,671::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bfedcda2-8725-420d-b74e-2f594fb7668c`::Granted request
>Thread-30201::DEBUG::2014-04-30 09:22:19,671::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`85bb99fe-cdba-4076-be31-2381be6c1a5e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30201::DEBUG::2014-04-30 09:22:19,671::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30201::DEBUG::2014-04-30 09:22:19,672::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30201::DEBUG::2014-04-30 09:22:19,672::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`85bb99fe-cdba-4076-be31-2381be6c1a5e`::Granted request
>Thread-30201::INFO::2014-04-30 09:22:19,672::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30201::DEBUG::2014-04-30 09:22:19,673::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30201::DEBUG::2014-04-30 09:22:19,673::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30201::DEBUG::2014-04-30 09:22:19,673::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30201::DEBUG::2014-04-30 09:22:19,673::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30201::DEBUG::2014-04-30 09:22:19,674::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30201::DEBUG::2014-04-30 09:22:19,674::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30201::DEBUG::2014-04-30 09:22:19,674::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30201::DEBUG::2014-04-30 09:22:19,674::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30201::DEBUG::2014-04-30 09:22:19,674::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30201::DEBUG::2014-04-30 09:22:19,675::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30201::DEBUG::2014-04-30 09:22:19,675::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30201::DEBUG::2014-04-30 09:22:19,699::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30201::DEBUG::2014-04-30 09:22:19,699::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30201::DEBUG::2014-04-30 09:22:19,700::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30201::DEBUG::2014-04-30 09:22:19,701::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30201::DEBUG::2014-04-30 09:22:19,701::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30201::DEBUG::2014-04-30 09:22:19,701::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30201::DEBUG::2014-04-30 09:22:19,701::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30201::DEBUG::2014-04-30 09:22:19,702::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30201::DEBUG::2014-04-30 09:22:19,702::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30201::DEBUG::2014-04-30 09:22:19,702::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30201::ERROR::2014-04-30 09:22:19,702::task::850::TaskManager.Task::(_setError) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30201::DEBUG::2014-04-30 09:22:19,704::task::869::TaskManager.Task::(_run) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::Task._run: 64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30201::DEBUG::2014-04-30 09:22:19,704::task::1194::TaskManager.Task::(stop) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::stopping in state preparing (force False)
>Thread-30201::DEBUG::2014-04-30 09:22:19,704::task::974::TaskManager.Task::(_decref) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::ref 1 aborting True
>Thread-30201::INFO::2014-04-30 09:22:19,704::task::1151::TaskManager.Task::(prepare) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30201::DEBUG::2014-04-30 09:22:19,705::task::1156::TaskManager.Task::(prepare) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30201::DEBUG::2014-04-30 09:22:19,705::task::974::TaskManager.Task::(_decref) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::ref 0 aborting True
>Thread-30201::DEBUG::2014-04-30 09:22:19,705::task::909::TaskManager.Task::(_doAbort) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::Task._doAbort: force False
>Thread-30201::DEBUG::2014-04-30 09:22:19,705::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30201::DEBUG::2014-04-30 09:22:19,706::task::579::TaskManager.Task::(_updateState) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::moving from state preparing -> state aborting
>Thread-30201::DEBUG::2014-04-30 09:22:19,706::task::534::TaskManager.Task::(__state_aborting) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::_aborting: recover policy none
>Thread-30201::DEBUG::2014-04-30 09:22:19,706::task::579::TaskManager.Task::(_updateState) Task=`64f38fd1-cfd2-461d-9d55-bfe79c3f8cb8`::moving from state aborting -> state failed
>Thread-30201::DEBUG::2014-04-30 09:22:19,706::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30201::DEBUG::2014-04-30 09:22:19,706::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30201::ERROR::2014-04-30 09:22:19,707::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30200::ERROR::2014-04-30 09:23:29,578::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30200::ERROR::2014-04-30 09:23:29,579::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30200::DEBUG::2014-04-30 09:23:29,579::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30200::DEBUG::2014-04-30 09:23:29,583::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30200::DEBUG::2014-04-30 09:23:29,584::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30200::DEBUG::2014-04-30 09:23:29,584::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30200::INFO::2014-04-30 09:23:29,585::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30200::DEBUG::2014-04-30 09:23:29,585::task::1168::TaskManager.Task::(prepare) Task=`a489630f-5286-4dd7-a891-179fd92e91b2`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30200::DEBUG::2014-04-30 09:23:29,585::task::579::TaskManager.Task::(_updateState) Task=`a489630f-5286-4dd7-a891-179fd92e91b2`::moving from state preparing -> state finished
>Thread-30200::DEBUG::2014-04-30 09:23:29,585::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30200::DEBUG::2014-04-30 09:23:29,586::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30200::DEBUG::2014-04-30 09:23:29,586::task::974::TaskManager.Task::(_decref) Task=`a489630f-5286-4dd7-a891-179fd92e91b2`::ref 0 aborting False
>Thread-30202::DEBUG::2014-04-30 09:24:17,357::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [79d3d5b5]
>Thread-30202::DEBUG::2014-04-30 09:24:17,358::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30202::DEBUG::2014-04-30 09:24:17,372::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30202::DEBUG::2014-04-30 09:24:17,413::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30203::DEBUG::2014-04-30 09:24:17,640::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [79d3d5b5]
>Thread-30203::ERROR::2014-04-30 09:24:17,641::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30203::DEBUG::2014-04-30 09:24:17,641::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30204::DEBUG::2014-04-30 09:24:20,042::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [10535fa3]
>Thread-30204::DEBUG::2014-04-30 09:24:20,043::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30204::DEBUG::2014-04-30 09:24:20,057::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30204::DEBUG::2014-04-30 09:24:20,095::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30205::DEBUG::2014-04-30 09:24:20,328::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [10535fa3]
>Thread-30205::ERROR::2014-04-30 09:24:20,329::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30205::DEBUG::2014-04-30 09:24:20,329::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30207::DEBUG::2014-04-30 09:24:20,816::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [24e619b9]
>Thread-30207::DEBUG::2014-04-30 09:24:20,817::task::579::TaskManager.Task::(_updateState) Task=`2959ae12-3cd9-4d35-845a-fa46a423bea2`::moving from state init -> state preparing
>Thread-30207::INFO::2014-04-30 09:24:20,818::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30207::DEBUG::2014-04-30 09:24:20,822::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30207::ERROR::2014-04-30 09:26:25,865::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30207::ERROR::2014-04-30 09:26:25,867::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30207::DEBUG::2014-04-30 09:26:25,867::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30207::DEBUG::2014-04-30 09:26:25,872::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30208::DEBUG::2014-04-30 09:27:20,996::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30208::DEBUG::2014-04-30 09:27:20,997::task::579::TaskManager.Task::(_updateState) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::moving from state init -> state preparing
>Thread-30208::INFO::2014-04-30 09:27:20,997::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30208::DEBUG::2014-04-30 09:27:20,998::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a95e1799-05dc-490a-a729-2f41bcf575bb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30208::DEBUG::2014-04-30 09:27:20,998::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30208::DEBUG::2014-04-30 09:27:20,998::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30208::DEBUG::2014-04-30 09:27:20,999::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a95e1799-05dc-490a-a729-2f41bcf575bb`::Granted request
>Thread-30208::DEBUG::2014-04-30 09:27:20,999::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`40249235-4ff0-4d18-8345-d342222b5d2e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30208::DEBUG::2014-04-30 09:27:20,999::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30208::DEBUG::2014-04-30 09:27:21,000::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30208::DEBUG::2014-04-30 09:27:21,000::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`40249235-4ff0-4d18-8345-d342222b5d2e`::Granted request
>Thread-30208::INFO::2014-04-30 09:27:21,000::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30208::DEBUG::2014-04-30 09:27:21,001::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30208::DEBUG::2014-04-30 09:27:21,001::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30208::DEBUG::2014-04-30 09:27:21,001::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30208::DEBUG::2014-04-30 09:27:21,002::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30208::DEBUG::2014-04-30 09:27:21,002::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30208::DEBUG::2014-04-30 09:27:21,002::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30208::DEBUG::2014-04-30 09:27:21,002::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30208::DEBUG::2014-04-30 09:27:21,002::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30208::DEBUG::2014-04-30 09:27:21,003::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30208::DEBUG::2014-04-30 09:27:21,003::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30208::DEBUG::2014-04-30 09:27:21,003::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30208::DEBUG::2014-04-30 09:27:21,025::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30208::DEBUG::2014-04-30 09:27:21,026::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30208::DEBUG::2014-04-30 09:27:21,027::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30208::DEBUG::2014-04-30 09:27:21,027::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30208::DEBUG::2014-04-30 09:27:21,027::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30208::DEBUG::2014-04-30 09:27:21,028::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30208::DEBUG::2014-04-30 09:27:21,028::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30208::DEBUG::2014-04-30 09:27:21,028::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30208::DEBUG::2014-04-30 09:27:21,029::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30208::DEBUG::2014-04-30 09:27:21,029::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30208::ERROR::2014-04-30 09:27:21,029::task::850::TaskManager.Task::(_setError) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30208::DEBUG::2014-04-30 09:27:21,030::task::869::TaskManager.Task::(_run) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::Task._run: a6c38318-1d28-42bd-91d7-66e94cb34be9 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30208::DEBUG::2014-04-30 09:27:21,031::task::1194::TaskManager.Task::(stop) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::stopping in state preparing (force False)
>Thread-30208::DEBUG::2014-04-30 09:27:21,031::task::974::TaskManager.Task::(_decref) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::ref 1 aborting True
>Thread-30208::INFO::2014-04-30 09:27:21,031::task::1151::TaskManager.Task::(prepare) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30208::DEBUG::2014-04-30 09:27:21,031::task::1156::TaskManager.Task::(prepare) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30208::DEBUG::2014-04-30 09:27:21,032::task::974::TaskManager.Task::(_decref) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::ref 0 aborting True
>Thread-30208::DEBUG::2014-04-30 09:27:21,032::task::909::TaskManager.Task::(_doAbort) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::Task._doAbort: force False
>Thread-30208::DEBUG::2014-04-30 09:27:21,032::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30208::DEBUG::2014-04-30 09:27:21,032::task::579::TaskManager.Task::(_updateState) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::moving from state preparing -> state aborting
>Thread-30208::DEBUG::2014-04-30 09:27:21,032::task::534::TaskManager.Task::(__state_aborting) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::_aborting: recover policy none
>Thread-30208::DEBUG::2014-04-30 09:27:21,033::task::579::TaskManager.Task::(_updateState) Task=`a6c38318-1d28-42bd-91d7-66e94cb34be9`::moving from state aborting -> state failed
>Thread-30208::DEBUG::2014-04-30 09:27:21,033::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30208::DEBUG::2014-04-30 09:27:21,033::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30208::ERROR::2014-04-30 09:27:21,033::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30207::ERROR::2014-04-30 09:28:30,916::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30207::ERROR::2014-04-30 09:28:30,918::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30207::DEBUG::2014-04-30 09:28:30,919::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30207::DEBUG::2014-04-30 09:28:30,923::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30207::DEBUG::2014-04-30 09:28:30,925::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30207::DEBUG::2014-04-30 09:28:30,926::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30207::INFO::2014-04-30 09:28:30,926::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30207::DEBUG::2014-04-30 09:28:30,926::task::1168::TaskManager.Task::(prepare) Task=`2959ae12-3cd9-4d35-845a-fa46a423bea2`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30207::DEBUG::2014-04-30 09:28:30,926::task::579::TaskManager.Task::(_updateState) Task=`2959ae12-3cd9-4d35-845a-fa46a423bea2`::moving from state preparing -> state finished
>Thread-30207::DEBUG::2014-04-30 09:28:30,927::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30207::DEBUG::2014-04-30 09:28:30,927::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30207::DEBUG::2014-04-30 09:28:30,927::task::974::TaskManager.Task::(_decref) Task=`2959ae12-3cd9-4d35-845a-fa46a423bea2`::ref 0 aborting False
>Thread-30209::DEBUG::2014-04-30 09:29:17,330::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5a37dfa0]
>Thread-30209::DEBUG::2014-04-30 09:29:17,331::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30209::DEBUG::2014-04-30 09:29:17,346::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30209::DEBUG::2014-04-30 09:29:17,383::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30210::DEBUG::2014-04-30 09:29:17,620::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5a37dfa0]
>Thread-30210::ERROR::2014-04-30 09:29:17,621::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30210::DEBUG::2014-04-30 09:29:17,622::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30211::DEBUG::2014-04-30 09:29:18,386::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [301ea09e]
>Thread-30211::DEBUG::2014-04-30 09:29:18,387::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30211::DEBUG::2014-04-30 09:29:18,400::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30211::DEBUG::2014-04-30 09:29:18,438::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30212::DEBUG::2014-04-30 09:29:18,663::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [301ea09e]
>Thread-30212::ERROR::2014-04-30 09:29:18,664::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30212::DEBUG::2014-04-30 09:29:18,664::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30214::DEBUG::2014-04-30 09:29:19,152::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [1d92fada]
>Thread-30214::DEBUG::2014-04-30 09:29:19,153::task::579::TaskManager.Task::(_updateState) Task=`e4aa9674-9c1e-4851-af8e-1c5a766490cc`::moving from state init -> state preparing
>Thread-30214::INFO::2014-04-30 09:29:19,153::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30214::DEBUG::2014-04-30 09:29:19,158::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30214::ERROR::2014-04-30 09:31:24,200::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30214::ERROR::2014-04-30 09:31:24,204::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30214::DEBUG::2014-04-30 09:31:24,205::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30214::DEBUG::2014-04-30 09:31:24,210::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30215::DEBUG::2014-04-30 09:32:19,324::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30215::DEBUG::2014-04-30 09:32:19,325::task::579::TaskManager.Task::(_updateState) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::moving from state init -> state preparing
>Thread-30215::INFO::2014-04-30 09:32:19,325::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30215::DEBUG::2014-04-30 09:32:19,325::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`fbd55d58-8e0d-4b44-b4b0-b5718ff62506`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30215::DEBUG::2014-04-30 09:32:19,326::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30215::DEBUG::2014-04-30 09:32:19,326::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30215::DEBUG::2014-04-30 09:32:19,326::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`fbd55d58-8e0d-4b44-b4b0-b5718ff62506`::Granted request
>Thread-30215::DEBUG::2014-04-30 09:32:19,327::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`a7bae677-690f-4449-b104-55b71679e866`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30215::DEBUG::2014-04-30 09:32:19,327::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30215::DEBUG::2014-04-30 09:32:19,327::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30215::DEBUG::2014-04-30 09:32:19,327::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`a7bae677-690f-4449-b104-55b71679e866`::Granted request
>Thread-30215::INFO::2014-04-30 09:32:19,328::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30215::DEBUG::2014-04-30 09:32:19,328::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30215::DEBUG::2014-04-30 09:32:19,329::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30215::DEBUG::2014-04-30 09:32:19,329::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30215::DEBUG::2014-04-30 09:32:19,329::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30215::DEBUG::2014-04-30 09:32:19,329::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30215::DEBUG::2014-04-30 09:32:19,329::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30215::DEBUG::2014-04-30 09:32:19,330::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30215::DEBUG::2014-04-30 09:32:19,330::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30215::DEBUG::2014-04-30 09:32:19,330::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30215::DEBUG::2014-04-30 09:32:19,330::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30215::DEBUG::2014-04-30 09:32:19,330::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30215::DEBUG::2014-04-30 09:32:19,353::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30215::DEBUG::2014-04-30 09:32:19,353::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30215::DEBUG::2014-04-30 09:32:19,356::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30215::DEBUG::2014-04-30 09:32:19,356::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30215::DEBUG::2014-04-30 09:32:19,356::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30215::DEBUG::2014-04-30 09:32:19,356::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30215::DEBUG::2014-04-30 09:32:19,357::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30215::DEBUG::2014-04-30 09:32:19,357::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30215::DEBUG::2014-04-30 09:32:19,357::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30215::DEBUG::2014-04-30 09:32:19,358::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30215::ERROR::2014-04-30 09:32:19,358::task::850::TaskManager.Task::(_setError) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30215::DEBUG::2014-04-30 09:32:19,359::task::869::TaskManager.Task::(_run) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::Task._run: cd8ddf83-2ddd-400a-bd62-c54a6667cc8c ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30215::DEBUG::2014-04-30 09:32:19,359::task::1194::TaskManager.Task::(stop) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::stopping in state preparing (force False)
>Thread-30215::DEBUG::2014-04-30 09:32:19,359::task::974::TaskManager.Task::(_decref) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::ref 1 aborting True
>Thread-30215::INFO::2014-04-30 09:32:19,360::task::1151::TaskManager.Task::(prepare) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30215::DEBUG::2014-04-30 09:32:19,360::task::1156::TaskManager.Task::(prepare) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30215::DEBUG::2014-04-30 09:32:19,360::task::974::TaskManager.Task::(_decref) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::ref 0 aborting True
>Thread-30215::DEBUG::2014-04-30 09:32:19,361::task::909::TaskManager.Task::(_doAbort) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::Task._doAbort: force False
>Thread-30215::DEBUG::2014-04-30 09:32:19,361::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30215::DEBUG::2014-04-30 09:32:19,361::task::579::TaskManager.Task::(_updateState) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::moving from state preparing -> state aborting
>Thread-30215::DEBUG::2014-04-30 09:32:19,361::task::534::TaskManager.Task::(__state_aborting) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::_aborting: recover policy none
>Thread-30215::DEBUG::2014-04-30 09:32:19,361::task::579::TaskManager.Task::(_updateState) Task=`cd8ddf83-2ddd-400a-bd62-c54a6667cc8c`::moving from state aborting -> state failed
>Thread-30215::DEBUG::2014-04-30 09:32:19,362::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30215::DEBUG::2014-04-30 09:32:19,362::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30215::ERROR::2014-04-30 09:32:19,362::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30214::ERROR::2014-04-30 09:33:29,250::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30214::ERROR::2014-04-30 09:33:29,251::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30214::DEBUG::2014-04-30 09:33:29,251::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30214::DEBUG::2014-04-30 09:33:29,255::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30214::DEBUG::2014-04-30 09:33:29,257::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30214::DEBUG::2014-04-30 09:33:29,257::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30214::INFO::2014-04-30 09:33:29,257::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30214::DEBUG::2014-04-30 09:33:29,257::task::1168::TaskManager.Task::(prepare) Task=`e4aa9674-9c1e-4851-af8e-1c5a766490cc`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30214::DEBUG::2014-04-30 09:33:29,258::task::579::TaskManager.Task::(_updateState) Task=`e4aa9674-9c1e-4851-af8e-1c5a766490cc`::moving from state preparing -> state finished
>Thread-30214::DEBUG::2014-04-30 09:33:29,258::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30214::DEBUG::2014-04-30 09:33:29,258::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30214::DEBUG::2014-04-30 09:33:29,258::task::974::TaskManager.Task::(_decref) Task=`e4aa9674-9c1e-4851-af8e-1c5a766490cc`::ref 0 aborting False
>Thread-30216::DEBUG::2014-04-30 09:34:17,323::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2dd94949]
>Thread-30216::DEBUG::2014-04-30 09:34:17,324::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30216::DEBUG::2014-04-30 09:34:17,338::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30216::DEBUG::2014-04-30 09:34:17,376::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30217::DEBUG::2014-04-30 09:34:17,619::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2dd94949]
>Thread-30217::ERROR::2014-04-30 09:34:17,619::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30217::DEBUG::2014-04-30 09:34:17,620::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30218::DEBUG::2014-04-30 09:34:19,704::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [25458d88]
>Thread-30218::DEBUG::2014-04-30 09:34:19,705::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30218::DEBUG::2014-04-30 09:34:19,719::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30218::DEBUG::2014-04-30 09:34:19,756::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30219::DEBUG::2014-04-30 09:34:19,991::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [25458d88]
>Thread-30219::ERROR::2014-04-30 09:34:19,991::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30219::DEBUG::2014-04-30 09:34:19,992::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30221::DEBUG::2014-04-30 09:34:20,463::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [3211229c]
>Thread-30221::DEBUG::2014-04-30 09:34:20,463::task::579::TaskManager.Task::(_updateState) Task=`d6ec1599-5db2-46ab-be37-840bc546625d`::moving from state init -> state preparing
>Thread-30221::INFO::2014-04-30 09:34:20,464::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30221::DEBUG::2014-04-30 09:34:20,467::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30221::ERROR::2014-04-30 09:36:25,508::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30221::ERROR::2014-04-30 09:36:25,509::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30221::DEBUG::2014-04-30 09:36:25,509::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30221::DEBUG::2014-04-30 09:36:25,514::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30222::DEBUG::2014-04-30 09:37:20,646::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30222::DEBUG::2014-04-30 09:37:20,647::task::579::TaskManager.Task::(_updateState) Task=`65cfb374-bada-412b-9f74-11796b80a980`::moving from state init -> state preparing
>Thread-30222::INFO::2014-04-30 09:37:20,647::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30222::DEBUG::2014-04-30 09:37:20,648::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`12372762-b1c9-4a93-a013-d0c511b9aa79`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30222::DEBUG::2014-04-30 09:37:20,648::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30222::DEBUG::2014-04-30 09:37:20,648::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30222::DEBUG::2014-04-30 09:37:20,649::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`12372762-b1c9-4a93-a013-d0c511b9aa79`::Granted request
>Thread-30222::DEBUG::2014-04-30 09:37:20,649::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`e8cc60c3-abd3-4699-9eaf-3809063f0807`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30222::DEBUG::2014-04-30 09:37:20,649::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30222::DEBUG::2014-04-30 09:37:20,650::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30222::DEBUG::2014-04-30 09:37:20,650::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`e8cc60c3-abd3-4699-9eaf-3809063f0807`::Granted request
>Thread-30222::INFO::2014-04-30 09:37:20,650::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30222::DEBUG::2014-04-30 09:37:20,651::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30222::DEBUG::2014-04-30 09:37:20,651::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30222::DEBUG::2014-04-30 09:37:20,651::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30222::DEBUG::2014-04-30 09:37:20,651::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30222::DEBUG::2014-04-30 09:37:20,652::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30222::DEBUG::2014-04-30 09:37:20,652::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30222::DEBUG::2014-04-30 09:37:20,652::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30222::DEBUG::2014-04-30 09:37:20,652::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30222::DEBUG::2014-04-30 09:37:20,652::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30222::DEBUG::2014-04-30 09:37:20,653::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30222::DEBUG::2014-04-30 09:37:20,653::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30222::DEBUG::2014-04-30 09:37:20,675::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30222::DEBUG::2014-04-30 09:37:20,676::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30222::DEBUG::2014-04-30 09:37:20,676::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30222::DEBUG::2014-04-30 09:37:20,677::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30222::DEBUG::2014-04-30 09:37:20,677::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30222::DEBUG::2014-04-30 09:37:20,677::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30222::DEBUG::2014-04-30 09:37:20,678::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30222::DEBUG::2014-04-30 09:37:20,678::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30222::DEBUG::2014-04-30 09:37:20,678::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30222::DEBUG::2014-04-30 09:37:20,678::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30222::ERROR::2014-04-30 09:37:20,679::task::850::TaskManager.Task::(_setError) Task=`65cfb374-bada-412b-9f74-11796b80a980`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30222::DEBUG::2014-04-30 09:37:20,680::task::869::TaskManager.Task::(_run) Task=`65cfb374-bada-412b-9f74-11796b80a980`::Task._run: 65cfb374-bada-412b-9f74-11796b80a980 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30222::DEBUG::2014-04-30 09:37:20,680::task::1194::TaskManager.Task::(stop) Task=`65cfb374-bada-412b-9f74-11796b80a980`::stopping in state preparing (force False)
>Thread-30222::DEBUG::2014-04-30 09:37:20,680::task::974::TaskManager.Task::(_decref) Task=`65cfb374-bada-412b-9f74-11796b80a980`::ref 1 aborting True
>Thread-30222::INFO::2014-04-30 09:37:20,681::task::1151::TaskManager.Task::(prepare) Task=`65cfb374-bada-412b-9f74-11796b80a980`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30222::DEBUG::2014-04-30 09:37:20,681::task::1156::TaskManager.Task::(prepare) Task=`65cfb374-bada-412b-9f74-11796b80a980`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30222::DEBUG::2014-04-30 09:37:20,681::task::974::TaskManager.Task::(_decref) Task=`65cfb374-bada-412b-9f74-11796b80a980`::ref 0 aborting True
>Thread-30222::DEBUG::2014-04-30 09:37:20,681::task::909::TaskManager.Task::(_doAbort) Task=`65cfb374-bada-412b-9f74-11796b80a980`::Task._doAbort: force False
>Thread-30222::DEBUG::2014-04-30 09:37:20,682::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30222::DEBUG::2014-04-30 09:37:20,682::task::579::TaskManager.Task::(_updateState) Task=`65cfb374-bada-412b-9f74-11796b80a980`::moving from state preparing -> state aborting
>Thread-30222::DEBUG::2014-04-30 09:37:20,682::task::534::TaskManager.Task::(__state_aborting) Task=`65cfb374-bada-412b-9f74-11796b80a980`::_aborting: recover policy none
>Thread-30222::DEBUG::2014-04-30 09:37:20,682::task::579::TaskManager.Task::(_updateState) Task=`65cfb374-bada-412b-9f74-11796b80a980`::moving from state aborting -> state failed
>Thread-30222::DEBUG::2014-04-30 09:37:20,682::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30222::DEBUG::2014-04-30 09:37:20,683::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30222::ERROR::2014-04-30 09:37:20,683::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30221::ERROR::2014-04-30 09:38:30,555::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30221::ERROR::2014-04-30 09:38:30,556::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30221::DEBUG::2014-04-30 09:38:30,556::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30221::DEBUG::2014-04-30 09:38:30,560::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30221::DEBUG::2014-04-30 09:38:30,561::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30221::DEBUG::2014-04-30 09:38:30,562::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30221::INFO::2014-04-30 09:38:30,562::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30221::DEBUG::2014-04-30 09:38:30,562::task::1168::TaskManager.Task::(prepare) Task=`d6ec1599-5db2-46ab-be37-840bc546625d`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30221::DEBUG::2014-04-30 09:38:30,563::task::579::TaskManager.Task::(_updateState) Task=`d6ec1599-5db2-46ab-be37-840bc546625d`::moving from state preparing -> state finished
>Thread-30221::DEBUG::2014-04-30 09:38:30,563::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30221::DEBUG::2014-04-30 09:38:30,563::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30221::DEBUG::2014-04-30 09:38:30,563::task::974::TaskManager.Task::(_decref) Task=`d6ec1599-5db2-46ab-be37-840bc546625d`::ref 0 aborting False
>Thread-30223::DEBUG::2014-04-30 09:39:17,314::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5e4b3e2]
>Thread-30223::DEBUG::2014-04-30 09:39:17,315::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30223::DEBUG::2014-04-30 09:39:17,329::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30223::DEBUG::2014-04-30 09:39:17,366::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30224::DEBUG::2014-04-30 09:39:17,592::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5e4b3e2]
>Thread-30224::ERROR::2014-04-30 09:39:17,593::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30224::DEBUG::2014-04-30 09:39:17,593::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30225::DEBUG::2014-04-30 09:39:18,022::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2231d6f0]
>Thread-30225::DEBUG::2014-04-30 09:39:18,023::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30225::DEBUG::2014-04-30 09:39:18,037::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30225::DEBUG::2014-04-30 09:39:18,074::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30226::DEBUG::2014-04-30 09:39:18,315::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2231d6f0]
>Thread-30226::ERROR::2014-04-30 09:39:18,316::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30226::DEBUG::2014-04-30 09:39:18,317::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30228::DEBUG::2014-04-30 09:39:18,818::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [47fec6e7]
>Thread-30228::DEBUG::2014-04-30 09:39:18,819::task::579::TaskManager.Task::(_updateState) Task=`48881fd5-fa7d-44d5-ad0a-d3a286ae61fb`::moving from state init -> state preparing
>Thread-30228::INFO::2014-04-30 09:39:18,820::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30228::DEBUG::2014-04-30 09:39:18,824::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30228::ERROR::2014-04-30 09:41:23,864::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30228::ERROR::2014-04-30 09:41:23,865::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30228::DEBUG::2014-04-30 09:41:23,866::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30228::DEBUG::2014-04-30 09:41:23,871::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30229::DEBUG::2014-04-30 09:42:18,981::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30229::DEBUG::2014-04-30 09:42:18,982::task::579::TaskManager.Task::(_updateState) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::moving from state init -> state preparing
>Thread-30229::INFO::2014-04-30 09:42:18,982::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30229::DEBUG::2014-04-30 09:42:18,983::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`735defcf-67be-4858-be34-35566ceca259`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30229::DEBUG::2014-04-30 09:42:18,983::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30229::DEBUG::2014-04-30 09:42:18,984::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30229::DEBUG::2014-04-30 09:42:18,984::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`735defcf-67be-4858-be34-35566ceca259`::Granted request
>Thread-30229::DEBUG::2014-04-30 09:42:18,985::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`d840c832-0b4d-4c12-bebb-c20db87c88ec`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30229::DEBUG::2014-04-30 09:42:18,985::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30229::DEBUG::2014-04-30 09:42:18,985::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30229::DEBUG::2014-04-30 09:42:18,985::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`d840c832-0b4d-4c12-bebb-c20db87c88ec`::Granted request
>Thread-30229::INFO::2014-04-30 09:42:18,986::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30229::DEBUG::2014-04-30 09:42:18,986::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30229::DEBUG::2014-04-30 09:42:18,987::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30229::DEBUG::2014-04-30 09:42:18,987::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30229::DEBUG::2014-04-30 09:42:18,987::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30229::DEBUG::2014-04-30 09:42:18,988::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30229::DEBUG::2014-04-30 09:42:18,988::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30229::DEBUG::2014-04-30 09:42:18,988::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30229::DEBUG::2014-04-30 09:42:18,988::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30229::DEBUG::2014-04-30 09:42:18,989::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30229::DEBUG::2014-04-30 09:42:18,989::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30229::DEBUG::2014-04-30 09:42:18,989::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30229::DEBUG::2014-04-30 09:42:19,011::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30229::DEBUG::2014-04-30 09:42:19,012::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30229::DEBUG::2014-04-30 09:42:19,012::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30229::DEBUG::2014-04-30 09:42:19,013::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30229::DEBUG::2014-04-30 09:42:19,013::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30229::DEBUG::2014-04-30 09:42:19,013::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30229::DEBUG::2014-04-30 09:42:19,014::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30229::DEBUG::2014-04-30 09:42:19,014::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30229::DEBUG::2014-04-30 09:42:19,014::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30229::DEBUG::2014-04-30 09:42:19,014::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30229::ERROR::2014-04-30 09:42:19,015::task::850::TaskManager.Task::(_setError) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30229::DEBUG::2014-04-30 09:42:19,016::task::869::TaskManager.Task::(_run) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::Task._run: 2e66ad9d-1260-4f86-881e-9ccab5e5e540 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30229::DEBUG::2014-04-30 09:42:19,016::task::1194::TaskManager.Task::(stop) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::stopping in state preparing (force False)
>Thread-30229::DEBUG::2014-04-30 09:42:19,016::task::974::TaskManager.Task::(_decref) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::ref 1 aborting True
>Thread-30229::INFO::2014-04-30 09:42:19,017::task::1151::TaskManager.Task::(prepare) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30229::DEBUG::2014-04-30 09:42:19,017::task::1156::TaskManager.Task::(prepare) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30229::DEBUG::2014-04-30 09:42:19,017::task::974::TaskManager.Task::(_decref) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::ref 0 aborting True
>Thread-30229::DEBUG::2014-04-30 09:42:19,017::task::909::TaskManager.Task::(_doAbort) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::Task._doAbort: force False
>Thread-30229::DEBUG::2014-04-30 09:42:19,018::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30229::DEBUG::2014-04-30 09:42:19,018::task::579::TaskManager.Task::(_updateState) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::moving from state preparing -> state aborting
>Thread-30229::DEBUG::2014-04-30 09:42:19,018::task::534::TaskManager.Task::(__state_aborting) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::_aborting: recover policy none
>Thread-30229::DEBUG::2014-04-30 09:42:19,018::task::579::TaskManager.Task::(_updateState) Task=`2e66ad9d-1260-4f86-881e-9ccab5e5e540`::moving from state aborting -> state failed
>Thread-30229::DEBUG::2014-04-30 09:42:19,018::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30229::DEBUG::2014-04-30 09:42:19,019::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30229::ERROR::2014-04-30 09:42:19,019::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30228::ERROR::2014-04-30 09:43:28,912::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30228::ERROR::2014-04-30 09:43:28,913::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30228::DEBUG::2014-04-30 09:43:28,914::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30228::DEBUG::2014-04-30 09:43:28,918::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30228::DEBUG::2014-04-30 09:43:28,919::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30228::DEBUG::2014-04-30 09:43:28,919::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30228::INFO::2014-04-30 09:43:28,920::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30228::DEBUG::2014-04-30 09:43:28,920::task::1168::TaskManager.Task::(prepare) Task=`48881fd5-fa7d-44d5-ad0a-d3a286ae61fb`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30228::DEBUG::2014-04-30 09:43:28,920::task::579::TaskManager.Task::(_updateState) Task=`48881fd5-fa7d-44d5-ad0a-d3a286ae61fb`::moving from state preparing -> state finished
>Thread-30228::DEBUG::2014-04-30 09:43:28,921::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30228::DEBUG::2014-04-30 09:43:28,921::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30228::DEBUG::2014-04-30 09:43:28,921::task::974::TaskManager.Task::(_decref) Task=`48881fd5-fa7d-44d5-ad0a-d3a286ae61fb`::ref 0 aborting False
>Thread-30230::DEBUG::2014-04-30 09:44:17,314::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [75ab5f8e]
>Thread-30230::DEBUG::2014-04-30 09:44:17,315::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30230::DEBUG::2014-04-30 09:44:17,328::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30230::DEBUG::2014-04-30 09:44:17,365::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30231::DEBUG::2014-04-30 09:44:17,590::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [75ab5f8e]
>Thread-30231::ERROR::2014-04-30 09:44:17,591::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30231::DEBUG::2014-04-30 09:44:17,591::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30232::DEBUG::2014-04-30 09:44:19,342::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [29ea26d]
>Thread-30232::DEBUG::2014-04-30 09:44:19,343::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30232::DEBUG::2014-04-30 09:44:19,357::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30232::DEBUG::2014-04-30 09:44:19,409::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30233::DEBUG::2014-04-30 09:44:19,635::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [29ea26d]
>Thread-30233::ERROR::2014-04-30 09:44:19,636::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30233::DEBUG::2014-04-30 09:44:19,637::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30235::DEBUG::2014-04-30 09:44:20,102::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [608ec64d]
>Thread-30235::DEBUG::2014-04-30 09:44:20,103::task::579::TaskManager.Task::(_updateState) Task=`d3ef3430-8d26-4f14-a629-3f081f1a6b34`::moving from state init -> state preparing
>Thread-30235::INFO::2014-04-30 09:44:20,103::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30235::DEBUG::2014-04-30 09:44:20,108::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30235::ERROR::2014-04-30 09:46:25,152::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30235::ERROR::2014-04-30 09:46:25,153::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30235::DEBUG::2014-04-30 09:46:25,154::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30235::DEBUG::2014-04-30 09:46:25,158::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30236::DEBUG::2014-04-30 09:47:20,288::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30236::DEBUG::2014-04-30 09:47:20,289::task::579::TaskManager.Task::(_updateState) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::moving from state init -> state preparing
>Thread-30236::INFO::2014-04-30 09:47:20,289::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30236::DEBUG::2014-04-30 09:47:20,290::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e7efdac8-012c-4886-ae94-95b5aa360d40`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30236::DEBUG::2014-04-30 09:47:20,290::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30236::DEBUG::2014-04-30 09:47:20,290::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30236::DEBUG::2014-04-30 09:47:20,290::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e7efdac8-012c-4886-ae94-95b5aa360d40`::Granted request
>Thread-30236::DEBUG::2014-04-30 09:47:20,291::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`7a780f4d-1e72-44ef-bf39-ca7a2b38af30`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30236::DEBUG::2014-04-30 09:47:20,291::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30236::DEBUG::2014-04-30 09:47:20,291::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30236::DEBUG::2014-04-30 09:47:20,292::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`7a780f4d-1e72-44ef-bf39-ca7a2b38af30`::Granted request
>Thread-30236::INFO::2014-04-30 09:47:20,292::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30236::DEBUG::2014-04-30 09:47:20,293::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30236::DEBUG::2014-04-30 09:47:20,293::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30236::DEBUG::2014-04-30 09:47:20,293::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30236::DEBUG::2014-04-30 09:47:20,293::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30236::DEBUG::2014-04-30 09:47:20,293::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30236::DEBUG::2014-04-30 09:47:20,294::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30236::DEBUG::2014-04-30 09:47:20,294::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30236::DEBUG::2014-04-30 09:47:20,294::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30236::DEBUG::2014-04-30 09:47:20,294::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30236::DEBUG::2014-04-30 09:47:20,294::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30236::DEBUG::2014-04-30 09:47:20,295::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30236::DEBUG::2014-04-30 09:47:20,318::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30236::DEBUG::2014-04-30 09:47:20,319::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30236::DEBUG::2014-04-30 09:47:20,319::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30236::DEBUG::2014-04-30 09:47:20,320::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30236::DEBUG::2014-04-30 09:47:20,320::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30236::DEBUG::2014-04-30 09:47:20,320::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30236::DEBUG::2014-04-30 09:47:20,321::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30236::DEBUG::2014-04-30 09:47:20,321::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30236::DEBUG::2014-04-30 09:47:20,321::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30236::DEBUG::2014-04-30 09:47:20,321::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30236::ERROR::2014-04-30 09:47:20,322::task::850::TaskManager.Task::(_setError) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30236::DEBUG::2014-04-30 09:47:20,324::task::869::TaskManager.Task::(_run) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::Task._run: 322463ee-6dff-44d0-9736-703e221b10a5 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30236::DEBUG::2014-04-30 09:47:20,324::task::1194::TaskManager.Task::(stop) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::stopping in state preparing (force False)
>Thread-30236::DEBUG::2014-04-30 09:47:20,324::task::974::TaskManager.Task::(_decref) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::ref 1 aborting True
>Thread-30236::INFO::2014-04-30 09:47:20,324::task::1151::TaskManager.Task::(prepare) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30236::DEBUG::2014-04-30 09:47:20,325::task::1156::TaskManager.Task::(prepare) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30236::DEBUG::2014-04-30 09:47:20,325::task::974::TaskManager.Task::(_decref) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::ref 0 aborting True
>Thread-30236::DEBUG::2014-04-30 09:47:20,325::task::909::TaskManager.Task::(_doAbort) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::Task._doAbort: force False
>Thread-30236::DEBUG::2014-04-30 09:47:20,325::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30236::DEBUG::2014-04-30 09:47:20,326::task::579::TaskManager.Task::(_updateState) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::moving from state preparing -> state aborting
>Thread-30236::DEBUG::2014-04-30 09:47:20,326::task::534::TaskManager.Task::(__state_aborting) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::_aborting: recover policy none
>Thread-30236::DEBUG::2014-04-30 09:47:20,326::task::579::TaskManager.Task::(_updateState) Task=`322463ee-6dff-44d0-9736-703e221b10a5`::moving from state aborting -> state failed
>Thread-30236::DEBUG::2014-04-30 09:47:20,326::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30236::DEBUG::2014-04-30 09:47:20,326::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30236::ERROR::2014-04-30 09:47:20,327::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30235::ERROR::2014-04-30 09:48:30,203::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30235::ERROR::2014-04-30 09:48:30,204::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30235::DEBUG::2014-04-30 09:48:30,205::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30235::DEBUG::2014-04-30 09:48:30,210::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30235::DEBUG::2014-04-30 09:48:30,211::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30235::DEBUG::2014-04-30 09:48:30,211::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30235::INFO::2014-04-30 09:48:30,212::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30235::DEBUG::2014-04-30 09:48:30,212::task::1168::TaskManager.Task::(prepare) Task=`d3ef3430-8d26-4f14-a629-3f081f1a6b34`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30235::DEBUG::2014-04-30 09:48:30,212::task::579::TaskManager.Task::(_updateState) Task=`d3ef3430-8d26-4f14-a629-3f081f1a6b34`::moving from state preparing -> state finished
>Thread-30235::DEBUG::2014-04-30 09:48:30,212::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30235::DEBUG::2014-04-30 09:48:30,213::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30235::DEBUG::2014-04-30 09:48:30,213::task::974::TaskManager.Task::(_decref) Task=`d3ef3430-8d26-4f14-a629-3f081f1a6b34`::ref 0 aborting False
>Thread-30237::DEBUG::2014-04-30 09:49:17,304::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5a4c26da]
>Thread-30237::DEBUG::2014-04-30 09:49:17,305::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30237::DEBUG::2014-04-30 09:49:17,321::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30237::DEBUG::2014-04-30 09:49:17,364::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30238::DEBUG::2014-04-30 09:49:17,593::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5a4c26da]
>Thread-30238::ERROR::2014-04-30 09:49:17,594::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30238::DEBUG::2014-04-30 09:49:17,594::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30239::DEBUG::2014-04-30 09:49:20,674::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [11eeed00]
>Thread-30239::DEBUG::2014-04-30 09:49:20,675::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30239::DEBUG::2014-04-30 09:49:20,688::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30239::DEBUG::2014-04-30 09:49:20,726::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30240::DEBUG::2014-04-30 09:49:20,951::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [11eeed00]
>Thread-30240::ERROR::2014-04-30 09:49:20,951::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30240::DEBUG::2014-04-30 09:49:20,952::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30242::DEBUG::2014-04-30 09:49:21,421::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [2da7b7a]
>Thread-30242::DEBUG::2014-04-30 09:49:21,422::task::579::TaskManager.Task::(_updateState) Task=`f7caa3e3-0f69-44da-8236-9b46b3954792`::moving from state init -> state preparing
>Thread-30242::INFO::2014-04-30 09:49:21,423::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30242::DEBUG::2014-04-30 09:49:21,428::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30242::ERROR::2014-04-30 09:51:26,471::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30242::ERROR::2014-04-30 09:51:26,473::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30242::DEBUG::2014-04-30 09:51:26,473::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30242::DEBUG::2014-04-30 09:51:26,478::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30243::DEBUG::2014-04-30 09:52:21,607::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30243::DEBUG::2014-04-30 09:52:21,608::task::579::TaskManager.Task::(_updateState) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::moving from state init -> state preparing
>Thread-30243::INFO::2014-04-30 09:52:21,609::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30243::DEBUG::2014-04-30 09:52:21,609::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`16fd6cad-2fbc-49a8-8a22-adcc7e9faf59`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30243::DEBUG::2014-04-30 09:52:21,609::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30243::DEBUG::2014-04-30 09:52:21,610::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30243::DEBUG::2014-04-30 09:52:21,610::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`16fd6cad-2fbc-49a8-8a22-adcc7e9faf59`::Granted request
>Thread-30243::DEBUG::2014-04-30 09:52:21,610::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`5e530a06-6ac8-47ed-aafe-3db3be2b0652`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30243::DEBUG::2014-04-30 09:52:21,611::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30243::DEBUG::2014-04-30 09:52:21,611::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30243::DEBUG::2014-04-30 09:52:21,611::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`5e530a06-6ac8-47ed-aafe-3db3be2b0652`::Granted request
>Thread-30243::INFO::2014-04-30 09:52:21,612::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30243::DEBUG::2014-04-30 09:52:21,612::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30243::DEBUG::2014-04-30 09:52:21,612::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30243::DEBUG::2014-04-30 09:52:21,613::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30243::DEBUG::2014-04-30 09:52:21,613::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30243::DEBUG::2014-04-30 09:52:21,613::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30243::DEBUG::2014-04-30 09:52:21,613::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30243::DEBUG::2014-04-30 09:52:21,614::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30243::DEBUG::2014-04-30 09:52:21,614::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30243::DEBUG::2014-04-30 09:52:21,614::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30243::DEBUG::2014-04-30 09:52:21,614::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30243::DEBUG::2014-04-30 09:52:21,614::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30243::DEBUG::2014-04-30 09:52:21,636::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30243::DEBUG::2014-04-30 09:52:21,637::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30243::DEBUG::2014-04-30 09:52:21,638::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30243::DEBUG::2014-04-30 09:52:21,638::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30243::DEBUG::2014-04-30 09:52:21,638::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30243::DEBUG::2014-04-30 09:52:21,639::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30243::DEBUG::2014-04-30 09:52:21,639::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30243::DEBUG::2014-04-30 09:52:21,639::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30243::DEBUG::2014-04-30 09:52:21,640::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30243::DEBUG::2014-04-30 09:52:21,640::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30243::ERROR::2014-04-30 09:52:21,640::task::850::TaskManager.Task::(_setError) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30243::DEBUG::2014-04-30 09:52:21,641::task::869::TaskManager.Task::(_run) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::Task._run: 89530244-c3ce-4fa6-bfbf-bd8af9e77082 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30243::DEBUG::2014-04-30 09:52:21,641::task::1194::TaskManager.Task::(stop) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::stopping in state preparing (force False)
>Thread-30243::DEBUG::2014-04-30 09:52:21,642::task::974::TaskManager.Task::(_decref) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::ref 1 aborting True
>Thread-30243::INFO::2014-04-30 09:52:21,642::task::1151::TaskManager.Task::(prepare) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30243::DEBUG::2014-04-30 09:52:21,642::task::1156::TaskManager.Task::(prepare) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30243::DEBUG::2014-04-30 09:52:21,643::task::974::TaskManager.Task::(_decref) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::ref 0 aborting True
>Thread-30243::DEBUG::2014-04-30 09:52:21,643::task::909::TaskManager.Task::(_doAbort) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::Task._doAbort: force False
>Thread-30243::DEBUG::2014-04-30 09:52:21,643::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30243::DEBUG::2014-04-30 09:52:21,643::task::579::TaskManager.Task::(_updateState) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::moving from state preparing -> state aborting
>Thread-30243::DEBUG::2014-04-30 09:52:21,643::task::534::TaskManager.Task::(__state_aborting) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::_aborting: recover policy none
>Thread-30243::DEBUG::2014-04-30 09:52:21,644::task::579::TaskManager.Task::(_updateState) Task=`89530244-c3ce-4fa6-bfbf-bd8af9e77082`::moving from state aborting -> state failed
>Thread-30243::DEBUG::2014-04-30 09:52:21,644::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30243::DEBUG::2014-04-30 09:52:21,644::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30243::ERROR::2014-04-30 09:52:21,644::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30242::ERROR::2014-04-30 09:53:31,519::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30242::ERROR::2014-04-30 09:53:31,520::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30242::DEBUG::2014-04-30 09:53:31,520::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30242::DEBUG::2014-04-30 09:53:31,525::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30242::DEBUG::2014-04-30 09:53:31,527::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30242::DEBUG::2014-04-30 09:53:31,527::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30242::INFO::2014-04-30 09:53:31,528::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30242::DEBUG::2014-04-30 09:53:31,528::task::1168::TaskManager.Task::(prepare) Task=`f7caa3e3-0f69-44da-8236-9b46b3954792`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30242::DEBUG::2014-04-30 09:53:31,528::task::579::TaskManager.Task::(_updateState) Task=`f7caa3e3-0f69-44da-8236-9b46b3954792`::moving from state preparing -> state finished
>Thread-30242::DEBUG::2014-04-30 09:53:31,528::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30242::DEBUG::2014-04-30 09:53:31,529::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30242::DEBUG::2014-04-30 09:53:31,529::task::974::TaskManager.Task::(_decref) Task=`f7caa3e3-0f69-44da-8236-9b46b3954792`::ref 0 aborting False
>Thread-30244::DEBUG::2014-04-30 09:54:17,301::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2387c207]
>Thread-30244::DEBUG::2014-04-30 09:54:17,302::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30244::DEBUG::2014-04-30 09:54:17,317::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30244::DEBUG::2014-04-30 09:54:17,354::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30245::DEBUG::2014-04-30 09:54:17,580::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2387c207]
>Thread-30245::ERROR::2014-04-30 09:54:17,581::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30245::DEBUG::2014-04-30 09:54:17,582::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30246::DEBUG::2014-04-30 09:54:18,964::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [4680f5a6]
>Thread-30246::DEBUG::2014-04-30 09:54:18,965::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30246::DEBUG::2014-04-30 09:54:18,978::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30246::DEBUG::2014-04-30 09:54:19,016::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30247::DEBUG::2014-04-30 09:54:19,243::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [4680f5a6]
>Thread-30247::ERROR::2014-04-30 09:54:19,244::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30247::DEBUG::2014-04-30 09:54:19,244::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30249::DEBUG::2014-04-30 09:54:19,707::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [64e8eaac]
>Thread-30249::DEBUG::2014-04-30 09:54:19,708::task::579::TaskManager.Task::(_updateState) Task=`ef8dcd38-7c03-400c-8c2a-d223121aab25`::moving from state init -> state preparing
>Thread-30249::INFO::2014-04-30 09:54:19,708::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30249::DEBUG::2014-04-30 09:54:19,713::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30249::ERROR::2014-04-30 09:56:24,755::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30249::ERROR::2014-04-30 09:56:24,763::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30249::DEBUG::2014-04-30 09:56:24,764::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30249::DEBUG::2014-04-30 09:56:24,769::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30250::DEBUG::2014-04-30 09:57:19,885::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30250::DEBUG::2014-04-30 09:57:19,886::task::579::TaskManager.Task::(_updateState) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::moving from state init -> state preparing
>Thread-30250::INFO::2014-04-30 09:57:19,886::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30250::DEBUG::2014-04-30 09:57:19,887::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`709ca4ba-ff46-4a78-8bf0-e9862b14bce8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30250::DEBUG::2014-04-30 09:57:19,887::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30250::DEBUG::2014-04-30 09:57:19,888::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30250::DEBUG::2014-04-30 09:57:19,888::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`709ca4ba-ff46-4a78-8bf0-e9862b14bce8`::Granted request
>Thread-30250::DEBUG::2014-04-30 09:57:19,889::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`27ae6043-7f31-46a2-b9d6-a9d3ed881a2f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30250::DEBUG::2014-04-30 09:57:19,889::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30250::DEBUG::2014-04-30 09:57:19,889::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30250::DEBUG::2014-04-30 09:57:19,889::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`27ae6043-7f31-46a2-b9d6-a9d3ed881a2f`::Granted request
>Thread-30250::INFO::2014-04-30 09:57:19,890::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30250::DEBUG::2014-04-30 09:57:19,891::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30250::DEBUG::2014-04-30 09:57:19,891::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30250::DEBUG::2014-04-30 09:57:19,891::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30250::DEBUG::2014-04-30 09:57:19,891::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30250::DEBUG::2014-04-30 09:57:19,892::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30250::DEBUG::2014-04-30 09:57:19,892::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30250::DEBUG::2014-04-30 09:57:19,892::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30250::DEBUG::2014-04-30 09:57:19,892::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30250::DEBUG::2014-04-30 09:57:19,893::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30250::DEBUG::2014-04-30 09:57:19,893::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30250::DEBUG::2014-04-30 09:57:19,893::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30250::DEBUG::2014-04-30 09:57:19,915::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30250::DEBUG::2014-04-30 09:57:19,915::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30250::DEBUG::2014-04-30 09:57:19,916::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30250::DEBUG::2014-04-30 09:57:19,917::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30250::DEBUG::2014-04-30 09:57:19,917::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30250::DEBUG::2014-04-30 09:57:19,917::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30250::DEBUG::2014-04-30 09:57:19,918::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30250::DEBUG::2014-04-30 09:57:19,918::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30250::DEBUG::2014-04-30 09:57:19,918::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30250::DEBUG::2014-04-30 09:57:19,918::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30250::ERROR::2014-04-30 09:57:19,919::task::850::TaskManager.Task::(_setError) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30250::DEBUG::2014-04-30 09:57:19,920::task::869::TaskManager.Task::(_run) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::Task._run: 366f7f5d-7a14-45cd-8128-16ba0c0359bf ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30250::DEBUG::2014-04-30 09:57:19,920::task::1194::TaskManager.Task::(stop) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::stopping in state preparing (force False)
>Thread-30250::DEBUG::2014-04-30 09:57:19,920::task::974::TaskManager.Task::(_decref) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::ref 1 aborting True
>Thread-30250::INFO::2014-04-30 09:57:19,921::task::1151::TaskManager.Task::(prepare) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30250::DEBUG::2014-04-30 09:57:19,921::task::1156::TaskManager.Task::(prepare) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30250::DEBUG::2014-04-30 09:57:19,921::task::974::TaskManager.Task::(_decref) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::ref 0 aborting True
>Thread-30250::DEBUG::2014-04-30 09:57:19,921::task::909::TaskManager.Task::(_doAbort) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::Task._doAbort: force False
>Thread-30250::DEBUG::2014-04-30 09:57:19,921::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30250::DEBUG::2014-04-30 09:57:19,922::task::579::TaskManager.Task::(_updateState) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::moving from state preparing -> state aborting
>Thread-30250::DEBUG::2014-04-30 09:57:19,922::task::534::TaskManager.Task::(__state_aborting) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::_aborting: recover policy none
>Thread-30250::DEBUG::2014-04-30 09:57:19,922::task::579::TaskManager.Task::(_updateState) Task=`366f7f5d-7a14-45cd-8128-16ba0c0359bf`::moving from state aborting -> state failed
>Thread-30250::DEBUG::2014-04-30 09:57:19,922::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30250::DEBUG::2014-04-30 09:57:19,923::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30250::ERROR::2014-04-30 09:57:19,923::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30249::ERROR::2014-04-30 09:58:29,810::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30249::ERROR::2014-04-30 09:58:29,811::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30249::DEBUG::2014-04-30 09:58:29,811::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30249::DEBUG::2014-04-30 09:58:29,815::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30249::DEBUG::2014-04-30 09:58:29,817::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30249::DEBUG::2014-04-30 09:58:29,817::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30249::INFO::2014-04-30 09:58:29,818::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30249::DEBUG::2014-04-30 09:58:29,818::task::1168::TaskManager.Task::(prepare) Task=`ef8dcd38-7c03-400c-8c2a-d223121aab25`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30249::DEBUG::2014-04-30 09:58:29,818::task::579::TaskManager.Task::(_updateState) Task=`ef8dcd38-7c03-400c-8c2a-d223121aab25`::moving from state preparing -> state finished
>Thread-30249::DEBUG::2014-04-30 09:58:29,818::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30249::DEBUG::2014-04-30 09:58:29,819::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30249::DEBUG::2014-04-30 09:58:29,819::task::974::TaskManager.Task::(_decref) Task=`ef8dcd38-7c03-400c-8c2a-d223121aab25`::ref 0 aborting False
>Thread-30251::DEBUG::2014-04-30 09:59:17,302::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [3199c393]
>Thread-30251::DEBUG::2014-04-30 09:59:17,303::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30251::DEBUG::2014-04-30 09:59:17,317::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30251::DEBUG::2014-04-30 09:59:17,367::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30252::DEBUG::2014-04-30 09:59:17,604::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [3199c393]
>Thread-30252::ERROR::2014-04-30 09:59:17,604::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30252::DEBUG::2014-04-30 09:59:17,605::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30253::DEBUG::2014-04-30 09:59:20,245::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [2c3ed00d]
>Thread-30253::DEBUG::2014-04-30 09:59:20,246::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30253::DEBUG::2014-04-30 09:59:20,268::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30253::DEBUG::2014-04-30 09:59:20,307::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30254::DEBUG::2014-04-30 09:59:20,538::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [2c3ed00d]
>Thread-30254::ERROR::2014-04-30 09:59:20,539::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30254::DEBUG::2014-04-30 09:59:20,539::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30256::DEBUG::2014-04-30 09:59:21,002::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [1cfc9fe9]
>Thread-30256::DEBUG::2014-04-30 09:59:21,002::task::579::TaskManager.Task::(_updateState) Task=`3c1b8ad1-a447-430e-a144-cadab32c5479`::moving from state init -> state preparing
>Thread-30256::INFO::2014-04-30 09:59:21,003::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30256::DEBUG::2014-04-30 09:59:21,008::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30256::ERROR::2014-04-30 10:01:26,065::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30256::ERROR::2014-04-30 10:01:26,067::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30256::DEBUG::2014-04-30 10:01:26,068::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30256::DEBUG::2014-04-30 10:01:26,073::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30257::DEBUG::2014-04-30 10:02:21,181::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30257::DEBUG::2014-04-30 10:02:21,182::task::579::TaskManager.Task::(_updateState) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::moving from state init -> state preparing
>Thread-30257::INFO::2014-04-30 10:02:21,182::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30257::DEBUG::2014-04-30 10:02:21,183::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`836446f5-af7e-49c3-b32b-6865e5bdebac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30257::DEBUG::2014-04-30 10:02:21,183::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30257::DEBUG::2014-04-30 10:02:21,183::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30257::DEBUG::2014-04-30 10:02:21,183::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`836446f5-af7e-49c3-b32b-6865e5bdebac`::Granted request
>Thread-30257::DEBUG::2014-04-30 10:02:21,184::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`f3ee0c3d-5eac-4345-8c5e-f9991f85343e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30257::DEBUG::2014-04-30 10:02:21,184::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30257::DEBUG::2014-04-30 10:02:21,185::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30257::DEBUG::2014-04-30 10:02:21,185::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`f3ee0c3d-5eac-4345-8c5e-f9991f85343e`::Granted request
>Thread-30257::INFO::2014-04-30 10:02:21,186::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30257::DEBUG::2014-04-30 10:02:21,186::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30257::DEBUG::2014-04-30 10:02:21,187::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30257::DEBUG::2014-04-30 10:02:21,187::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30257::DEBUG::2014-04-30 10:02:21,187::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30257::DEBUG::2014-04-30 10:02:21,187::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30257::DEBUG::2014-04-30 10:02:21,188::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30257::DEBUG::2014-04-30 10:02:21,188::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30257::DEBUG::2014-04-30 10:02:21,188::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30257::DEBUG::2014-04-30 10:02:21,188::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30257::DEBUG::2014-04-30 10:02:21,189::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30257::DEBUG::2014-04-30 10:02:21,189::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30257::DEBUG::2014-04-30 10:02:21,211::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30257::DEBUG::2014-04-30 10:02:21,212::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30257::DEBUG::2014-04-30 10:02:21,214::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30257::DEBUG::2014-04-30 10:02:21,214::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30257::DEBUG::2014-04-30 10:02:21,215::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30257::DEBUG::2014-04-30 10:02:21,215::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30257::DEBUG::2014-04-30 10:02:21,215::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30257::DEBUG::2014-04-30 10:02:21,216::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30257::DEBUG::2014-04-30 10:02:21,216::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30257::DEBUG::2014-04-30 10:02:21,216::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30257::ERROR::2014-04-30 10:02:21,216::task::850::TaskManager.Task::(_setError) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30257::DEBUG::2014-04-30 10:02:21,217::task::869::TaskManager.Task::(_run) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::Task._run: f69ef5c3-1e3a-4302-9a5a-918010e27bbc ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30257::DEBUG::2014-04-30 10:02:21,218::task::1194::TaskManager.Task::(stop) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::stopping in state preparing (force False)
>Thread-30257::DEBUG::2014-04-30 10:02:21,218::task::974::TaskManager.Task::(_decref) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::ref 1 aborting True
>Thread-30257::INFO::2014-04-30 10:02:21,218::task::1151::TaskManager.Task::(prepare) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30257::DEBUG::2014-04-30 10:02:21,218::task::1156::TaskManager.Task::(prepare) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30257::DEBUG::2014-04-30 10:02:21,219::task::974::TaskManager.Task::(_decref) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::ref 0 aborting True
>Thread-30257::DEBUG::2014-04-30 10:02:21,219::task::909::TaskManager.Task::(_doAbort) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::Task._doAbort: force False
>Thread-30257::DEBUG::2014-04-30 10:02:21,219::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30257::DEBUG::2014-04-30 10:02:21,219::task::579::TaskManager.Task::(_updateState) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::moving from state preparing -> state aborting
>Thread-30257::DEBUG::2014-04-30 10:02:21,220::task::534::TaskManager.Task::(__state_aborting) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::_aborting: recover policy none
>Thread-30257::DEBUG::2014-04-30 10:02:21,220::task::579::TaskManager.Task::(_updateState) Task=`f69ef5c3-1e3a-4302-9a5a-918010e27bbc`::moving from state aborting -> state failed
>Thread-30257::DEBUG::2014-04-30 10:02:21,220::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30257::DEBUG::2014-04-30 10:02:21,220::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30257::ERROR::2014-04-30 10:02:21,220::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30256::ERROR::2014-04-30 10:03:31,117::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30256::ERROR::2014-04-30 10:03:31,119::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30256::DEBUG::2014-04-30 10:03:31,119::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30256::DEBUG::2014-04-30 10:03:31,124::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30256::DEBUG::2014-04-30 10:03:31,126::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30256::DEBUG::2014-04-30 10:03:31,126::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30256::INFO::2014-04-30 10:03:31,126::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30256::DEBUG::2014-04-30 10:03:31,127::task::1168::TaskManager.Task::(prepare) Task=`3c1b8ad1-a447-430e-a144-cadab32c5479`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30256::DEBUG::2014-04-30 10:03:31,127::task::579::TaskManager.Task::(_updateState) Task=`3c1b8ad1-a447-430e-a144-cadab32c5479`::moving from state preparing -> state finished
>Thread-30256::DEBUG::2014-04-30 10:03:31,127::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30256::DEBUG::2014-04-30 10:03:31,127::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30256::DEBUG::2014-04-30 10:03:31,128::task::974::TaskManager.Task::(_decref) Task=`3c1b8ad1-a447-430e-a144-cadab32c5479`::ref 0 aborting False
>Thread-30258::DEBUG::2014-04-30 10:04:17,281::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [48fbf8d9]
>Thread-30258::DEBUG::2014-04-30 10:04:17,282::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30258::DEBUG::2014-04-30 10:04:17,299::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30258::DEBUG::2014-04-30 10:04:17,337::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30259::DEBUG::2014-04-30 10:04:17,561::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [48fbf8d9]
>Thread-30259::ERROR::2014-04-30 10:04:17,562::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30259::DEBUG::2014-04-30 10:04:17,562::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30260::DEBUG::2014-04-30 10:04:18,567::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [7bb09a3d]
>Thread-30260::DEBUG::2014-04-30 10:04:18,568::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30260::DEBUG::2014-04-30 10:04:18,582::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30260::DEBUG::2014-04-30 10:04:18,617::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30261::DEBUG::2014-04-30 10:04:18,841::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [7bb09a3d]
>Thread-30261::ERROR::2014-04-30 10:04:18,841::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30261::DEBUG::2014-04-30 10:04:18,842::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30263::DEBUG::2014-04-30 10:04:19,392::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [4357d965]
>Thread-30263::DEBUG::2014-04-30 10:04:19,393::task::579::TaskManager.Task::(_updateState) Task=`df6056b9-9d79-4f30-99bd-47be9a01ce5c`::moving from state init -> state preparing
>Thread-30263::INFO::2014-04-30 10:04:19,393::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30263::DEBUG::2014-04-30 10:04:19,397::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30263::ERROR::2014-04-30 10:06:24,439::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30263::ERROR::2014-04-30 10:06:24,444::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30263::DEBUG::2014-04-30 10:06:24,444::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30263::DEBUG::2014-04-30 10:06:24,449::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30264::DEBUG::2014-04-30 10:07:19,592::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30264::DEBUG::2014-04-30 10:07:19,593::task::579::TaskManager.Task::(_updateState) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::moving from state init -> state preparing
>Thread-30264::INFO::2014-04-30 10:07:19,593::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30264::DEBUG::2014-04-30 10:07:19,594::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f71a85de-b2c8-4363-b2ab-dae1771ec42d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30264::DEBUG::2014-04-30 10:07:19,594::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30264::DEBUG::2014-04-30 10:07:19,594::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30264::DEBUG::2014-04-30 10:07:19,594::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f71a85de-b2c8-4363-b2ab-dae1771ec42d`::Granted request
>Thread-30264::DEBUG::2014-04-30 10:07:19,595::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`7b3e95ad-660f-4f0a-a66c-77221364f531`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30264::DEBUG::2014-04-30 10:07:19,595::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30264::DEBUG::2014-04-30 10:07:19,595::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30264::DEBUG::2014-04-30 10:07:19,596::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`7b3e95ad-660f-4f0a-a66c-77221364f531`::Granted request
>Thread-30264::INFO::2014-04-30 10:07:19,596::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30264::DEBUG::2014-04-30 10:07:19,597::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30264::DEBUG::2014-04-30 10:07:19,597::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30264::DEBUG::2014-04-30 10:07:19,597::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30264::DEBUG::2014-04-30 10:07:19,597::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30264::DEBUG::2014-04-30 10:07:19,597::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30264::DEBUG::2014-04-30 10:07:19,598::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30264::DEBUG::2014-04-30 10:07:19,598::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30264::DEBUG::2014-04-30 10:07:19,598::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30264::DEBUG::2014-04-30 10:07:19,598::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30264::DEBUG::2014-04-30 10:07:19,598::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30264::DEBUG::2014-04-30 10:07:19,599::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30264::DEBUG::2014-04-30 10:07:19,620::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30264::DEBUG::2014-04-30 10:07:19,621::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30264::DEBUG::2014-04-30 10:07:19,622::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30264::DEBUG::2014-04-30 10:07:19,622::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30264::DEBUG::2014-04-30 10:07:19,622::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30264::DEBUG::2014-04-30 10:07:19,623::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30264::DEBUG::2014-04-30 10:07:19,623::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30264::DEBUG::2014-04-30 10:07:19,623::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30264::DEBUG::2014-04-30 10:07:19,624::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30264::DEBUG::2014-04-30 10:07:19,624::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30264::ERROR::2014-04-30 10:07:19,624::task::850::TaskManager.Task::(_setError) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30264::DEBUG::2014-04-30 10:07:19,625::task::869::TaskManager.Task::(_run) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::Task._run: 4ffd17f6-d746-4d76-9355-8f19ddb23a79 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30264::DEBUG::2014-04-30 10:07:19,625::task::1194::TaskManager.Task::(stop) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::stopping in state preparing (force False)
>Thread-30264::DEBUG::2014-04-30 10:07:19,626::task::974::TaskManager.Task::(_decref) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::ref 1 aborting True
>Thread-30264::INFO::2014-04-30 10:07:19,626::task::1151::TaskManager.Task::(prepare) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30264::DEBUG::2014-04-30 10:07:19,626::task::1156::TaskManager.Task::(prepare) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30264::DEBUG::2014-04-30 10:07:19,627::task::974::TaskManager.Task::(_decref) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::ref 0 aborting True
>Thread-30264::DEBUG::2014-04-30 10:07:19,627::task::909::TaskManager.Task::(_doAbort) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::Task._doAbort: force False
>Thread-30264::DEBUG::2014-04-30 10:07:19,627::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30264::DEBUG::2014-04-30 10:07:19,627::task::579::TaskManager.Task::(_updateState) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::moving from state preparing -> state aborting
>Thread-30264::DEBUG::2014-04-30 10:07:19,627::task::534::TaskManager.Task::(__state_aborting) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::_aborting: recover policy none
>Thread-30264::DEBUG::2014-04-30 10:07:19,628::task::579::TaskManager.Task::(_updateState) Task=`4ffd17f6-d746-4d76-9355-8f19ddb23a79`::moving from state aborting -> state failed
>Thread-30264::DEBUG::2014-04-30 10:07:19,628::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30264::DEBUG::2014-04-30 10:07:19,628::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30264::ERROR::2014-04-30 10:07:19,628::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30263::ERROR::2014-04-30 10:08:29,489::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30263::ERROR::2014-04-30 10:08:29,490::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30263::DEBUG::2014-04-30 10:08:29,490::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30263::DEBUG::2014-04-30 10:08:29,494::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30263::DEBUG::2014-04-30 10:08:29,496::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30263::DEBUG::2014-04-30 10:08:29,496::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30263::INFO::2014-04-30 10:08:29,496::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30263::DEBUG::2014-04-30 10:08:29,496::task::1168::TaskManager.Task::(prepare) Task=`df6056b9-9d79-4f30-99bd-47be9a01ce5c`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30263::DEBUG::2014-04-30 10:08:29,497::task::579::TaskManager.Task::(_updateState) Task=`df6056b9-9d79-4f30-99bd-47be9a01ce5c`::moving from state preparing -> state finished
>Thread-30263::DEBUG::2014-04-30 10:08:29,497::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30263::DEBUG::2014-04-30 10:08:29,497::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30263::DEBUG::2014-04-30 10:08:29,497::task::974::TaskManager.Task::(_decref) Task=`df6056b9-9d79-4f30-99bd-47be9a01ce5c`::ref 0 aborting False
>Thread-30265::DEBUG::2014-04-30 10:09:17,279::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [b9435e8]
>Thread-30265::DEBUG::2014-04-30 10:09:17,280::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30265::DEBUG::2014-04-30 10:09:17,293::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30265::DEBUG::2014-04-30 10:09:17,331::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30266::DEBUG::2014-04-30 10:09:17,562::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [b9435e8]
>Thread-30266::ERROR::2014-04-30 10:09:17,562::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30266::DEBUG::2014-04-30 10:09:17,563::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30267::DEBUG::2014-04-30 10:09:19,979::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [bf06694]
>Thread-30267::DEBUG::2014-04-30 10:09:19,980::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30267::DEBUG::2014-04-30 10:09:19,993::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30267::DEBUG::2014-04-30 10:09:20,031::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30268::DEBUG::2014-04-30 10:09:20,257::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [bf06694]
>Thread-30268::ERROR::2014-04-30 10:09:20,258::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30268::DEBUG::2014-04-30 10:09:20,258::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30270::DEBUG::2014-04-30 10:09:20,741::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [1cdc077a]
>Thread-30270::DEBUG::2014-04-30 10:09:20,741::task::579::TaskManager.Task::(_updateState) Task=`53a0a4cf-fc51-4cb3-8d91-34c855a09b19`::moving from state init -> state preparing
>Thread-30270::INFO::2014-04-30 10:09:20,742::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30270::DEBUG::2014-04-30 10:09:20,746::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30270::ERROR::2014-04-30 10:11:25,787::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30270::ERROR::2014-04-30 10:11:25,788::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30270::DEBUG::2014-04-30 10:11:25,789::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30270::DEBUG::2014-04-30 10:11:25,793::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30271::DEBUG::2014-04-30 10:12:20,916::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30271::DEBUG::2014-04-30 10:12:20,917::task::579::TaskManager.Task::(_updateState) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::moving from state init -> state preparing
>Thread-30271::INFO::2014-04-30 10:12:20,917::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30271::DEBUG::2014-04-30 10:12:20,918::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`39db09f0-435f-4b2d-b21d-40a2f581b440`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30271::DEBUG::2014-04-30 10:12:20,918::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30271::DEBUG::2014-04-30 10:12:20,918::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30271::DEBUG::2014-04-30 10:12:20,918::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`39db09f0-435f-4b2d-b21d-40a2f581b440`::Granted request
>Thread-30271::DEBUG::2014-04-30 10:12:20,919::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`58f3bc0d-511b-4c5b-ba0d-a7797f42aa26`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30271::DEBUG::2014-04-30 10:12:20,919::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30271::DEBUG::2014-04-30 10:12:20,919::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30271::DEBUG::2014-04-30 10:12:20,920::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`58f3bc0d-511b-4c5b-ba0d-a7797f42aa26`::Granted request
>Thread-30271::INFO::2014-04-30 10:12:20,920::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30271::DEBUG::2014-04-30 10:12:20,921::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30271::DEBUG::2014-04-30 10:12:20,921::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30271::DEBUG::2014-04-30 10:12:20,921::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30271::DEBUG::2014-04-30 10:12:20,921::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30271::DEBUG::2014-04-30 10:12:20,922::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30271::DEBUG::2014-04-30 10:12:20,922::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30271::DEBUG::2014-04-30 10:12:20,922::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30271::DEBUG::2014-04-30 10:12:20,922::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30271::DEBUG::2014-04-30 10:12:20,922::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30271::DEBUG::2014-04-30 10:12:20,922::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30271::DEBUG::2014-04-30 10:12:20,923::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30271::DEBUG::2014-04-30 10:12:20,944::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30271::DEBUG::2014-04-30 10:12:20,945::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30271::DEBUG::2014-04-30 10:12:20,946::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30271::DEBUG::2014-04-30 10:12:20,946::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30271::DEBUG::2014-04-30 10:12:20,946::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30271::DEBUG::2014-04-30 10:12:20,947::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30271::DEBUG::2014-04-30 10:12:20,947::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30271::DEBUG::2014-04-30 10:12:20,947::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30271::DEBUG::2014-04-30 10:12:20,948::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30271::DEBUG::2014-04-30 10:12:20,948::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30271::ERROR::2014-04-30 10:12:20,948::task::850::TaskManager.Task::(_setError) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30271::DEBUG::2014-04-30 10:12:20,949::task::869::TaskManager.Task::(_run) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::Task._run: d8eb40a5-bc6a-41df-8c52-3bb647bae6d0 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30271::DEBUG::2014-04-30 10:12:20,949::task::1194::TaskManager.Task::(stop) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::stopping in state preparing (force False)
>Thread-30271::DEBUG::2014-04-30 10:12:20,950::task::974::TaskManager.Task::(_decref) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::ref 1 aborting True
>Thread-30271::INFO::2014-04-30 10:12:20,950::task::1151::TaskManager.Task::(prepare) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30271::DEBUG::2014-04-30 10:12:20,950::task::1156::TaskManager.Task::(prepare) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30271::DEBUG::2014-04-30 10:12:20,951::task::974::TaskManager.Task::(_decref) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::ref 0 aborting True
>Thread-30271::DEBUG::2014-04-30 10:12:20,951::task::909::TaskManager.Task::(_doAbort) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::Task._doAbort: force False
>Thread-30271::DEBUG::2014-04-30 10:12:20,951::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30271::DEBUG::2014-04-30 10:12:20,951::task::579::TaskManager.Task::(_updateState) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::moving from state preparing -> state aborting
>Thread-30271::DEBUG::2014-04-30 10:12:20,951::task::534::TaskManager.Task::(__state_aborting) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::_aborting: recover policy none
>Thread-30271::DEBUG::2014-04-30 10:12:20,952::task::579::TaskManager.Task::(_updateState) Task=`d8eb40a5-bc6a-41df-8c52-3bb647bae6d0`::moving from state aborting -> state failed
>Thread-30271::DEBUG::2014-04-30 10:12:20,952::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30271::DEBUG::2014-04-30 10:12:20,952::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30271::ERROR::2014-04-30 10:12:20,952::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30270::ERROR::2014-04-30 10:13:30,840::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30270::ERROR::2014-04-30 10:13:30,841::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30270::DEBUG::2014-04-30 10:13:30,841::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30270::DEBUG::2014-04-30 10:13:30,845::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30270::DEBUG::2014-04-30 10:13:30,846::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30270::DEBUG::2014-04-30 10:13:30,847::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30270::INFO::2014-04-30 10:13:30,847::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30270::DEBUG::2014-04-30 10:13:30,847::task::1168::TaskManager.Task::(prepare) Task=`53a0a4cf-fc51-4cb3-8d91-34c855a09b19`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30270::DEBUG::2014-04-30 10:13:30,847::task::579::TaskManager.Task::(_updateState) Task=`53a0a4cf-fc51-4cb3-8d91-34c855a09b19`::moving from state preparing -> state finished
>Thread-30270::DEBUG::2014-04-30 10:13:30,848::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30270::DEBUG::2014-04-30 10:13:30,848::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30270::DEBUG::2014-04-30 10:13:30,848::task::974::TaskManager.Task::(_decref) Task=`53a0a4cf-fc51-4cb3-8d91-34c855a09b19`::ref 0 aborting False
>Thread-30272::DEBUG::2014-04-30 10:14:17,267::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [6d67f0b9]
>Thread-30272::DEBUG::2014-04-30 10:14:17,268::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30272::DEBUG::2014-04-30 10:14:17,282::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30272::DEBUG::2014-04-30 10:14:17,319::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30273::DEBUG::2014-04-30 10:14:17,544::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [6d67f0b9]
>Thread-30273::ERROR::2014-04-30 10:14:17,544::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30273::DEBUG::2014-04-30 10:14:17,545::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30274::DEBUG::2014-04-30 10:14:18,279::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [4b6e1e62]
>Thread-30274::DEBUG::2014-04-30 10:14:18,281::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30274::DEBUG::2014-04-30 10:14:18,294::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30274::DEBUG::2014-04-30 10:14:18,331::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30275::DEBUG::2014-04-30 10:14:18,558::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [4b6e1e62]
>Thread-30275::ERROR::2014-04-30 10:14:18,558::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30275::DEBUG::2014-04-30 10:14:18,559::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30277::DEBUG::2014-04-30 10:14:19,022::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [60c4164d]
>Thread-30277::DEBUG::2014-04-30 10:14:19,023::task::579::TaskManager.Task::(_updateState) Task=`67fbdbd9-9611-496f-82a5-571e3453fc08`::moving from state init -> state preparing
>Thread-30277::INFO::2014-04-30 10:14:19,023::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30277::DEBUG::2014-04-30 10:14:19,028::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30277::ERROR::2014-04-30 10:16:24,070::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30277::ERROR::2014-04-30 10:16:24,072::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30277::DEBUG::2014-04-30 10:16:24,072::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30277::DEBUG::2014-04-30 10:16:24,077::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30278::DEBUG::2014-04-30 10:17:19,212::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30278::DEBUG::2014-04-30 10:17:19,212::task::579::TaskManager.Task::(_updateState) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::moving from state init -> state preparing
>Thread-30278::INFO::2014-04-30 10:17:19,213::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30278::DEBUG::2014-04-30 10:17:19,213::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4f5e1009-17f2-4bf7-8b9a-16246e4095e0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30278::DEBUG::2014-04-30 10:17:19,213::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30278::DEBUG::2014-04-30 10:17:19,214::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30278::DEBUG::2014-04-30 10:17:19,214::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4f5e1009-17f2-4bf7-8b9a-16246e4095e0`::Granted request
>Thread-30278::DEBUG::2014-04-30 10:17:19,214::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`bd3aa9ec-2732-4c53-a342-104f15e9d18d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30278::DEBUG::2014-04-30 10:17:19,215::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30278::DEBUG::2014-04-30 10:17:19,215::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30278::DEBUG::2014-04-30 10:17:19,215::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`bd3aa9ec-2732-4c53-a342-104f15e9d18d`::Granted request
>Thread-30278::INFO::2014-04-30 10:17:19,216::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30278::DEBUG::2014-04-30 10:17:19,216::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30278::DEBUG::2014-04-30 10:17:19,216::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30278::DEBUG::2014-04-30 10:17:19,217::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30278::DEBUG::2014-04-30 10:17:19,217::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30278::DEBUG::2014-04-30 10:17:19,217::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30278::DEBUG::2014-04-30 10:17:19,217::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30278::DEBUG::2014-04-30 10:17:19,217::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30278::DEBUG::2014-04-30 10:17:19,218::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30278::DEBUG::2014-04-30 10:17:19,218::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30278::DEBUG::2014-04-30 10:17:19,218::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30278::DEBUG::2014-04-30 10:17:19,218::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30278::DEBUG::2014-04-30 10:17:19,240::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30278::DEBUG::2014-04-30 10:17:19,241::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30278::DEBUG::2014-04-30 10:17:19,241::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30278::DEBUG::2014-04-30 10:17:19,242::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30278::DEBUG::2014-04-30 10:17:19,242::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30278::DEBUG::2014-04-30 10:17:19,242::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30278::DEBUG::2014-04-30 10:17:19,243::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30278::DEBUG::2014-04-30 10:17:19,243::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30278::DEBUG::2014-04-30 10:17:19,243::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30278::DEBUG::2014-04-30 10:17:19,243::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30278::ERROR::2014-04-30 10:17:19,244::task::850::TaskManager.Task::(_setError) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30278::DEBUG::2014-04-30 10:17:19,245::task::869::TaskManager.Task::(_run) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::Task._run: 42fb8c78-059d-407d-b028-c8fe35937432 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30278::DEBUG::2014-04-30 10:17:19,245::task::1194::TaskManager.Task::(stop) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::stopping in state preparing (force False)
>Thread-30278::DEBUG::2014-04-30 10:17:19,245::task::974::TaskManager.Task::(_decref) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::ref 1 aborting True
>Thread-30278::INFO::2014-04-30 10:17:19,246::task::1151::TaskManager.Task::(prepare) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30278::DEBUG::2014-04-30 10:17:19,246::task::1156::TaskManager.Task::(prepare) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30278::DEBUG::2014-04-30 10:17:19,246::task::974::TaskManager.Task::(_decref) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::ref 0 aborting True
>Thread-30278::DEBUG::2014-04-30 10:17:19,246::task::909::TaskManager.Task::(_doAbort) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::Task._doAbort: force False
>Thread-30278::DEBUG::2014-04-30 10:17:19,247::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30278::DEBUG::2014-04-30 10:17:19,247::task::579::TaskManager.Task::(_updateState) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::moving from state preparing -> state aborting
>Thread-30278::DEBUG::2014-04-30 10:17:19,247::task::534::TaskManager.Task::(__state_aborting) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::_aborting: recover policy none
>Thread-30278::DEBUG::2014-04-30 10:17:19,247::task::579::TaskManager.Task::(_updateState) Task=`42fb8c78-059d-407d-b028-c8fe35937432`::moving from state aborting -> state failed
>Thread-30278::DEBUG::2014-04-30 10:17:19,247::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30278::DEBUG::2014-04-30 10:17:19,248::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30278::ERROR::2014-04-30 10:17:19,248::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30277::ERROR::2014-04-30 10:18:29,123::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30277::ERROR::2014-04-30 10:18:29,124::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30277::DEBUG::2014-04-30 10:18:29,125::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30277::DEBUG::2014-04-30 10:18:29,129::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30277::DEBUG::2014-04-30 10:18:29,130::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30277::DEBUG::2014-04-30 10:18:29,130::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30277::INFO::2014-04-30 10:18:29,130::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30277::DEBUG::2014-04-30 10:18:29,131::task::1168::TaskManager.Task::(prepare) Task=`67fbdbd9-9611-496f-82a5-571e3453fc08`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30277::DEBUG::2014-04-30 10:18:29,131::task::579::TaskManager.Task::(_updateState) Task=`67fbdbd9-9611-496f-82a5-571e3453fc08`::moving from state preparing -> state finished
>Thread-30277::DEBUG::2014-04-30 10:18:29,131::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30277::DEBUG::2014-04-30 10:18:29,131::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30277::DEBUG::2014-04-30 10:18:29,132::task::974::TaskManager.Task::(_decref) Task=`67fbdbd9-9611-496f-82a5-571e3453fc08`::ref 0 aborting False
>Thread-30279::DEBUG::2014-04-30 10:19:17,260::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [7f555d0e]
>Thread-30279::DEBUG::2014-04-30 10:19:17,261::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30279::DEBUG::2014-04-30 10:19:17,275::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30279::DEBUG::2014-04-30 10:19:17,312::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30280::DEBUG::2014-04-30 10:19:17,537::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [7f555d0e]
>Thread-30280::ERROR::2014-04-30 10:19:17,537::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30280::DEBUG::2014-04-30 10:19:17,539::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30281::DEBUG::2014-04-30 10:19:19,631::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [40945c72]
>Thread-30281::DEBUG::2014-04-30 10:19:19,632::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30281::DEBUG::2014-04-30 10:19:19,645::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30281::DEBUG::2014-04-30 10:19:19,682::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30282::DEBUG::2014-04-30 10:19:19,924::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [40945c72]
>Thread-30282::ERROR::2014-04-30 10:19:19,924::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30282::DEBUG::2014-04-30 10:19:19,925::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30284::DEBUG::2014-04-30 10:19:20,394::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [2498d1d4]
>Thread-30284::DEBUG::2014-04-30 10:19:20,395::task::579::TaskManager.Task::(_updateState) Task=`8233a311-0409-4d50-abb6-dc79795a7945`::moving from state init -> state preparing
>Thread-30284::INFO::2014-04-30 10:19:20,395::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30284::DEBUG::2014-04-30 10:19:20,400::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30284::ERROR::2014-04-30 10:21:25,443::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30284::ERROR::2014-04-30 10:21:25,444::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30284::DEBUG::2014-04-30 10:21:25,444::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30284::DEBUG::2014-04-30 10:21:25,450::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30285::DEBUG::2014-04-30 10:22:20,582::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30285::DEBUG::2014-04-30 10:22:20,583::task::579::TaskManager.Task::(_updateState) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::moving from state init -> state preparing
>Thread-30285::INFO::2014-04-30 10:22:20,583::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30285::DEBUG::2014-04-30 10:22:20,584::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`725ab0b0-c041-482b-8ade-c66ca9b25f17`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30285::DEBUG::2014-04-30 10:22:20,584::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30285::DEBUG::2014-04-30 10:22:20,584::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30285::DEBUG::2014-04-30 10:22:20,584::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`725ab0b0-c041-482b-8ade-c66ca9b25f17`::Granted request
>Thread-30285::DEBUG::2014-04-30 10:22:20,585::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`aea619a6-64e6-4d59-8d0c-fad1544eb0e9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30285::DEBUG::2014-04-30 10:22:20,585::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30285::DEBUG::2014-04-30 10:22:20,585::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30285::DEBUG::2014-04-30 10:22:20,586::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`aea619a6-64e6-4d59-8d0c-fad1544eb0e9`::Granted request
>Thread-30285::INFO::2014-04-30 10:22:20,586::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30285::DEBUG::2014-04-30 10:22:20,587::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30285::DEBUG::2014-04-30 10:22:20,587::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30285::DEBUG::2014-04-30 10:22:20,587::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30285::DEBUG::2014-04-30 10:22:20,587::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30285::DEBUG::2014-04-30 10:22:20,588::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30285::DEBUG::2014-04-30 10:22:20,588::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30285::DEBUG::2014-04-30 10:22:20,588::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30285::DEBUG::2014-04-30 10:22:20,588::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30285::DEBUG::2014-04-30 10:22:20,588::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30285::DEBUG::2014-04-30 10:22:20,588::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30285::DEBUG::2014-04-30 10:22:20,589::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30285::DEBUG::2014-04-30 10:22:20,611::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30285::DEBUG::2014-04-30 10:22:20,612::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30285::DEBUG::2014-04-30 10:22:20,613::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30285::DEBUG::2014-04-30 10:22:20,613::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30285::DEBUG::2014-04-30 10:22:20,613::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30285::DEBUG::2014-04-30 10:22:20,613::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30285::DEBUG::2014-04-30 10:22:20,614::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30285::DEBUG::2014-04-30 10:22:20,614::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30285::DEBUG::2014-04-30 10:22:20,614::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30285::DEBUG::2014-04-30 10:22:20,614::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30285::ERROR::2014-04-30 10:22:20,615::task::850::TaskManager.Task::(_setError) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30285::DEBUG::2014-04-30 10:22:20,616::task::869::TaskManager.Task::(_run) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::Task._run: 4afd405b-33ad-4f2b-a71d-f34739ec0526 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30285::DEBUG::2014-04-30 10:22:20,616::task::1194::TaskManager.Task::(stop) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::stopping in state preparing (force False)
>Thread-30285::DEBUG::2014-04-30 10:22:20,616::task::974::TaskManager.Task::(_decref) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::ref 1 aborting True
>Thread-30285::INFO::2014-04-30 10:22:20,617::task::1151::TaskManager.Task::(prepare) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30285::DEBUG::2014-04-30 10:22:20,617::task::1156::TaskManager.Task::(prepare) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30285::DEBUG::2014-04-30 10:22:20,617::task::974::TaskManager.Task::(_decref) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::ref 0 aborting True
>Thread-30285::DEBUG::2014-04-30 10:22:20,617::task::909::TaskManager.Task::(_doAbort) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::Task._doAbort: force False
>Thread-30285::DEBUG::2014-04-30 10:22:20,618::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30285::DEBUG::2014-04-30 10:22:20,618::task::579::TaskManager.Task::(_updateState) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::moving from state preparing -> state aborting
>Thread-30285::DEBUG::2014-04-30 10:22:20,618::task::534::TaskManager.Task::(__state_aborting) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::_aborting: recover policy none
>Thread-30285::DEBUG::2014-04-30 10:22:20,618::task::579::TaskManager.Task::(_updateState) Task=`4afd405b-33ad-4f2b-a71d-f34739ec0526`::moving from state aborting -> state failed
>Thread-30285::DEBUG::2014-04-30 10:22:20,619::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30285::DEBUG::2014-04-30 10:22:20,619::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30285::ERROR::2014-04-30 10:22:20,619::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30284::ERROR::2014-04-30 10:23:30,491::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30284::ERROR::2014-04-30 10:23:30,492::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30284::DEBUG::2014-04-30 10:23:30,492::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30284::DEBUG::2014-04-30 10:23:30,496::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30284::DEBUG::2014-04-30 10:23:30,497::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30284::DEBUG::2014-04-30 10:23:30,498::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30284::INFO::2014-04-30 10:23:30,498::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30284::DEBUG::2014-04-30 10:23:30,498::task::1168::TaskManager.Task::(prepare) Task=`8233a311-0409-4d50-abb6-dc79795a7945`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30284::DEBUG::2014-04-30 10:23:30,499::task::579::TaskManager.Task::(_updateState) Task=`8233a311-0409-4d50-abb6-dc79795a7945`::moving from state preparing -> state finished
>Thread-30284::DEBUG::2014-04-30 10:23:30,499::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30284::DEBUG::2014-04-30 10:23:30,499::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30284::DEBUG::2014-04-30 10:23:30,499::task::974::TaskManager.Task::(_decref) Task=`8233a311-0409-4d50-abb6-dc79795a7945`::ref 0 aborting False
>Thread-30286::DEBUG::2014-04-30 10:24:17,251::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [469b770b]
>Thread-30286::DEBUG::2014-04-30 10:24:17,252::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30286::DEBUG::2014-04-30 10:24:17,267::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30286::DEBUG::2014-04-30 10:24:17,304::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30287::DEBUG::2014-04-30 10:24:17,531::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [469b770b]
>Thread-30287::ERROR::2014-04-30 10:24:17,532::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30287::DEBUG::2014-04-30 10:24:17,532::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30288::DEBUG::2014-04-30 10:24:18,074::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [610f48e]
>Thread-30288::DEBUG::2014-04-30 10:24:18,076::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30288::DEBUG::2014-04-30 10:24:18,089::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30288::DEBUG::2014-04-30 10:24:18,127::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30289::DEBUG::2014-04-30 10:24:18,360::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [610f48e]
>Thread-30289::ERROR::2014-04-30 10:24:18,361::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30289::DEBUG::2014-04-30 10:24:18,361::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30291::DEBUG::2014-04-30 10:24:18,842::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [ae9461e]
>Thread-30291::DEBUG::2014-04-30 10:24:18,842::task::579::TaskManager.Task::(_updateState) Task=`92d517e0-7548-40e8-9900-435c587f26a0`::moving from state init -> state preparing
>Thread-30291::INFO::2014-04-30 10:24:18,843::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30291::DEBUG::2014-04-30 10:24:18,847::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30291::ERROR::2014-04-30 10:26:23,894::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30291::ERROR::2014-04-30 10:26:23,896::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30291::DEBUG::2014-04-30 10:26:23,897::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30291::DEBUG::2014-04-30 10:26:23,901::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30292::DEBUG::2014-04-30 10:27:19,018::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30292::DEBUG::2014-04-30 10:27:19,019::task::579::TaskManager.Task::(_updateState) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::moving from state init -> state preparing
>Thread-30292::INFO::2014-04-30 10:27:19,019::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30292::DEBUG::2014-04-30 10:27:19,020::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`88731efe-7e0e-4e67-b531-62761e5640fe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30292::DEBUG::2014-04-30 10:27:19,020::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30292::DEBUG::2014-04-30 10:27:19,021::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30292::DEBUG::2014-04-30 10:27:19,021::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`88731efe-7e0e-4e67-b531-62761e5640fe`::Granted request
>Thread-30292::DEBUG::2014-04-30 10:27:19,022::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`4b05761c-768d-4033-9748-c103ce8771c8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30292::DEBUG::2014-04-30 10:27:19,022::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30292::DEBUG::2014-04-30 10:27:19,022::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30292::DEBUG::2014-04-30 10:27:19,022::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`4b05761c-768d-4033-9748-c103ce8771c8`::Granted request
>Thread-30292::INFO::2014-04-30 10:27:19,023::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30292::DEBUG::2014-04-30 10:27:19,024::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30292::DEBUG::2014-04-30 10:27:19,024::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30292::DEBUG::2014-04-30 10:27:19,024::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30292::DEBUG::2014-04-30 10:27:19,025::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30292::DEBUG::2014-04-30 10:27:19,025::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30292::DEBUG::2014-04-30 10:27:19,025::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30292::DEBUG::2014-04-30 10:27:19,025::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30292::DEBUG::2014-04-30 10:27:19,026::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30292::DEBUG::2014-04-30 10:27:19,026::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30292::DEBUG::2014-04-30 10:27:19,026::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30292::DEBUG::2014-04-30 10:27:19,026::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30292::DEBUG::2014-04-30 10:27:19,048::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30292::DEBUG::2014-04-30 10:27:19,049::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30292::DEBUG::2014-04-30 10:27:19,050::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30292::DEBUG::2014-04-30 10:27:19,050::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30292::DEBUG::2014-04-30 10:27:19,050::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30292::DEBUG::2014-04-30 10:27:19,050::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30292::DEBUG::2014-04-30 10:27:19,051::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30292::DEBUG::2014-04-30 10:27:19,051::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30292::DEBUG::2014-04-30 10:27:19,051::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30292::DEBUG::2014-04-30 10:27:19,052::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30292::ERROR::2014-04-30 10:27:19,052::task::850::TaskManager.Task::(_setError) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30292::DEBUG::2014-04-30 10:27:19,053::task::869::TaskManager.Task::(_run) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::Task._run: 47957181-36f1-4c6f-a1a2-d2b38e310d5d ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30292::DEBUG::2014-04-30 10:27:19,053::task::1194::TaskManager.Task::(stop) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::stopping in state preparing (force False)
>Thread-30292::DEBUG::2014-04-30 10:27:19,054::task::974::TaskManager.Task::(_decref) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::ref 1 aborting True
>Thread-30292::INFO::2014-04-30 10:27:19,054::task::1151::TaskManager.Task::(prepare) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30292::DEBUG::2014-04-30 10:27:19,054::task::1156::TaskManager.Task::(prepare) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30292::DEBUG::2014-04-30 10:27:19,054::task::974::TaskManager.Task::(_decref) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::ref 0 aborting True
>Thread-30292::DEBUG::2014-04-30 10:27:19,055::task::909::TaskManager.Task::(_doAbort) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::Task._doAbort: force False
>Thread-30292::DEBUG::2014-04-30 10:27:19,055::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30292::DEBUG::2014-04-30 10:27:19,055::task::579::TaskManager.Task::(_updateState) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::moving from state preparing -> state aborting
>Thread-30292::DEBUG::2014-04-30 10:27:19,055::task::534::TaskManager.Task::(__state_aborting) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::_aborting: recover policy none
>Thread-30292::DEBUG::2014-04-30 10:27:19,055::task::579::TaskManager.Task::(_updateState) Task=`47957181-36f1-4c6f-a1a2-d2b38e310d5d`::moving from state aborting -> state failed
>Thread-30292::DEBUG::2014-04-30 10:27:19,056::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30292::DEBUG::2014-04-30 10:27:19,056::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30292::ERROR::2014-04-30 10:27:19,056::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30291::ERROR::2014-04-30 10:28:28,944::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30291::ERROR::2014-04-30 10:28:28,946::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30291::DEBUG::2014-04-30 10:28:28,946::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30291::DEBUG::2014-04-30 10:28:28,950::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30291::DEBUG::2014-04-30 10:28:28,952::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30291::DEBUG::2014-04-30 10:28:28,953::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30291::INFO::2014-04-30 10:28:28,953::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30291::DEBUG::2014-04-30 10:28:28,953::task::1168::TaskManager.Task::(prepare) Task=`92d517e0-7548-40e8-9900-435c587f26a0`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30291::DEBUG::2014-04-30 10:28:28,953::task::579::TaskManager.Task::(_updateState) Task=`92d517e0-7548-40e8-9900-435c587f26a0`::moving from state preparing -> state finished
>Thread-30291::DEBUG::2014-04-30 10:28:28,954::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30291::DEBUG::2014-04-30 10:28:28,954::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30291::DEBUG::2014-04-30 10:28:28,954::task::974::TaskManager.Task::(_decref) Task=`92d517e0-7548-40e8-9900-435c587f26a0`::ref 0 aborting False
>Thread-30293::DEBUG::2014-04-30 10:29:17,273::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5ea05800]
>Thread-30293::DEBUG::2014-04-30 10:29:17,274::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30293::DEBUG::2014-04-30 10:29:17,289::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30293::DEBUG::2014-04-30 10:29:17,327::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30294::DEBUG::2014-04-30 10:29:17,551::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5ea05800]
>Thread-30294::ERROR::2014-04-30 10:29:17,552::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30294::DEBUG::2014-04-30 10:29:17,553::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30295::DEBUG::2014-04-30 10:29:19,392::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [27ffa3f8]
>Thread-30295::DEBUG::2014-04-30 10:29:19,394::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30295::DEBUG::2014-04-30 10:29:19,408::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30295::DEBUG::2014-04-30 10:29:19,447::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30296::DEBUG::2014-04-30 10:29:19,675::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [27ffa3f8]
>Thread-30296::ERROR::2014-04-30 10:29:19,676::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30296::DEBUG::2014-04-30 10:29:19,676::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30298::DEBUG::2014-04-30 10:29:20,145::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [236497c]
>Thread-30298::DEBUG::2014-04-30 10:29:20,146::task::579::TaskManager.Task::(_updateState) Task=`dc4fa853-9039-458d-8b95-e26af2a98ebf`::moving from state init -> state preparing
>Thread-30298::INFO::2014-04-30 10:29:20,146::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}, {'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30298::DEBUG::2014-04-30 10:29:20,151::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30298::ERROR::2014-04-30 10:31:25,194::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30298::ERROR::2014-04-30 10:31:25,199::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30298::DEBUG::2014-04-30 10:31:25,199::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30298::DEBUG::2014-04-30 10:31:25,204::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30299::DEBUG::2014-04-30 10:32:20,319::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30299::DEBUG::2014-04-30 10:32:20,325::task::579::TaskManager.Task::(_updateState) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::moving from state init -> state preparing
>Thread-30299::INFO::2014-04-30 10:32:20,325::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30299::DEBUG::2014-04-30 10:32:20,326::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ebd74554-b5a1-4c66-a736-03f322dd8752`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30299::DEBUG::2014-04-30 10:32:20,326::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30299::DEBUG::2014-04-30 10:32:20,326::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30299::DEBUG::2014-04-30 10:32:20,327::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ebd74554-b5a1-4c66-a736-03f322dd8752`::Granted request
>Thread-30299::DEBUG::2014-04-30 10:32:20,327::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`e9414606-150c-49cd-b0ba-460079beab76`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30299::DEBUG::2014-04-30 10:32:20,328::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30299::DEBUG::2014-04-30 10:32:20,328::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30299::DEBUG::2014-04-30 10:32:20,328::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`e9414606-150c-49cd-b0ba-460079beab76`::Granted request
>Thread-30299::INFO::2014-04-30 10:32:20,329::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30299::DEBUG::2014-04-30 10:32:20,329::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30299::DEBUG::2014-04-30 10:32:20,330::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30299::DEBUG::2014-04-30 10:32:20,330::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30299::DEBUG::2014-04-30 10:32:20,330::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30299::DEBUG::2014-04-30 10:32:20,331::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30299::DEBUG::2014-04-30 10:32:20,331::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30299::DEBUG::2014-04-30 10:32:20,331::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30299::DEBUG::2014-04-30 10:32:20,331::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30299::DEBUG::2014-04-30 10:32:20,332::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30299::DEBUG::2014-04-30 10:32:20,332::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30299::DEBUG::2014-04-30 10:32:20,332::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30299::DEBUG::2014-04-30 10:32:20,355::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30299::DEBUG::2014-04-30 10:32:20,356::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30299::DEBUG::2014-04-30 10:32:20,357::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30299::DEBUG::2014-04-30 10:32:20,357::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30299::DEBUG::2014-04-30 10:32:20,357::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30299::DEBUG::2014-04-30 10:32:20,357::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30299::DEBUG::2014-04-30 10:32:20,358::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30299::DEBUG::2014-04-30 10:32:20,358::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30299::DEBUG::2014-04-30 10:32:20,358::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30299::DEBUG::2014-04-30 10:32:20,359::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30299::ERROR::2014-04-30 10:32:20,359::task::850::TaskManager.Task::(_setError) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30299::DEBUG::2014-04-30 10:32:20,360::task::869::TaskManager.Task::(_run) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::Task._run: 6fd07a80-4f3a-4c09-be7d-365d0898f3a8 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30299::DEBUG::2014-04-30 10:32:20,360::task::1194::TaskManager.Task::(stop) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::stopping in state preparing (force False)
>Thread-30299::DEBUG::2014-04-30 10:32:20,360::task::974::TaskManager.Task::(_decref) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::ref 1 aborting True
>Thread-30299::INFO::2014-04-30 10:32:20,361::task::1151::TaskManager.Task::(prepare) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30299::DEBUG::2014-04-30 10:32:20,361::task::1156::TaskManager.Task::(prepare) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30299::DEBUG::2014-04-30 10:32:20,361::task::974::TaskManager.Task::(_decref) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::ref 0 aborting True
>Thread-30299::DEBUG::2014-04-30 10:32:20,361::task::909::TaskManager.Task::(_doAbort) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::Task._doAbort: force False
>Thread-30299::DEBUG::2014-04-30 10:32:20,362::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30299::DEBUG::2014-04-30 10:32:20,362::task::579::TaskManager.Task::(_updateState) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::moving from state preparing -> state aborting
>Thread-30299::DEBUG::2014-04-30 10:32:20,362::task::534::TaskManager.Task::(__state_aborting) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::_aborting: recover policy none
>Thread-30299::DEBUG::2014-04-30 10:32:20,362::task::579::TaskManager.Task::(_updateState) Task=`6fd07a80-4f3a-4c09-be7d-365d0898f3a8`::moving from state aborting -> state failed
>Thread-30299::DEBUG::2014-04-30 10:32:20,363::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30299::DEBUG::2014-04-30 10:32:20,363::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30299::ERROR::2014-04-30 10:32:20,363::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30298::ERROR::2014-04-30 10:33:30,246::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30298::ERROR::2014-04-30 10:33:30,247::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30298::DEBUG::2014-04-30 10:33:30,247::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30298::DEBUG::2014-04-30 10:33:30,252::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30298::DEBUG::2014-04-30 10:33:30,253::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30298::DEBUG::2014-04-30 10:33:30,253::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30298::INFO::2014-04-30 10:33:30,253::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30298::DEBUG::2014-04-30 10:33:30,254::task::1168::TaskManager.Task::(prepare) Task=`dc4fa853-9039-458d-8b95-e26af2a98ebf`::finished: {'statuslist': [{'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}, {'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30298::DEBUG::2014-04-30 10:33:30,254::task::579::TaskManager.Task::(_updateState) Task=`dc4fa853-9039-458d-8b95-e26af2a98ebf`::moving from state preparing -> state finished
>Thread-30298::DEBUG::2014-04-30 10:33:30,254::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30298::DEBUG::2014-04-30 10:33:30,254::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30298::DEBUG::2014-04-30 10:33:30,255::task::974::TaskManager.Task::(_decref) Task=`dc4fa853-9039-458d-8b95-e26af2a98ebf`::ref 0 aborting False
>Thread-30300::DEBUG::2014-04-30 10:39:17,511::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [7da255bb]
>Thread-30300::DEBUG::2014-04-30 10:39:17,514::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30300::DEBUG::2014-04-30 10:39:17,530::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30300::DEBUG::2014-04-30 10:39:17,566::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30301::DEBUG::2014-04-30 10:39:17,823::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [7da255bb]
>Thread-30301::ERROR::2014-04-30 10:39:17,823::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30301::DEBUG::2014-04-30 10:39:17,824::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30302::DEBUG::2014-04-30 10:39:19,674::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {}
>Thread-30302::DEBUG::2014-04-30 10:39:19,675::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30302::DEBUG::2014-04-30 10:39:19,688::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30302::DEBUG::2014-04-30 10:39:19,726::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30303::DEBUG::2014-04-30 10:39:19,971::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {}
>Thread-30303::ERROR::2014-04-30 10:39:19,972::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30303::DEBUG::2014-04-30 10:39:19,972::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30304::DEBUG::2014-04-30 10:39:20,243::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {}
>Thread-30304::DEBUG::2014-04-30 10:39:20,244::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30304::DEBUG::2014-04-30 10:39:20,258::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30304::DEBUG::2014-04-30 10:39:20,295::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30305::DEBUG::2014-04-30 10:39:20,537::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {}
>Thread-30305::ERROR::2014-04-30 10:39:20,538::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30305::DEBUG::2014-04-30 10:39:20,538::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30307::DEBUG::2014-04-30 10:39:21,081::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [4f88e7c2]
>Thread-30307::DEBUG::2014-04-30 10:39:21,082::task::579::TaskManager.Task::(_updateState) Task=`9e256988-77d4-43a7-8f0e-33686ddb26e1`::moving from state init -> state preparing
>Thread-30307::INFO::2014-04-30 10:39:21,082::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}, {'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}], options=None)
>Thread-30307::DEBUG::2014-04-30 10:39:21,086::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30307::ERROR::2014-04-30 10:41:26,127::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30307::ERROR::2014-04-30 10:41:26,128::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30307::DEBUG::2014-04-30 10:41:26,128::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30307::DEBUG::2014-04-30 10:41:26,132::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30307::DEBUG::2014-04-30 10:41:26,134::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30307::DEBUG::2014-04-30 10:41:26,134::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30307::DEBUG::2014-04-30 10:41:26,138::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30308::DEBUG::2014-04-30 10:42:21,719::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30308::DEBUG::2014-04-30 10:42:21,720::task::579::TaskManager.Task::(_updateState) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::moving from state init -> state preparing
>Thread-30308::INFO::2014-04-30 10:42:21,721::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30308::DEBUG::2014-04-30 10:42:21,721::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f6dd58f1-0151-42e1-ac7e-e364b90fdd5f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30308::DEBUG::2014-04-30 10:42:21,721::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30308::DEBUG::2014-04-30 10:42:21,722::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30308::DEBUG::2014-04-30 10:42:21,722::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f6dd58f1-0151-42e1-ac7e-e364b90fdd5f`::Granted request
>Thread-30308::DEBUG::2014-04-30 10:42:21,722::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`324f6f97-5ab4-4076-874f-b6bb40895a66`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30308::DEBUG::2014-04-30 10:42:21,723::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30308::DEBUG::2014-04-30 10:42:21,723::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30308::DEBUG::2014-04-30 10:42:21,723::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`324f6f97-5ab4-4076-874f-b6bb40895a66`::Granted request
>Thread-30308::INFO::2014-04-30 10:42:21,725::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30308::DEBUG::2014-04-30 10:42:21,725::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30308::DEBUG::2014-04-30 10:42:21,726::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30308::DEBUG::2014-04-30 10:42:21,726::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30308::DEBUG::2014-04-30 10:42:21,726::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30308::DEBUG::2014-04-30 10:42:21,726::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30308::DEBUG::2014-04-30 10:42:21,726::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30308::DEBUG::2014-04-30 10:42:21,727::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30308::DEBUG::2014-04-30 10:42:21,727::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30308::DEBUG::2014-04-30 10:42:21,727::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30308::DEBUG::2014-04-30 10:42:21,727::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30308::DEBUG::2014-04-30 10:42:21,727::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30308::DEBUG::2014-04-30 10:42:21,749::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30308::DEBUG::2014-04-30 10:42:21,750::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30308::DEBUG::2014-04-30 10:42:21,751::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30308::DEBUG::2014-04-30 10:42:21,751::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30308::DEBUG::2014-04-30 10:42:21,751::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30308::DEBUG::2014-04-30 10:42:21,752::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30308::DEBUG::2014-04-30 10:42:21,752::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30308::DEBUG::2014-04-30 10:42:21,752::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30308::DEBUG::2014-04-30 10:42:21,752::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30308::DEBUG::2014-04-30 10:42:21,753::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30308::ERROR::2014-04-30 10:42:21,753::task::850::TaskManager.Task::(_setError) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30308::DEBUG::2014-04-30 10:42:21,754::task::869::TaskManager.Task::(_run) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::Task._run: 69861b12-6e17-4181-8da9-4aeac9e7a3c3 ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30308::DEBUG::2014-04-30 10:42:21,754::task::1194::TaskManager.Task::(stop) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::stopping in state preparing (force False)
>Thread-30308::DEBUG::2014-04-30 10:42:21,755::task::974::TaskManager.Task::(_decref) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::ref 1 aborting True
>Thread-30308::INFO::2014-04-30 10:42:21,755::task::1151::TaskManager.Task::(prepare) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30308::DEBUG::2014-04-30 10:42:21,755::task::1156::TaskManager.Task::(prepare) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30308::DEBUG::2014-04-30 10:42:21,755::task::974::TaskManager.Task::(_decref) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::ref 0 aborting True
>Thread-30308::DEBUG::2014-04-30 10:42:21,756::task::909::TaskManager.Task::(_doAbort) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::Task._doAbort: force False
>Thread-30308::DEBUG::2014-04-30 10:42:21,756::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30308::DEBUG::2014-04-30 10:42:21,756::task::579::TaskManager.Task::(_updateState) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::moving from state preparing -> state aborting
>Thread-30308::DEBUG::2014-04-30 10:42:21,756::task::534::TaskManager.Task::(__state_aborting) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::_aborting: recover policy none
>Thread-30308::DEBUG::2014-04-30 10:42:21,756::task::579::TaskManager.Task::(_updateState) Task=`69861b12-6e17-4181-8da9-4aeac9e7a3c3`::moving from state aborting -> state failed
>Thread-30308::DEBUG::2014-04-30 10:42:21,757::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30308::DEBUG::2014-04-30 10:42:21,757::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30308::ERROR::2014-04-30 10:42:21,757::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30307::ERROR::2014-04-30 10:43:31,180::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30307::ERROR::2014-04-30 10:43:31,181::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30307::DEBUG::2014-04-30 10:43:31,181::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30307::INFO::2014-04-30 10:43:31,182::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}, {'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}]}
>Thread-30307::DEBUG::2014-04-30 10:43:31,182::task::1168::TaskManager.Task::(prepare) Task=`9e256988-77d4-43a7-8f0e-33686ddb26e1`::finished: {'statuslist': [{'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}, {'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}]}
>Thread-30307::DEBUG::2014-04-30 10:43:31,182::task::579::TaskManager.Task::(_updateState) Task=`9e256988-77d4-43a7-8f0e-33686ddb26e1`::moving from state preparing -> state finished
>Thread-30307::DEBUG::2014-04-30 10:43:31,182::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30307::DEBUG::2014-04-30 10:43:31,183::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30307::DEBUG::2014-04-30 10:43:31,183::task::974::TaskManager.Task::(_decref) Task=`9e256988-77d4-43a7-8f0e-33686ddb26e1`::ref 0 aborting False
>Thread-30309::DEBUG::2014-04-30 10:44:17,262::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [24c9d654]
>Thread-30309::DEBUG::2014-04-30 10:44:17,263::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30309::DEBUG::2014-04-30 10:44:17,277::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30309::DEBUG::2014-04-30 10:44:17,314::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30310::DEBUG::2014-04-30 10:44:17,546::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [24c9d654]
>Thread-30310::ERROR::2014-04-30 10:44:17,547::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30310::DEBUG::2014-04-30 10:44:17,547::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30311::DEBUG::2014-04-30 10:44:19,281::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {}
>Thread-30311::DEBUG::2014-04-30 10:44:19,282::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30311::DEBUG::2014-04-30 10:44:19,295::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30311::DEBUG::2014-04-30 10:44:19,332::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30312::DEBUG::2014-04-30 10:44:19,569::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {}
>Thread-30312::ERROR::2014-04-30 10:44:19,570::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30312::DEBUG::2014-04-30 10:44:19,570::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30314::DEBUG::2014-04-30 10:44:20,084::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [78b42ea0]
>Thread-30314::DEBUG::2014-04-30 10:44:20,085::task::579::TaskManager.Task::(_updateState) Task=`bdade6e9-c003-4b21-846d-93d83591c407`::moving from state init -> state preparing
>Thread-30314::INFO::2014-04-30 10:44:20,085::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}, {'connection': '10.34.1.251:/mnt/export2', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '8925e659-8b90-40f7-8e79-184a928c496f', 'port': ''}], options=None)
>Thread-30314::DEBUG::2014-04-30 10:44:20,090::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30314::ERROR::2014-04-30 10:46:25,132::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30314::ERROR::2014-04-30 10:46:25,133::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30314::DEBUG::2014-04-30 10:46:25,134::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30314::DEBUG::2014-04-30 10:46:25,138::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30314::DEBUG::2014-04-30 10:46:25,139::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30314::DEBUG::2014-04-30 10:46:25,139::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30314::DEBUG::2014-04-30 10:46:25,144::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/export2 /rhev/data-center/mnt/10.34.1.251:_mnt_export2' (cwd None)
>Thread-30315::DEBUG::2014-04-30 10:47:20,264::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30315::DEBUG::2014-04-30 10:47:20,265::task::579::TaskManager.Task::(_updateState) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::moving from state init -> state preparing
>Thread-30315::INFO::2014-04-30 10:47:20,265::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30315::DEBUG::2014-04-30 10:47:20,266::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`206586c1-5278-4c57-8b27-d6266c5dbb0e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30315::DEBUG::2014-04-30 10:47:20,266::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30315::DEBUG::2014-04-30 10:47:20,266::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30315::DEBUG::2014-04-30 10:47:20,267::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`206586c1-5278-4c57-8b27-d6266c5dbb0e`::Granted request
>Thread-30315::DEBUG::2014-04-30 10:47:20,267::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`70ac7b3d-bb35-4b4b-8078-108ab15d595b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30315::DEBUG::2014-04-30 10:47:20,268::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30315::DEBUG::2014-04-30 10:47:20,268::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30315::DEBUG::2014-04-30 10:47:20,268::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`70ac7b3d-bb35-4b4b-8078-108ab15d595b`::Granted request
>Thread-30315::INFO::2014-04-30 10:47:20,269::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30315::DEBUG::2014-04-30 10:47:20,269::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30315::DEBUG::2014-04-30 10:47:20,270::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30315::DEBUG::2014-04-30 10:47:20,270::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30315::DEBUG::2014-04-30 10:47:20,270::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30315::DEBUG::2014-04-30 10:47:20,271::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30315::DEBUG::2014-04-30 10:47:20,271::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30315::DEBUG::2014-04-30 10:47:20,271::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30315::DEBUG::2014-04-30 10:47:20,271::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30315::DEBUG::2014-04-30 10:47:20,272::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30315::DEBUG::2014-04-30 10:47:20,272::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30315::DEBUG::2014-04-30 10:47:20,272::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30315::DEBUG::2014-04-30 10:47:20,296::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30315::DEBUG::2014-04-30 10:47:20,297::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30315::DEBUG::2014-04-30 10:47:20,297::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30315::DEBUG::2014-04-30 10:47:20,298::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30315::DEBUG::2014-04-30 10:47:20,298::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30315::DEBUG::2014-04-30 10:47:20,298::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30315::DEBUG::2014-04-30 10:47:20,299::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30315::DEBUG::2014-04-30 10:47:20,299::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30315::DEBUG::2014-04-30 10:47:20,299::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30315::DEBUG::2014-04-30 10:47:20,299::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30315::ERROR::2014-04-30 10:47:20,300::task::850::TaskManager.Task::(_setError) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30315::DEBUG::2014-04-30 10:47:20,301::task::869::TaskManager.Task::(_run) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::Task._run: 3f2d7df3-7741-46bf-a340-563d3c51216a ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30315::DEBUG::2014-04-30 10:47:20,302::task::1194::TaskManager.Task::(stop) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::stopping in state preparing (force False)
>Thread-30315::DEBUG::2014-04-30 10:47:20,302::task::974::TaskManager.Task::(_decref) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::ref 1 aborting True
>Thread-30315::INFO::2014-04-30 10:47:20,302::task::1151::TaskManager.Task::(prepare) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30315::DEBUG::2014-04-30 10:47:20,303::task::1156::TaskManager.Task::(prepare) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30315::DEBUG::2014-04-30 10:47:20,303::task::974::TaskManager.Task::(_decref) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::ref 0 aborting True
>Thread-30315::DEBUG::2014-04-30 10:47:20,303::task::909::TaskManager.Task::(_doAbort) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::Task._doAbort: force False
>Thread-30315::DEBUG::2014-04-30 10:47:20,303::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30315::DEBUG::2014-04-30 10:47:20,303::task::579::TaskManager.Task::(_updateState) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::moving from state preparing -> state aborting
>Thread-30315::DEBUG::2014-04-30 10:47:20,304::task::534::TaskManager.Task::(__state_aborting) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::_aborting: recover policy none
>Thread-30315::DEBUG::2014-04-30 10:47:20,304::task::579::TaskManager.Task::(_updateState) Task=`3f2d7df3-7741-46bf-a340-563d3c51216a`::moving from state aborting -> state failed
>Thread-30315::DEBUG::2014-04-30 10:47:20,304::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30315::DEBUG::2014-04-30 10:47:20,304::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30315::ERROR::2014-04-30 10:47:20,305::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory
>Thread-30316::DEBUG::2014-04-30 10:48:25,061::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {} flowID [5aba1b3a]
>Thread-30316::DEBUG::2014-04-30 10:48:25,062::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30316::DEBUG::2014-04-30 10:48:25,076::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30316::DEBUG::2014-04-30 10:48:25,115::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30317::DEBUG::2014-04-30 10:48:25,374::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {} flowID [5aba1b3a]
>Thread-30317::ERROR::2014-04-30 10:48:25,375::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30317::DEBUG::2014-04-30 10:48:25,375::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30318::DEBUG::2014-04-30 10:48:26,650::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getCapabilities with () {}
>Thread-30318::DEBUG::2014-04-30 10:48:26,651::utils::505::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-30318::DEBUG::2014-04-30 10:48:26,665::utils::525::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-30318::DEBUG::2014-04-30 10:48:26,703::BindingXMLRPC::982::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.5.1.el6.x86_64', 'buildtime': 1389361603.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1385990636L, 'version': '0.12.4'}, 'vdsm': {'release': '0.13.el6ev', 'buildtime': 1395572343L, 'version': '4.13.2'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1395830257L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1397214683L, 'version': '0.12.1.2'}, 'mom': {'release': '8.el6ev', 'buildtime': 1385547925L, 'version': '0.3.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz', 'hooks': {'before_vm_start': {'50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'gateway': '10.34.27.254', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'qosInbound': '', 'qosOutbound': '', 'mtu': '1500', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'ports': ['eth2']}}, 'bridges': {'rhevm': {'addr': '10.34.27.209', 'cfg': {'IPV6_AUTOCONF': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'yes'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64', '2620:52:0:221a:21c:c4ff:fe38:af16/64'], 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ipv6gateway': 'fe80::6273:5cff:fe85:d6c5', 'gateway': '10.34.27.254', 'ports': ['eth2']}}, 'uuid': 'A9ED61F2-B5C6-DC11-BBDA-C438AF16001C', 'lastClientIface': 'rhevm', 'nics': {'eth2': {'netmask': '', 'addr': '', 'hwaddr': '00:1c:c4:38:af:16', 'cfg': {'BRIDGE': 'rhevm', 'IPV6INIT': 'yes', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'yes', 'HWADDR': '00:1C:C4:38:AF:16', 'DEVICE': 'eth2', 'ONBOOT': 'yes', 'UUID': '8ad68aad-1317-4427-901c-365219d32f52'}, 'ipv6addrs': ['fe80::21c:c4ff:fe38:af16/64'], 'speed': 100, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:10:18:30:bd:4c', 'cfg': {'UUID': '1cb0d8fe-54d9-4dce-b7fd-838c5c4a4177', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:10:18:30:BD:4C', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}}, 'software_revision': '0.13', 'management_ip': '0.0.0.0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,lahf_lm,dts,tpr_shadow,model_Conroe,model_coreduo,model_core2duo,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:2f1dc6d4a3c0', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.13', 'memSize': '3830', 'cpuSpeed': '2133.000', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '2', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.5.0.1.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.34.57.206'}}
>Thread-30319::DEBUG::2014-04-30 10:48:26,935::BindingXMLRPC::975::vds::(wrapper) client [10.34.57.206]::call getHardwareInfo with () {}
>Thread-30319::ERROR::2014-04-30 10:48:26,936::API::1180::vds::(getHardwareInfo) failed to retrieve hardware info
>Traceback (most recent call last):
>  File "/usr/share/vdsm/API.py", line 1177, in getHardwareInfo
>    hw = supervdsm.getProxy().getHardwareInfo()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in getHardwareInfo
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30319::DEBUG::2014-04-30 10:48:26,936::BindingXMLRPC::982::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Failed to read hardware information', 'code': 57}}
>Thread-30321::DEBUG::2014-04-30 10:48:27,436::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206] flowID [5eb88fd7]
>Thread-30321::DEBUG::2014-04-30 10:48:27,437::task::579::TaskManager.Task::(_updateState) Task=`cab3a818-658f-476f-b9d9-b9fdee8ec57b`::moving from state init -> state preparing
>Thread-30321::INFO::2014-04-30 10:48:27,437::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='f03e56af-56f8-4150-9517-271bec2305d9', conList=[{'connection': '10.34.1.251:/mnt/storage2', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2', 'port': ''}, {'connection': '10.34.27.209:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '', 'password': '******', 'id': '854c3686-561c-4079-af7a-2115dab8d0b5', 'port': ''}], options=None)
>Thread-30321::DEBUG::2014-04-30 10:48:27,441::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.34.1.251:/mnt/storage2 /rhev/data-center/mnt/10.34.1.251:_mnt_storage2' (cwd None)
>Thread-30314::ERROR::2014-04-30 10:48:30,187::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30314::ERROR::2014-04-30 10:48:30,188::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30314::DEBUG::2014-04-30 10:48:30,188::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30314::INFO::2014-04-30 10:48:30,188::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}, {'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}]}
>Thread-30314::DEBUG::2014-04-30 10:48:30,189::task::1168::TaskManager.Task::(prepare) Task=`bdade6e9-c003-4b21-846d-93d83591c407`::finished: {'statuslist': [{'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}, {'status': 477, 'id': '8925e659-8b90-40f7-8e79-184a928c496f'}]}
>Thread-30314::DEBUG::2014-04-30 10:48:30,189::task::579::TaskManager.Task::(_updateState) Task=`bdade6e9-c003-4b21-846d-93d83591c407`::moving from state preparing -> state finished
>Thread-30314::DEBUG::2014-04-30 10:48:30,189::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30314::DEBUG::2014-04-30 10:48:30,189::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30314::DEBUG::2014-04-30 10:48:30,190::task::974::TaskManager.Task::(_decref) Task=`bdade6e9-c003-4b21-846d-93d83591c407`::ref 0 aborting False
>Thread-30321::ERROR::2014-04-30 10:50:32,495::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30321::ERROR::2014-04-30 10:50:32,505::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Connection timed out\n')
>Thread-30321::DEBUG::2014-04-30 10:50:32,505::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30321::DEBUG::2014-04-30 10:50:32,511::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.27.209:_mnt_storage
>Thread-30321::DEBUG::2014-04-30 10:50:32,512::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b115b57b-7418-4b60-86b2-64af22fcd1db',)
>Thread-30321::DEBUG::2014-04-30 10:50:32,513::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {b115b57b-7418-4b60-86b2-64af22fcd1db: storage.nfsSD.findDomain, e4aced3b-4048-4151-9a29-3e917e9ed8fd: storage.localFsSD.findDomain, 6ba7421b-4b80-4aff-ba01-1c08d3ef4619: storage.nfsSD.findDomain, c0532ddb-5faf-4ab6-b93f-c490c588cae4: storage.nfsSD.findDomain}
>Thread-30321::INFO::2014-04-30 10:50:32,513::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30321::DEBUG::2014-04-30 10:50:32,513::task::1168::TaskManager.Task::(prepare) Task=`cab3a818-658f-476f-b9d9-b9fdee8ec57b`::finished: {'statuslist': [{'status': 477, 'id': '19363eb9-2267-4f2b-9c03-2035530f41d2'}, {'status': 0, 'id': '854c3686-561c-4079-af7a-2115dab8d0b5'}]}
>Thread-30321::DEBUG::2014-04-30 10:50:32,513::task::579::TaskManager.Task::(_updateState) Task=`cab3a818-658f-476f-b9d9-b9fdee8ec57b`::moving from state preparing -> state finished
>Thread-30321::DEBUG::2014-04-30 10:50:32,514::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30321::DEBUG::2014-04-30 10:50:32,514::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30321::DEBUG::2014-04-30 10:50:32,514::task::974::TaskManager.Task::(_decref) Task=`cab3a818-658f-476f-b9d9-b9fdee8ec57b`::ref 0 aborting False
>Thread-30322::DEBUG::2014-04-30 10:50:32,971::BindingXMLRPC::167::vds::(wrapper) client [10.34.57.206]
>Thread-30322::DEBUG::2014-04-30 10:50:32,972::task::579::TaskManager.Task::(_updateState) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::moving from state init -> state preparing
>Thread-30322::INFO::2014-04-30 10:50:32,972::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='f03e56af-56f8-4150-9517-271bec2305d9', hostID=2, scsiKey='f03e56af-56f8-4150-9517-271bec2305d9', msdUUID='b115b57b-7418-4b60-86b2-64af22fcd1db', masterVersion=1, options=None)
>Thread-30322::DEBUG::2014-04-30 10:50:32,973::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`37a3e1d4-aaed-41ea-aa5b-c17db7635e1a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1013' at 'connectStoragePool'
>Thread-30322::DEBUG::2014-04-30 10:50:32,973::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-30322::DEBUG::2014-04-30 10:50:32,973::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-30322::DEBUG::2014-04-30 10:50:32,973::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`37a3e1d4-aaed-41ea-aa5b-c17db7635e1a`::Granted request
>Thread-30322::DEBUG::2014-04-30 10:50:32,974::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`ad345262-bff8-4eb4-a9d2-2d40bbb719a9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1044' at '_connectStoragePool'
>Thread-30322::DEBUG::2014-04-30 10:50:32,974::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' for lock type 'exclusive'
>Thread-30322::DEBUG::2014-04-30 10:50:32,975::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free. Now locking as 'exclusive' (1 active user)
>Thread-30322::DEBUG::2014-04-30 10:50:32,975::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.f03e56af-56f8-4150-9517-271bec2305d9`ReqID=`ad345262-bff8-4eb4-a9d2-2d40bbb719a9`::Granted request
>Thread-30322::INFO::2014-04-30 10:50:32,975::sp::674::Storage.StoragePool::(connect) Connect host #2 to the storage pool f03e56af-56f8-4150-9517-271bec2305d9 with master domain: b115b57b-7418-4b60-86b2-64af22fcd1db (ver = 1)
>Thread-30322::DEBUG::2014-04-30 10:50:32,976::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30322::DEBUG::2014-04-30 10:50:32,977::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30322::DEBUG::2014-04-30 10:50:32,977::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30322::DEBUG::2014-04-30 10:50:32,977::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30322::DEBUG::2014-04-30 10:50:32,977::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-30322::DEBUG::2014-04-30 10:50:32,978::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-30322::DEBUG::2014-04-30 10:50:32,978::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-30322::DEBUG::2014-04-30 10:50:32,978::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30322::DEBUG::2014-04-30 10:50:32,978::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-30322::DEBUG::2014-04-30 10:50:32,979::misc::811::SamplingMethod::(__call__) Got in to sampling method
>Thread-30322::DEBUG::2014-04-30 10:50:32,979::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-30322::DEBUG::2014-04-30 10:50:33,004::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21
>Thread-30322::DEBUG::2014-04-30 10:50:33,005::misc::819::SamplingMethod::(__call__) Returning last result
>Thread-30322::DEBUG::2014-04-30 10:50:33,005::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9'
>Thread-30322::DEBUG::2014-04-30 10:50:33,006::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' (0 active users)
>Thread-30322::DEBUG::2014-04-30 10:50:33,006::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9' is free, finding out if anyone is waiting for it.
>Thread-30322::DEBUG::2014-04-30 10:50:33,006::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f03e56af-56f8-4150-9517-271bec2305d9', Clearing records.
>Thread-30322::DEBUG::2014-04-30 10:50:33,007::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-30322::DEBUG::2014-04-30 10:50:33,007::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-30322::DEBUG::2014-04-30 10:50:33,007::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-30322::DEBUG::2014-04-30 10:50:33,007::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-30322::ERROR::2014-04-30 10:50:33,008::task::850::TaskManager.Task::(_setError) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 857, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1015, in connectStoragePool
>    masterVersion, options)
>  File "/usr/share/vdsm/storage/hsm.py", line 1069, in _connectStoragePool
>    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 692, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1238, in __rebuild
>    masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain
>    domain = sdCache.produce(msdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 120, in _realProduce
>    self.refreshStorage()
>  File "/usr/share/vdsm/storage/misc.py", line 828, in helper
>    return sm(*args, **kwargs)
>  File "/usr/share/vdsm/storage/misc.py", line 813, in __call__
>    self.__lastResult = self.__func(*args, **kwargs)
>  File "/usr/share/vdsm/storage/sdc.py", line 83, in refreshStorage
>    multipath.rescan()
>  File "/usr/share/vdsm/storage/multipath.py", line 108, in rescan
>    supervdsm.getProxy().forceScsiScan()
>  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>    return callMethod()
>  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>    **kwargs)
>  File "<string>", line 2, in forceScsiScan
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
>    self._connect()
>  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
>    conn = self._Client(self._token.address, authkey=self._authkey)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 143, in Client
>    c = SocketClient(address)
>  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 263, in SocketClient
>    s.connect(address)
>  File "<string>", line 1, in connect
>error: [Errno 2] No such file or directory
>Thread-30322::DEBUG::2014-04-30 10:50:33,009::task::869::TaskManager.Task::(_run) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::Task._run: 9550303f-342d-48f0-86da-6dd0c61cff0f ('f03e56af-56f8-4150-9517-271bec2305d9', 2, 'f03e56af-56f8-4150-9517-271bec2305d9', 'b115b57b-7418-4b60-86b2-64af22fcd1db', 1) {} failed - stopping task
>Thread-30322::DEBUG::2014-04-30 10:50:33,009::task::1194::TaskManager.Task::(stop) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::stopping in state preparing (force False)
>Thread-30322::DEBUG::2014-04-30 10:50:33,009::task::974::TaskManager.Task::(_decref) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::ref 1 aborting True
>Thread-30322::INFO::2014-04-30 10:50:33,010::task::1151::TaskManager.Task::(prepare) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::aborting: Task is aborted: u'[Errno 2] No such file or directory' - code 100
>Thread-30322::DEBUG::2014-04-30 10:50:33,010::task::1156::TaskManager.Task::(prepare) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::Prepare: aborted: [Errno 2] No such file or directory
>Thread-30322::DEBUG::2014-04-30 10:50:33,010::task::974::TaskManager.Task::(_decref) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::ref 0 aborting True
>Thread-30322::DEBUG::2014-04-30 10:50:33,010::task::909::TaskManager.Task::(_doAbort) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::Task._doAbort: force False
>Thread-30322::DEBUG::2014-04-30 10:50:33,011::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30322::DEBUG::2014-04-30 10:50:33,011::task::579::TaskManager.Task::(_updateState) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::moving from state preparing -> state aborting
>Thread-30322::DEBUG::2014-04-30 10:50:33,011::task::534::TaskManager.Task::(__state_aborting) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::_aborting: recover policy none
>Thread-30322::DEBUG::2014-04-30 10:50:33,011::task::579::TaskManager.Task::(_updateState) Task=`9550303f-342d-48f0-86da-6dd0c61cff0f`::moving from state aborting -> state failed
>Thread-30322::DEBUG::2014-04-30 10:50:33,011::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-30322::DEBUG::2014-04-30 10:50:33,012::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-30322::ERROR::2014-04-30 10:50:33,012::dispatcher::70::Storage.Dispatcher.Protect::(run) [Errno 2] No such file or directory
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 62, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 1159, in prepare
>    raise self.error
>error: [Errno 2] No such file or directory

Actions: View

Attachments on bug 1092914: 891084 | 891086