Bug 1372199

Summary: VDSM service fails to start
Product: [oVirt] vdsm Reporter: Liron Aravot <laravot>
Component: CoreAssignee: Dan Kenigsberg <danken>
Status: CLOSED WORKSFORME QA Contact: Pavel Stehlik <pstehlik>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: ---CC: acanan, bugs, eterzella, fdeutsch, mperina, nsoffer, oourfali, tnisan
Target Milestone: ---Flags: oourfali: needinfo? (eterzella)
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1371217 Environment:
Last Closed: 2016-09-21 08:22:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Liron Aravot 2016-09-01 07:35:34 UTC 
+++ This bug was initially created as a clone of Bug #1371217 +++

Description of problem:

The storage domain was unavailable, and it is no longer possible to activate the Date (MASTER).

Version-Release number of selected component (if applicable):

Ovirt 3.6.3

How reproducible:

When click in DataCenter -> Storage -> Select Data(Master) -> Activate, 

Steps to Reproduce:
1. Select DataCenter
2. Select Storage
3. Select Data(Master)
4. Activate storage


Actual results:

The storage domain was unavailable, and it is no longer possible to activate the 


Expected results:

Activated storage domain

Additional info:

engine.log

016-08-29 09:05:34,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] hostFromVds::selectedVds - 'ovirt01', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 09:05:34,596 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'
2016-08-29 09:05:34,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] SPM selection - vds seems as spm 'ovirt03'
2016-08-29 09:05:34,600 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] spm vds is non responsive, stopping spm selection.
2016-08-29 09:05:46,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] hostFromVds::selectedVds - 'ovirt02', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 09:05:46,603 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'
2016-08-29 09:05:12,213 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [] SPM selection - vds seems as spm 'ovirt03'
2016-08-29 09:05:12,213 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-95) [] spm vds is non responsive, stopping spm selection.
2016-08-29 09:05:23,019 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] hostFromVds::selectedVds - 'ovirt02', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 09:05:23,023 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'
2016-08-29 09:05:23,026 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] SPM selection - vds seems as spm 'ovirt03'
2016-08-29 09:05:23,026 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] spm vds is non responsive, stopping spm selection.
2016-08-29 09:05:34,593 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] hostFromVds::selectedVds - 'ovirt01', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 09:05:34,596 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'
2016-08-29 09:05:34,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] SPM selection - vds seems as spm 'ovirt03'
2016-08-29 09:05:34,600 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-80) [] spm vds is non responsive, stopping spm selection.
2016-08-29 09:05:46,600 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] hostFromVds::selectedVds - 'ovirt02', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 09:05:46,603 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-27) [] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'


ovirt1

jsonrpc.Executor/7::DEBUG::2016-08-29 10:13:56,442::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}
jsonrpc.Executor/5::INFO::2016-08-29 10:14:07,217::logUtils::51::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}}
jsonrpc.Executor/5::DEBUG::2016-08-29 10:14:07,217::task::1191::Storage.TaskManager.Task::(prepare) Task=`e124d9ef-862b-4645-a14c-f8acbff32733`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}}
jsonrpc.Executor/5::DEBUG::2016-08-29 10:14:07,218::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}
jsonrpc.Executor/1::INFO::2016-08-29 10:14:40,211::logUtils::51::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}}
jsonrpc.Executor/1::DEBUG::2016-08-29 10:14:40,212::task::1191::Storage.TaskManager.Task::(prepare) Task=`a1541a52-f252-44ee-aae0-666989ad1dc0`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}}
jsonrpc.Executor/1::DEBUG::2016-08-29 10:14:40,213::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}
jsonrpc.Executor/2::INFO::2016-08-29 10:15:13,225::logUtils::51::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}}
jsonrpc.Executor/2::DEBUG::2016-08-29 10:15:13,225::task::1191::Storage.TaskManager.Task::(prepare) Task=`e9c8e722-98ff-4ec3-854a-db7fe9a44ebe`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}}
jsonrpc.Executor/2::DEBUG::2016-08-29 10:15:13,226::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 8L}


ovirt03

r': 8L}}
jsonrpc.Executor/3::DEBUG::2016-08-29 08:22:23,343::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}
jsonrpc.Executor/6::INFO::2016-08-29 08:22:34,395::logUtils::51::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}}
jsonrpc.Executor/6::DEBUG::2016-08-29 08:22:34,395::task::1191::Storage.TaskManager.Task::(prepare) Task=`87d99318-40f8-43a4-9ec9-1f0d82053255`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}}
jsonrpc.Executor/6::DEBUG::2016-08-29 08:22:34,396::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}
jsonrpc.Executor/1::INFO::2016-08-29 08:22:44,558::logUtils::51::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}}
jsonrpc.Executor/1::DEBUG::2016-08-29 08:22:44,559::task::1191::Storage.TaskManager.Task::(prepare) Task=`5b616859-6092-4353-b4a3-e48fa507bc46`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}}
jsonrpc.Executor/1::DEBUG::2016-08-29 08:22:44,559::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}
jsonrpc.Executor/2::INFO::2016-08-29 08:22:55,014::logUtils::51::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}}
jsonrpc.Executor/2::DEBUG::2016-08-29 08:22:55,014::task::1191::Storage.TaskManager.Task::(prepare) Task=`82244e0e-5d10-42f7-9363-92da7658855a`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}}
jsonrpc.Executor/2::DEBUG::2016-08-29 08:22:55,014::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 8L}

--- Additional comment from Fabian Deutsch on 2016-08-29 11:25:18 EDT ---

Can you please provide more logs from the involved machiens?
At best the sosreport.

--- Additional comment from Eduardo Terzella on 2016-08-29 12:06:07 EDT ---


No problens, whats logs you need ?

More information

I have 4 servers

- ovirt1
- ovirt2
- ovirt3
- ovirtengine

[root@ovirtengine ovirt-engine]# rpm -qa | grep ovirt
ovirt-engine-sdk-python-3.6.3.0-1.el7.noarch
ovirt-engine-extension-aaa-jdbc-1.0.6-1.el7.noarch
ovirt-engine-setup-3.6.3.4-1.el7.centos.noarch
ovirt-vmconsole-proxy-1.0.0-1.el7.centos.noarch
ovirt-host-deploy-java-1.4.1-1.el7.centos.noarch
ovirt-engine-dbscripts-3.6.3.4-1.el7.centos.noarch
ovirt-engine-3.6.3.4-1.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-3.6.3.4-1.el7.centos.noarch
ovirt-engine-wildfly-overlay-8.0.4-1.el7.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-3.6.3.4-1.el7.centos.noarch
ovirt-engine-extensions-api-impl-3.6.3.4-1.el7.centos.noarch
ovirt-iso-uploader-3.6.0-1.el7.centos.noarch
ovirt-engine-userportal-3.6.3.4-1.el7.centos.noarch
ovirt-engine-backend-3.6.3.4-1.el7.centos.noarch
ovirt-engine-lib-3.6.3.4-1.el7.centos.noarch
ovirt-engine-setup-base-3.6.3.4-1.el7.centos.noarch
ovirt-engine-setup-plugin-websocket-proxy-3.6.3.4-1.el7.centos.noarch
ovirt-engine-websocket-proxy-3.6.3.4-1.el7.centos.noarch
ovirt-engine-vmconsole-proxy-helper-3.6.3.4-1.el7.centos.noarch
ovirt-engine-wildfly-8.2.1-1.el7.x86_64
ovirt-engine-tools-3.6.3.4-1.el7.centos.noarch
ovirt-engine-restapi-3.6.3.4-1.el7.centos.noarch
ovirt-release35-006-1.noarch
ovirt-release36-003-1.noarch
ovirt-image-uploader-3.6.0-1.el7.centos.noarch
ovirt-engine-cli-3.6.2.0-1.el7.centos.noarch
ovirt-engine-jboss-as-7.1.1-1.el7.x86_64
ovirt-setup-lib-1.0.1-1.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-3.6.3.4-1.el7.centos.noarch
ovirt-vmconsole-1.0.0-1.el7.centos.noarch
ovirt-host-deploy-1.4.1-1.el7.centos.noarch
ovirt-engine-webadmin-portal-3.6.3.4-1.el7.centos.noarch

[root@ovirt03 vdsm]# rpm -qa | grep vdsm
vdsm-hook-vmfex-dev-4.17.23-1.el7.noarch
vdsm-python-4.17.23-1.el7.noarch
vdsm-4.17.23-1.el7.noarch
vdsm-hook-openstacknet-4.17.23-1.el7.noarch
vdsm-xmlrpc-4.17.23-1.el7.noarch
vdsm-yajsonrpc-4.17.23-1.el7.noarch
vdsm-jsonrpc-4.17.23-1.el7.noarch
vdsm-hook-macspoof-4.17.23-1.el7.noarch
vdsm-cli-4.17.23-1.el7.noarch
vdsm-infra-4.17.23-1.el7.noarch

--- Additional comment from Fabian Deutsch on 2016-08-29 12:14:16 EDT ---

The sosreport logs would be nice

Just run

$ sosreport

on the commandline to get them packaged

You need to run this on each host

--- Additional comment from Eduardo Terzella on 2016-08-29 13:10:33 EDT ---

Thx, Fabian.

The size of files is too big.

251M on ovirt01, Where can I provide the files for you?

--- Additional comment from Eduardo Terzella on 2016-08-29 13:18:12 EDT ---

I believe the problem is due to the fact that the vdsmd ovirt3 which is currently the SPM, is not working properly.

[Root @ ovirt03 vdsmd] # netstat -an | grep 54321
[Root @ ovirt03 vdsmd] #

[root@ovirt03 vdsm]# netstat -an | grep 54321
[root@ovirt03 vdsm]#

[root@ovirt03 vdsm]# cat vdsmd.log
2016-08-29 12:13:55,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67123fd0
2016-08-29 12:13:55,607 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) [] hostFromVds::selectedVds - 'ovirt01.sfl.cloud1.stfcia.com.br', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 12:13:55,610 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) [] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'
2016-08-29 12:13:55,613 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) [] SPM selection - vds seems as spm 'ovirt03.sfl.cloud1.stfcia.com.br'
2016-08-29 12:13:55,613 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-40) [] spm vds is non responsive, stopping spm selection.

[root@ovirt03 vdsm]# cat mom.log

2016-08-29 12:12:46,404 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:13:01,421 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:13:16,435 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:13:31,451 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:13:46,467 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:14:01,483 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:14:16,489 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:14:31,506 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:14:46,517 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:15:01,532 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:15:16,548 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:15:31,565 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:15:46,577 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused
2016-08-29 12:16:01,593 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused

--- Additional comment from Eduardo Terzella on 2016-08-29 13:55:11 EDT ---

More logs, when try enable data(master) storage:

2016-08-29 12:51:13,812 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt03/192.168.200.3
2016-08-29 12:51:13,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) [] Command 'GetCapabilitiesVDSCommand(HostName = ovirt03, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='cdb61116-bfcd-4a13-a485-1338d67d5a44', vds='Host[ovirt03,cdb61116-bfcd-4a13-a485-1338d67d5a44]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:13,813 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-53) [] Failure to refresh Vds runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:13,813 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-53) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) [vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) [:1.7.0_95]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_95]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_95]
        at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
        at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:]
        ... 14 more

2016-08-29 12:51:14,612 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (default task-76) [6d189af] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ef211709-1b0b-4a0b-8dee-e63d75c6afea=<STORAGE, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-08-29 12:51:14,644 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] Running command: ActivateStorageDomainCommand internal: false. Entities affected :  ID: ef211709-1b0b-4a0b-8dee-e63d75c6afea Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2016-08-29 12:51:14,648 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] Lock freed to object 'EngineLock:{exclusiveLocks='[ef211709-1b0b-4a0b-8dee-e63d75c6afea=<STORAGE, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-08-29 12:51:14,648 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] ActivateStorage Domain. Before Connect all hosts to pool. Time: Mon Aug 29 12:51:14 CDT 2016
2016-08-29 12:51:14,655 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-16) [61438ca3] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2016-08-29 12:51:14,656 INFO  [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-8-thread-21) [2b818ec0] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2016-08-29 12:51:14,659 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [61438ca3] START, ConnectStorageServerVDSCommand(HostName = ovirt02, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='57ad6300-5c26-4501-a955-9c1ec440d57b', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='be56634c-48f4-4eba-8c4b-2ee32f95443f', connection='nfsdal0501d.service.softlayer.com:/IBM01SEV307364_1', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 77dd21eb
2016-08-29 12:51:14,660 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [2b818ec0] START, ConnectStorageServerVDSCommand(HostName = ovirt01, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a3fb5779-885f-43cf-a097-2af50023fa21', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='be56634c-48f4-4eba-8c4b-2ee32f95443f', connection='nfsdal0501d.service.softlayer.com:/IBM01SEV307364_1', iqn='null', vfsType='null', mountOptions='null', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 56de08c7
2016-08-29 12:51:14,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [61438ca3] FINISH, ConnectStorageServerVDSCommand, return: {be56634c-48f4-4eba-8c4b-2ee32f95443f=0}, log id: 77dd21eb
2016-08-29 12:51:15,361 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt03/192.168.200.3
2016-08-29 12:51:15,361 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [2b818ec0] FINISH, ConnectStorageServerVDSCommand, return: {be56634c-48f4-4eba-8c4b-2ee32f95443f=0}, log id: 56de08c7
2016-08-29 12:51:15,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-70) [] Command 'ListVDSCommand(HostName = ovirt03, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='cdb61116-bfcd-4a13-a485-1338d67d5a44', vds='Host[ovirt03,cdb61116-bfcd-4a13-a485-1338d67d5a44]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:15,362 INFO  [org.ovirt.engine.core.vdsbroker.PollVmStatsRefresher] (DefaultQuartzScheduler_Worker-70) [] Failed to fetch vms info for host 'ovirt03' - skipping VMs monitoring.
2016-08-29 12:51:15,363 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='00000002-0002-0002-0002-000000000394', ignoreFailoverLimit='false', storageDomainId='ef211709-1b0b-4a0b-8dee-e63d75c6afea'}), log id: 1fcfb7b4
2016-08-29 12:51:15,384 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] START, ConnectStoragePoolVDSCommand(HostName = ovirt02, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='57ad6300-5c26-4501-a955-9c1ec440d57b', vdsId='57ad6300-5c26-4501-a955-9c1ec440d57b', storagePoolId='00000002-0002-0002-0002-000000000394', masterVersion='15'}), log id: 3382f334
2016-08-29 12:51:16,693 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] FINISH, ConnectStoragePoolVDSCommand, log id: 3382f334
2016-08-29 12:51:16,736 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [6d189af] hostFromVds::selectedVds - 'ovirt02', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 12:51:16,739 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [6d189af] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'
2016-08-29 12:51:16,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [6d189af] SPM selection - vds seems as spm 'ovirt03'
2016-08-29 12:51:16,743 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-42) [6d189af] spm vds is non responsive, stopping spm selection.
2016-08-29 12:51:16,743 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] FINISH, ActivateStorageDomainVDSCommand, log id: 1fcfb7b4
2016-08-29 12:51:16,743 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] Command 'org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand' failed: EngineException: Cannot allocate IRS server (Failed with error IRS_REPOSITORY_NOT_FOUND and code 5009)
2016-08-29 12:51:16,744 INFO  [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-42) [6d189af] Command [id=70d03cbd-455c-4665-9f7a-e6758ef0db11]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot:{id='StoragePoolIsoMapId:{storagePoolId='00000002-0002-0002-0002-000000000394', storageId='ef211709-1b0b-4a0b-8dee-e63d75c6afea'}', status='Unknown'}.
2016-08-29 12:51:16,748 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-42) [6d189af] Correlation ID: 6d189af, Job ID: d44588b3-7959-4b1e-a51f-3c11882d5b7a, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain IBM01SEV307364_1_vms_t1 (Data Center SFL_DALLAS_5) by admin@internal
2016-08-29 12:51:16,818 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt03/192.168.200.3
2016-08-29 12:51:16,819 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [] Command 'GetCapabilitiesVDSCommand(HostName = ovirt03, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='cdb61116-bfcd-4a13-a485-1338d67d5a44', vds='Host[ovirt03,cdb61116-bfcd-4a13-a485-1338d67d5a44]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:16,820 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-15) [] Failure to refresh Vds runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:16,820 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-15) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) [vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) [:1.7.0_95]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_95]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_95]
        at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
        at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:]
        ... 14 more

2016-08-29 12:51:18,361 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt03/192.168.200.3
2016-08-29 12:51:18,362 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-47) [] Command 'ListVDSCommand(HostName = ovirt03, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='cdb61116-bfcd-4a13-a485-1338d67d5a44', vds='Host[ovirt03,cdb61116-bfcd-4a13-a485-1338d67d5a44]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:18,362 INFO  [org.ovirt.engine.core.vdsbroker.PollVmStatsRefresher] (DefaultQuartzScheduler_Worker-47) [] Failed to fetch vms info for host 'ovirt03' - skipping VMs monitoring.
2016-08-29 12:51:18,862 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) [] START, ConnectStoragePoolVDSCommand(HostName = ovirt01, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a3fb5779-885f-43cf-a097-2af50023fa21', vdsId='a3fb5779-885f-43cf-a097-2af50023fa21', storagePoolId='00000002-0002-0002-0002-000000000394', masterVersion='15'}), log id: 5a95cd60
2016-08-29 12:51:19,825 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt03/192.168.200.3
2016-08-29 12:51:19,826 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a95cd60
2016-08-29 12:51:19,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) [] Command 'GetCapabilitiesVDSCommand(HostName = ovirt03, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='cdb61116-bfcd-4a13-a485-1338d67d5a44', vds='Host[ovirt03,cdb61116-bfcd-4a13-a485-1338d67d5a44]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:19,827 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-16) [] Failure to refresh Vds runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
2016-08-29 12:51:19,827 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-16) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) [vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) [:1.7.0_95]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_95]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_95]
        at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
        at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:]
        ... 14 more

2016-08-29 12:51:19,882 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [] hostFromVds::selectedVds - 'ovirt01', spmStatus 'Free', storage pool 'SFL_DALLAS_5', storage pool version '3.5'
2016-08-29 12:51:19,885 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [] SPM Init: could not find reported vds or not up - pool: 'SFL_DALLAS_5' vds_spm_id: '3'
2016-08-29 12:51:19,888 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [] SPM selection - vds seems as spm 'ovirt03'
2016-08-29 12:51:19,888 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [] spm vds is non responsive, stopping spm selection.
^C

--- Additional comment from Fabian Deutsch on 2016-08-29 14:43:07 EDT ---

Yes, those logs look helpful. Let me move it along to get it to the right people.

--- Additional comment from Fabian Deutsch on 2016-08-29 14:43:43 EDT ---

F

--- Additional comment from Eduardo Terzella on 2016-08-29 14:58:41 EDT ---

Thx, Fabian

--- Additional comment from Yaniv Kaul on 2016-08-30 04:35:18 EDT ---

Eduardo - any reason you have not upgraded to the latest 3.6? between 3.6.3 and 3.6.7 we have fixed many issues - some of it may be relevant here (did not look at the logs yet).

--- Additional comment from Eduardo Terzella on 2016-08-30 15:55:30 EDT ---

Has no problem I upgrade to version 3.6.7.

My current concern is:

My domain storage is no-operation. I believe it is due to the fact that the ovirt03 is currently the SPM is not with vdsmd working properly, you do not receive the service more connection on port 54321

[Root @ ovirt03 VDSM] # netstat -an | grep 54321
[Root @ ovirt03 VDSM] #

I believe that if you restart the vdsmd ovirt03 the problem will be solved. You see a problem restart the vdsmd the node that is currently responsible for the SPM?

Currently this environment is production, and has about 150 vms.

--- Additional comment from Liron Aravot on 2016-08-30 18:23:35 EDT ---

Hi Eduardo, 
Generally speaking- the spm host is responsible to extend the disks residing on block storage domains for the running vms in the data center. It uses the shared storage to communicate with the other hosts. When the spm has access to the shared storage even if it has no access from the ovirt engine, restarting the vdsm service might affect the relevant virtual machines if extension is needed for them during that time (or if the host doesn't come up fast).

Your data center isn't usable as the spm is non responsive from the ovirt-engine. The data center won't be usable till the connectivity with the spm host will be restored or till you'll confirm that this host has been rebooted (by right clicking on it and choosing "confirm host has been rebooted).

I'd be happy if you could share the vdsm.log [1] from the current spm host so we'll be able to first understand what's the current status of vdsm on that host. adding nsoffer as well.

Thanks,
Liron.

[1] /var/log/vdsm/vdsm.log

--- Additional comment from Liron Aravot on 2016-08-30 18:34:52 EDT ---

Just for the documentation here (obviously that's less relevant for your case) - restarting vdsm service/host will stop any long running storage tasks on that host.

--- Additional comment from Eduardo Terzella on 2016-08-30 20:07:36 EDT ---

Thx for replay.

I restarted the vsdmd the ovirt03 and SPM went to ovirt02 as expected, and the storage domain again become active.

However the ovirt03 can not start vsdmd, show presents problem in

/bin/sh /usr/libexec/vdsm/vdsmd_init_common.sh --pre-start

--- Additional comment from Eduardo Terzella on 2016-08-30 20:19:48 EDT ---

more information:

-- Logs begin at Wed 2016-05-25 00:36:11 CDT, end at Tue 2016-08-30 19:17:22 CDT. --
Aug 30 19:17:13 ovirt03 vdsmd_init_common.sh[13422]: vdsm: Running mkdirs
Aug 30 19:17:13 ovirt03 vdsmd_init_common.sh[13422]: vdsm: Running configure_coredump
Aug 30 19:17:13 ovirt03 vdsmd_init_common.sh[13422]: vdsm: Running configure_vdsm_logs
Aug 30 19:17:13 ovirt03 vdsmd_init_common.sh[13422]: vdsm: Running wait_for_network

--- Additional comment from Liron Aravot on 2016-09-01 03:33:57 EDT ---

Eduardo, as the initial issue was caused by an expected behavior - I'm cloning this bug to a new one that will tackle the vdsm start issues and closing this one.
Comment 1 Red Hat Bugzilla Rules Engine 2016-09-01 07:35:46 UTC 
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.
Comment 2 Liron Aravot 2016-09-01 07:38:45 UTC 
Please contact the original bug reporter (eterzella) for any needed information.
Comment 3 Oved Ourfali 2016-09-05 12:17:08 UTC 
Can you test with the latest 3.6?
Comment 4 Martin Perina 2016-09-21 08:22:48 UTC 
Closing now as we are not able to reproduce. Feel free to reopen if you are able to reproduce with latest 3.6 version. Thanks