Description of problem: Addition of another oVirt host results in installation failure with the following errors (evident on oVirt UI): VDSM OvirtHost8 command failed: Connection issue null. Failed to configure management network on host OvirtHost8 due to setup networks failure. Host OvirtHost8 installation failed. Failed to configure management network on the host. Status of host OvirtHost8 was set to NonOperational. -- OvirtHost8 is the auto-recovered and setup resumes. Version-Release number of selected component (if applicable): libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-engine-sdk-python-3.6.3.0-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.4.3-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.3.4-1.el7.centos.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch vdsm-4.17.23-0.el7.centos.noarch vdsm-cli-4.17.23-0.el7.centos.noarch vdsm-gluster-4.17.23-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.23-0.el7.centos.noarch vdsm-infra-4.17.23-0.el7.centos.noarch vdsm-jsonrpc-4.17.23-0.el7.centos.noarch vdsm-python-4.17.23-0.el7.centos.noarch vdsm-xmlrpc-4.17.23-0.el7.centos.noarch vdsm-yajsonrpc-4.17.23-0.el7.centos.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy initial oVirt host (non-interactive) 2. Deploy additional oVirt host (non-interactive) Actual results: 1. VDSM/network failure experienced. Expected results: 1. Additional oVirt host deploys cleanly. Additional info:
Created attachment 1134257 [details] host-deploy log
Created attachment 1134258 [details] engine.log
Created attachment 1134259 [details] Screen Capture
Created attachment 1134260 [details] pserver8:/var/log/vdsm/mom.log Additional node mom.log
Created attachment 1134261 [details] /var/log/vdsm/supervdsm.log Additional node supervdsm.log
Created attachment 1134262 [details] /var/log/vdsm/vdsm.log Additional node vdsm.log
*** Bug 1315823 has been marked as a duplicate of this bug. ***
Created attachment 1134309 [details] Screen Capture 1 Physical server stuck in "Installing" status
Also want to mention that I've been having trouble adding any additional nodes after the second one (e.g. node3, node4 have been stuck in the Installing state). Looking at the vdsm logs, it appears to be the same error.
Dan, can you take a look?
vdsm seems happy to define ovirtmgmt, which ends up with the same IP address as the original em1 jsonrpc.Executor/3::DEBUG::2016-03-08 11:46:03,695::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.setupNetworks' in bridge with {u'bondings': {}, u'networks': {u'ovirtmgmt': {u'nic': u'em1', u'mtu': u'1500', u'bootproto': u'dhcp', u'STP': u'no', u'bridged': u'true', u'defaultRoute': True}}, u'options': {u'connectivityCheck': u'true', u'connectivityTimeout': 120}} jsonrpc.Executor/3::DEBUG::2016-03-08 11:46:26,972::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.setupNetworks' in bridge with {'message': 'Done', 'code': 0} sourceRoute::INFO::2016-03-08 11:46:17,637::sourceroute::75::root::(configure) Configuring gateway - ip: 192.168.200.8, network: 192.168.200.0/24, subnet: 255.255.255.0, gateway: 192.168.200.254, table: 3232286728, device: ovirtmgmt However, Engine complains about 2016-03-08 11:46:06,028 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-10) [1cf83f39] Command 'PollVDSCommand(HostName = OvirtHost8, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='453c3ea9-359f-4d24-8af0-426fe5913048'})' execution failed: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request 2016-03-08 11:46:09,038 ERROR [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-10) [1cf83f39] Command 'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Connection issue null (Failed with error VDS_NETWORK_ERROR and code 5022) Martin, can you tell why did Engine use xmlrpc? and more importantly - why did it lose connectivity with the host? Finally - Charlie: would you attach /var/log/vdsm/connectivity.log from the host?
Dan, I didn't save the configuration after opening the ticket. I'll rebuild our environment and attach the connectivity log.
(In reply to Charlie Inglese from comment #12) > Dan, > > I didn't save the configuration after opening the ticket. I'll rebuild our > environment and attach the connectivity log. Closing with insufficient data for now. Please reopen when you can reproduce the issue and attach the needed logs.
*** This bug has been marked as a duplicate of bug 1320128 ***