Bug 1315824 - Addition of another oVirt host results in installation failure
Addition of another oVirt host results in installation failure
Status: CLOSED DUPLICATE of bug 1320128
Product: ovirt-host-deploy
Classification: oVirt
Component: Core (Show other bugs)
1.4.1
x86_64 Linux
unspecified Severity high (vote)
: ovirt-3.6.6
: ---
Assigned To: Dan Kenigsberg
Pavel Stehlik
:
: 1315823 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-08 12:22 EST by Charlie Inglese
Modified: 2016-05-09 08:38 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-18 12:39:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ylavi: ovirt‑3.6.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
host-deploy log (106.01 KB, text/plain)
2016-03-08 12:23 EST, Charlie Inglese
no flags Details
engine.log (5.57 MB, text/plain)
2016-03-08 12:23 EST, Charlie Inglese
no flags Details
Screen Capture (130.68 KB, image/jpeg)
2016-03-08 12:24 EST, Charlie Inglese
no flags Details
pserver8:/var/log/vdsm/mom.log (26.18 KB, text/plain)
2016-03-08 12:24 EST, Charlie Inglese
no flags Details
/var/log/vdsm/supervdsm.log (1.47 MB, text/plain)
2016-03-08 12:25 EST, Charlie Inglese
no flags Details
/var/log/vdsm/vdsm.log (8.04 MB, text/plain)
2016-03-08 12:26 EST, Charlie Inglese
no flags Details
Screen Capture 1 (82.22 KB, image/jpeg)
2016-03-08 15:45 EST, Charlie Inglese
no flags Details

  None (edit)
Description Charlie Inglese 2016-03-08 12:22:32 EST
Description of problem:

Addition of another oVirt host results in installation failure with the following errors (evident on oVirt UI):
VDSM OvirtHost8 command failed: Connection issue null.
Failed to configure management network on host OvirtHost8 due to setup networks failure.
Host OvirtHost8 installation failed. Failed to configure management network on the host.
Status of host OvirtHost8 was set to NonOperational.

-- OvirtHost8 is the auto-recovered and setup resumes.


Version-Release number of selected component (if applicable):

libgovirt-0.3.3-1.el7_2.1.x86_64
ovirt-engine-sdk-python-3.6.3.0-1.el7.centos.noarch
ovirt-host-deploy-1.4.1-1.el7.centos.noarch
ovirt-hosted-engine-ha-1.3.4.3-1.el7.centos.noarch
ovirt-hosted-engine-setup-1.3.3.4-1.el7.centos.noarch
ovirt-setup-lib-1.0.1-1.el7.centos.noarch
ovirt-vmconsole-1.0.0-1.el7.centos.noarch
ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch
vdsm-4.17.23-0.el7.centos.noarch
vdsm-cli-4.17.23-0.el7.centos.noarch
vdsm-gluster-4.17.23-0.el7.centos.noarch
vdsm-hook-vmfex-dev-4.17.23-0.el7.centos.noarch
vdsm-infra-4.17.23-0.el7.centos.noarch
vdsm-jsonrpc-4.17.23-0.el7.centos.noarch
vdsm-python-4.17.23-0.el7.centos.noarch
vdsm-xmlrpc-4.17.23-0.el7.centos.noarch
vdsm-yajsonrpc-4.17.23-0.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy initial oVirt host (non-interactive)
2. Deploy additional oVirt host (non-interactive)

Actual results:
1. VDSM/network failure experienced.

Expected results:
1. Additional oVirt host deploys cleanly.

Additional info:
Comment 1 Charlie Inglese 2016-03-08 12:23 EST
Created attachment 1134257 [details]
host-deploy log
Comment 2 Charlie Inglese 2016-03-08 12:23 EST
Created attachment 1134258 [details]
engine.log
Comment 3 Charlie Inglese 2016-03-08 12:24 EST
Created attachment 1134259 [details]
Screen Capture
Comment 4 Charlie Inglese 2016-03-08 12:24 EST
Created attachment 1134260 [details]
pserver8:/var/log/vdsm/mom.log

Additional node mom.log
Comment 5 Charlie Inglese 2016-03-08 12:25 EST
Created attachment 1134261 [details]
/var/log/vdsm/supervdsm.log

Additional node supervdsm.log
Comment 6 Charlie Inglese 2016-03-08 12:26 EST
Created attachment 1134262 [details]
/var/log/vdsm/vdsm.log

Additional node vdsm.log
Comment 7 Charlie Inglese 2016-03-08 12:27:18 EST
*** Bug 1315823 has been marked as a duplicate of this bug. ***
Comment 8 Charlie Inglese 2016-03-08 15:45 EST
Created attachment 1134309 [details]
Screen Capture 1

Physical server stuck in "Installing" status
Comment 9 Charlie Inglese 2016-03-08 16:23:03 EST
Also want to mention that I've been having trouble adding any additional nodes after the second one (e.g. node3, node4 have been stuck in the Installing state).

Looking at the vdsm logs, it appears to be the same error.
Comment 10 Sandro Bonazzola 2016-03-11 09:59:00 EST
Dan, can you take a look?
Comment 11 Dan Kenigsberg 2016-03-13 07:26:59 EDT
vdsm seems happy to define ovirtmgmt, which ends up with the same IP address as the original em1

jsonrpc.Executor/3::DEBUG::2016-03-08 11:46:03,695::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.setupNetworks' in bridge with {u'bondings': {}, u'networks': {u'ovirtmgmt': {u'nic': u'em1', u'mtu': u'1500', u'bootproto': u'dhcp', u'STP': u'no', u'bridged': u'true', u'defaultRoute': True}}, u'options': {u'connectivityCheck': u'true', u'connectivityTimeout': 120}}
jsonrpc.Executor/3::DEBUG::2016-03-08 11:46:26,972::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.setupNetworks' in bridge with {'message': 'Done', 'code': 0}

sourceRoute::INFO::2016-03-08 11:46:17,637::sourceroute::75::root::(configure) Configuring gateway - ip: 192.168.200.8, network: 192.168.200.0/24, subnet: 255.255.255.0, gateway: 192.168.200.254, table: 3232286728, device: ovirtmgmt

However, Engine complains about

2016-03-08 11:46:06,028 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-10) [1cf83f39] Command 'PollVDSCommand(HostName = OvirtHost8, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='453c3ea9-359f-4d24-8af0-426fe5913048'})' execution failed: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request

2016-03-08 11:46:09,038 ERROR [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (org.ovirt.thread.pool-8-thread-10) [1cf83f39] Command 'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Connection issue null (Failed with error VDS_NETWORK_ERROR and code 5022)

Martin, can you tell why did Engine use xmlrpc? and more importantly - why did it lose connectivity with the host?

Finally - Charlie: would you attach /var/log/vdsm/connectivity.log from the host?
Comment 12 Charlie Inglese 2016-03-16 09:33:31 EDT
Dan,

I didn't save the configuration after opening the ticket. I'll rebuild our environment and attach the connectivity log.
Comment 13 Sandro Bonazzola 2016-03-18 12:39:09 EDT
(In reply to Charlie Inglese from comment #12)
> Dan,
> 
> I didn't save the configuration after opening the ticket. I'll rebuild our
> environment and attach the connectivity log.

Closing with insufficient data for now. Please reopen when you can reproduce the issue and attach the needed logs.
Comment 14 Charlie Inglese 2016-05-09 08:36:49 EDT

*** This bug has been marked as a duplicate of bug 1320128 ***

Note You need to log in before you can comment on or make changes to this bug.