Bug 1569487 - Failed to add new host to cluster VDS_SET_NONOPERATIONAL_NETWORK, Failed to configure management network on the host.
Summary: Failed to add new host to cluster VDS_SET_NONOPERATIONAL_NETWORK, Failed to c...
Keywords:
Status: CLOSED DUPLICATE of bug 1570388
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Network
Version: 4.2.2.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Dominik Holler
QA Contact: Meni Yakove
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-19 11:42 UTC by Sergii Melnyk
Modified: 2018-04-25 07:23 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-25 07:23:23 UTC
oVirt Team: Network


Attachments (Terms of Use)
engine deploy log (412.47 KB, text/plain)
2018-04-19 11:42 UTC, Sergii Melnyk
no flags Details
supervsdm log from deployed host (14.90 KB, text/plain)
2018-04-19 11:43 UTC, Sergii Melnyk
no flags Details
complete engine log (739.34 KB, application/x-gzip)
2018-04-20 14:32 UTC, Sergii Melnyk
no flags Details
vdsm log (12.46 MB, text/plain)
2018-04-20 14:33 UTC, Sergii Melnyk
no flags Details

Description Sergii Melnyk 2018-04-19 11:42:20 UTC
Created attachment 1424019 [details]
engine deploy log

Description of problem:
Dears, 
After update ovirt from 4.2.1 to 4.2.2 I was try to add new host to cluster, but can't complete deploy installs to host with ERROR: Host trk-ovhv-4 installation failed. Failed to configure management network on the host.

Version-Release number of selected component (if applicable):
On ovirt-engine node:
ovirt-engine-tools-backup-4.2.2.5-1.el7.centos.noarch
ovirt-engine-tools-4.2.2.5-1.el7.centos.noarch
ovirt-engine-dwh-4.2.2.2-1.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.2.2.6-1.el7.centos.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.2.2.6-1.el7.centos.noarch
ovirt-ansible-cluster-upgrade-1.1.6-1.el7.centos.noarch
ovirt-ansible-manageiq-1.1.6-1.el7.centos.noarch
ovirt-engine-wildfly-11.0.0-1.el7.centos.x86_64
ovirt-engine-websocket-proxy-4.2.2.6-1.el7.centos.noarch
ovirt-engine-cli-3.6.9.2-1.el7.centos.noarch
ovirt-engine-extensions-api-impl-4.2.2.6-1.el7.centos.noarch
ovirt-ansible-repositories-1.1.0-1.el7.centos.noarch
ovirt-vmconsole-1.0.4-1.el7.noarch
ovirt-setup-lib-1.1.4-1.el7.centos.noarch
ovirt-engine-dwh-setup-4.2.2.2-1.el7.centos.noarch
ovirt-engine-extension-aaa-jdbc-1.1.7-1.el7.centos.noarch
ovirt-js-dependencies-1.2.0-3.1.el7.centos.noarch
python-ovirt-engine-sdk4-4.2.4-2.el7.centos.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch
ovirt-host-deploy-java-1.7.3-1.el7.centos.noarch
ovirt-iso-uploader-4.2.0-1.el7.centos.noarch
ovirt-engine-dbscripts-4.2.2.5-1.el7.centos.noarch
ovirt-engine-webadmin-portal-4.2.2.5-1.el7.centos.noarch
ovirt-engine-restapi-4.2.2.5-1.el7.centos.noarch
ovirt-engine-backend-4.2.2.5-1.el7.centos.noarch
ovirt-engine-lib-4.2.2.6-1.el7.centos.noarch
ovirt-engine-setup-base-4.2.2.6-1.el7.centos.noarch
ovirt-imageio-proxy-setup-1.2.2-0.el7.centos.noarch
ovirt-provider-ovn-1.2.9-1.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.2.2.6-1.el7.centos.noarch
ovirt-engine-vmconsole-proxy-helper-4.2.2.6-1.el7.centos.noarch
ovirt-ansible-vm-infra-1.1.5-1.el7.centos.noarch
ovirt-ansible-infra-1.1.4-1.el7.centos.noarch
ovirt-engine-metrics-1.1.3.4-1.el7.centos.noarch
ovirt-engine-wildfly-overlay-11.0.1-1.el7.centos.noarch
ovirt-imageio-proxy-1.2.2-0.el7.centos.noarch
ovirt-release42-4.2.2-3.el7.centos.noarch
ovirt-web-ui-1.3.7-2.el7.centos.noarch
ovirt-ansible-engine-setup-1.1.0-1.el7.centos.noarch
ovirt-ansible-image-template-1.1.5-1.el7.centos.noarch
ovirt-vmconsole-proxy-1.0.4-1.el7.noarch
ovirt-cockpit-sso-0.0.4-1.el7.noarch
ovirt-engine-api-explorer-0.0.2-1.el7.centos.noarch
ovirt-host-deploy-1.7.3-1.el7.centos.noarch
ovirt-engine-dashboard-1.2.2-3.el7.centos.noarch
ovirt-engine-4.2.2.5-1.el7.centos.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.2.2.6-1.el7.centos.noarch
ovirt-imageio-common-1.2.2-0.el7.centos.noarch
ovirt-engine-setup-4.2.2.6-1.el7.centos.noarch
ovirt-ansible-disaster-recovery-0.3-1.el7.centos.noarch
ovirt-ansible-roles-1.1.3-1.el7.centos.noarch

On new add host to cluster:
ovirt-vmconsole-1.0.4-1.el7.noarch
ovirt-imageio-common-1.2.2-0.el7.centos.noarch
ovirt-setup-lib-1.1.4-1.el7.centos.noarch
cockpit-ovirt-dashboard-0.11.20-1.el7.centos.noarch
ovirt-release42-4.2.2-3.el7.centos.noarch
ovirt-imageio-daemon-1.2.2-0.el7.centos.noarch
ovirt-host-4.2.2-2.el7.centos.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch
python-ovirt-engine-sdk4-4.2.4-2.el7.centos.x86_64
ovirt-host-deploy-1.7.3-1.el7.centos.noarch
ovirt-host-dependencies-4.2.2-2.el7.centos.x86_64
ovirt-hosted-engine-ha-2.2.10-1.el7.centos.noarch
ovirt-vmconsole-host-1.0.4-1.el7.noarch
ovirt-hosted-engine-setup-2.2.16-1.el7.centos.noarch
ovirt-provider-ovn-driver-1.2.9-1.el7.centos.noarch
vdsm-hook-vmfex-dev-4.20.23-1.el7.centos.noarch
vdsm-hook-vfio-mdev-4.20.23-1.el7.centos.noarch
vdsm-hook-ethtool-options-4.20.23-1.el7.centos.noarch
vdsm-client-4.20.23-1.el7.centos.noarch
vdsm-python-4.20.23-1.el7.centos.noarch
vdsm-hook-openstacknet-4.20.23-1.el7.centos.noarch
vdsm-network-4.20.23-1.el7.centos.x86_64
vdsm-jsonrpc-4.20.23-1.el7.centos.noarch
vdsm-api-4.20.23-1.el7.centos.noarch
vdsm-hook-fcoe-4.20.23-1.el7.centos.noarch
vdsm-yajsonrpc-4.20.23-1.el7.centos.noarch
vdsm-common-4.20.23-1.el7.centos.noarch
vdsm-hook-vhostmd-4.20.23-1.el7.centos.noarch
vdsm-http-4.20.23-1.el7.centos.noarch
vdsm-4.20.23-1.el7.centos.x86_64


How reproducible:
Try to add new host to cluster from ovirt-engine web with standart method.

Steps to Reproduce:
1. Setup CentOS7 to new host
2. add repo: http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm 
3. prepare network interfaces and bond for ovirt host setup:
eth2:
DEVICE=eth2
NAME=eth2
TYPE=Ethernet
BOOTPROTO=none
NM_CONTROLLER=no
DEFROUTE=no
PEERDNS=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
eth3:
DEVICE=eth3
NAME=eth3
TYPE=Ethernet
BOOTPROTO=none
NM_CONTROLLER=no
DEFROUTE=no
PEERDNS=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
bond0:
DEVICE=bond0
NAME=bond0
BONDING_MASTER=yes
DEFROUTE=yes
IPADDR=10.0.55.10
PREFIX=24
GATEWAY=10.0.55.1
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=4 miimon=100 lacp_rate=0 xmit_hash_policy=layer2+3"

Actual results:
After try to deploy host to cluster in log I see error:
2018-04-19 13:50:35,448+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Host 'trk-ovhv-4' is set to Non-Operational, it is missing the following networks: 'vlan10,vlan11'
2018-04-19 13:50:35,505+03 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Exception: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035492 lastValidated=1524134463813 lastCheckedOut=1524135035481 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
2018-04-19 13:50:35,536+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Failed in 'CollectVdsNetworkDataAfterInstallationVDS' method, for vds: 'trk-ovhv-4'; host: 'trk-ovhv-4.kv.in.trkua.net': Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035527 lastValidated=1524134463813 lastCheckedOut=1524135035518 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
2018-04-19 13:50:35,537+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Command 'CollectVdsNetworkDataAfterInstallationVDSCommand(HostName = trk-ovhv-4, CollectHostNetworkDataVdsCommandParameters:{hostId='ff2dbd5a-908c-4120-856d-45a85cccdaba', vds='Host[trk-ovhv-4,ff2dbd5a-908c-4120-856d-45a85cccdaba]'})' execution failed: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035527 lastValidated=1524134463813 lastCheckedOut=1524135035518 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
2018-04-19 13:50:35,537+03 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Exception: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035527 lastValidated=1524134463813 lastCheckedOut=1524135035518 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null] (Failed with error ENGINE and code 5001)
2018-04-19 13:50:35,550+03 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Host installation failed for host 'ff2dbd5a-908c-4120-856d-45a85cccdaba', 'trk-ovhv-4': Failed to configure management network on the host
2018-04-19 13:50:35,610+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] EVENT_ID: VDS_INSTALL_FAILED(505), Host trk-ovhv-4 installation failed. Failed to configure management network on the host.

and host deploy going to  status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE'

Expected results:


Additional info:
vdsm not create interfaces:
tree /var/lib/vdsm/persistence/
/var/lib/vdsm/persistence/
├── netconf -> /var/lib/vdsm/persistence/netconf.TAv2Xhos
└── netconf.TAv2Xhos
    ├── bonds
    └── nets

Comment 1 Sergii Melnyk 2018-04-19 11:43:34 UTC
Created attachment 1424027 [details]
supervsdm log from deployed host

Comment 2 Dominik Holler 2018-04-20 10:36:13 UTC
More dangerous than the problem with adding a host seems to be the JDBC errors.
When do they start?

About the problem with adding a host, I miss a line in engine.log which explains the reason. Can you please share a larger snipped of engine.log?
The vdsm.log from the host might be helpful, too. Would you share this file?
What is the switch type of the cluster you try to add the host?

Comment 3 Sergii Melnyk 2018-04-20 14:32:06 UTC
Hi,Dominik
So, I'm not shure, JDBC errors in engine log is starts after upgarade ovirt from 4.2.1 to 4.2.2 in April, because in log before upgrade I don't saw this ERROR.

vdsm.log an full engine.log in attachment.
I'm new in Ovirt an I can't clear answere aboute switch type in cluster, I'm use default network installs in web.

Comment 4 Sergii Melnyk 2018-04-20 14:32:49 UTC
Created attachment 1424525 [details]
complete engine log

Comment 5 Sergii Melnyk 2018-04-20 14:33:34 UTC
Created attachment 1424526 [details]
vdsm log

Comment 6 Sergii Melnyk 2018-04-20 14:36:11 UTC
On worked node what I was setup in Ovirt 4.2.1 I have an a network tree:
tree /var/lib/vdsm/persistence/
/var/lib/vdsm/persistence/
├── netconf -> /var/lib/vdsm/persistence/netconf.B1ilzZvK
└── netconf.B1ilzZvK
    ├── bonds
    │   ├── bond0
    │   └── bond1
    └── nets
        ├── ovirtmgmt
        ├── vlan10
        └── vlan11

Comment 7 Eli Mesika 2018-04-22 10:04:08 UTC
Please add relevant PG log as well

Comment 8 Sergii Melnyk 2018-04-23 14:40:49 UTC
Hi, I can't found postgresql logs on engine host.
How can I make start this log file?

Comment 9 Alona Kaplan 2018-04-25 07:23:23 UTC

*** This bug has been marked as a duplicate of bug 1570388 ***


Note You need to log in before you can comment on or make changes to this bug.