Created attachment 1424019 [details] engine deploy log Description of problem: Dears, After update ovirt from 4.2.1 to 4.2.2 I was try to add new host to cluster, but can't complete deploy installs to host with ERROR: Host trk-ovhv-4 installation failed. Failed to configure management network on the host. Version-Release number of selected component (if applicable): On ovirt-engine node: ovirt-engine-tools-backup-4.2.2.5-1.el7.centos.noarch ovirt-engine-tools-4.2.2.5-1.el7.centos.noarch ovirt-engine-dwh-4.2.2.2-1.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-common-4.2.2.6-1.el7.centos.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.2.2.6-1.el7.centos.noarch ovirt-ansible-cluster-upgrade-1.1.6-1.el7.centos.noarch ovirt-ansible-manageiq-1.1.6-1.el7.centos.noarch ovirt-engine-wildfly-11.0.0-1.el7.centos.x86_64 ovirt-engine-websocket-proxy-4.2.2.6-1.el7.centos.noarch ovirt-engine-cli-3.6.9.2-1.el7.centos.noarch ovirt-engine-extensions-api-impl-4.2.2.6-1.el7.centos.noarch ovirt-ansible-repositories-1.1.0-1.el7.centos.noarch ovirt-vmconsole-1.0.4-1.el7.noarch ovirt-setup-lib-1.1.4-1.el7.centos.noarch ovirt-engine-dwh-setup-4.2.2.2-1.el7.centos.noarch ovirt-engine-extension-aaa-jdbc-1.1.7-1.el7.centos.noarch ovirt-js-dependencies-1.2.0-3.1.el7.centos.noarch python-ovirt-engine-sdk4-4.2.4-2.el7.centos.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch ovirt-host-deploy-java-1.7.3-1.el7.centos.noarch ovirt-iso-uploader-4.2.0-1.el7.centos.noarch ovirt-engine-dbscripts-4.2.2.5-1.el7.centos.noarch ovirt-engine-webadmin-portal-4.2.2.5-1.el7.centos.noarch ovirt-engine-restapi-4.2.2.5-1.el7.centos.noarch ovirt-engine-backend-4.2.2.5-1.el7.centos.noarch ovirt-engine-lib-4.2.2.6-1.el7.centos.noarch ovirt-engine-setup-base-4.2.2.6-1.el7.centos.noarch ovirt-imageio-proxy-setup-1.2.2-0.el7.centos.noarch ovirt-provider-ovn-1.2.9-1.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-4.2.2.6-1.el7.centos.noarch ovirt-engine-vmconsole-proxy-helper-4.2.2.6-1.el7.centos.noarch ovirt-ansible-vm-infra-1.1.5-1.el7.centos.noarch ovirt-ansible-infra-1.1.4-1.el7.centos.noarch ovirt-engine-metrics-1.1.3.4-1.el7.centos.noarch ovirt-engine-wildfly-overlay-11.0.1-1.el7.centos.noarch ovirt-imageio-proxy-1.2.2-0.el7.centos.noarch ovirt-release42-4.2.2-3.el7.centos.noarch ovirt-web-ui-1.3.7-2.el7.centos.noarch ovirt-ansible-engine-setup-1.1.0-1.el7.centos.noarch ovirt-ansible-image-template-1.1.5-1.el7.centos.noarch ovirt-vmconsole-proxy-1.0.4-1.el7.noarch ovirt-cockpit-sso-0.0.4-1.el7.noarch ovirt-engine-api-explorer-0.0.2-1.el7.centos.noarch ovirt-host-deploy-1.7.3-1.el7.centos.noarch ovirt-engine-dashboard-1.2.2-3.el7.centos.noarch ovirt-engine-4.2.2.5-1.el7.centos.noarch ovirt-engine-setup-plugin-websocket-proxy-4.2.2.6-1.el7.centos.noarch ovirt-imageio-common-1.2.2-0.el7.centos.noarch ovirt-engine-setup-4.2.2.6-1.el7.centos.noarch ovirt-ansible-disaster-recovery-0.3-1.el7.centos.noarch ovirt-ansible-roles-1.1.3-1.el7.centos.noarch On new add host to cluster: ovirt-vmconsole-1.0.4-1.el7.noarch ovirt-imageio-common-1.2.2-0.el7.centos.noarch ovirt-setup-lib-1.1.4-1.el7.centos.noarch cockpit-ovirt-dashboard-0.11.20-1.el7.centos.noarch ovirt-release42-4.2.2-3.el7.centos.noarch ovirt-imageio-daemon-1.2.2-0.el7.centos.noarch ovirt-host-4.2.2-2.el7.centos.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch python-ovirt-engine-sdk4-4.2.4-2.el7.centos.x86_64 ovirt-host-deploy-1.7.3-1.el7.centos.noarch ovirt-host-dependencies-4.2.2-2.el7.centos.x86_64 ovirt-hosted-engine-ha-2.2.10-1.el7.centos.noarch ovirt-vmconsole-host-1.0.4-1.el7.noarch ovirt-hosted-engine-setup-2.2.16-1.el7.centos.noarch ovirt-provider-ovn-driver-1.2.9-1.el7.centos.noarch vdsm-hook-vmfex-dev-4.20.23-1.el7.centos.noarch vdsm-hook-vfio-mdev-4.20.23-1.el7.centos.noarch vdsm-hook-ethtool-options-4.20.23-1.el7.centos.noarch vdsm-client-4.20.23-1.el7.centos.noarch vdsm-python-4.20.23-1.el7.centos.noarch vdsm-hook-openstacknet-4.20.23-1.el7.centos.noarch vdsm-network-4.20.23-1.el7.centos.x86_64 vdsm-jsonrpc-4.20.23-1.el7.centos.noarch vdsm-api-4.20.23-1.el7.centos.noarch vdsm-hook-fcoe-4.20.23-1.el7.centos.noarch vdsm-yajsonrpc-4.20.23-1.el7.centos.noarch vdsm-common-4.20.23-1.el7.centos.noarch vdsm-hook-vhostmd-4.20.23-1.el7.centos.noarch vdsm-http-4.20.23-1.el7.centos.noarch vdsm-4.20.23-1.el7.centos.x86_64 How reproducible: Try to add new host to cluster from ovirt-engine web with standart method. Steps to Reproduce: 1. Setup CentOS7 to new host 2. add repo: http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm 3. prepare network interfaces and bond for ovirt host setup: eth2: DEVICE=eth2 NAME=eth2 TYPE=Ethernet BOOTPROTO=none NM_CONTROLLER=no DEFROUTE=no PEERDNS=no ONBOOT=yes MASTER=bond0 SLAVE=yes eth3: DEVICE=eth3 NAME=eth3 TYPE=Ethernet BOOTPROTO=none NM_CONTROLLER=no DEFROUTE=no PEERDNS=no ONBOOT=yes MASTER=bond0 SLAVE=yes bond0: DEVICE=bond0 NAME=bond0 BONDING_MASTER=yes DEFROUTE=yes IPADDR=10.0.55.10 PREFIX=24 GATEWAY=10.0.55.1 ONBOOT=yes BOOTPROTO=none BONDING_OPTS="mode=4 miimon=100 lacp_rate=0 xmit_hash_policy=layer2+3" Actual results: After try to deploy host to cluster in log I see error: 2018-04-19 13:50:35,448+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Host 'trk-ovhv-4' is set to Non-Operational, it is missing the following networks: 'vlan10,vlan11' 2018-04-19 13:50:35,505+03 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Exception: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035492 lastValidated=1524134463813 lastCheckedOut=1524135035481 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null] 2018-04-19 13:50:35,536+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Failed in 'CollectVdsNetworkDataAfterInstallationVDS' method, for vds: 'trk-ovhv-4'; host: 'trk-ovhv-4.kv.in.trkua.net': Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035527 lastValidated=1524134463813 lastCheckedOut=1524135035518 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null] 2018-04-19 13:50:35,537+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Command 'CollectVdsNetworkDataAfterInstallationVDSCommand(HostName = trk-ovhv-4, CollectHostNetworkDataVdsCommandParameters:{hostId='ff2dbd5a-908c-4120-856d-45a85cccdaba', vds='Host[trk-ovhv-4,ff2dbd5a-908c-4120-856d-45a85cccdaba]'})' execution failed: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035527 lastValidated=1524134463813 lastCheckedOut=1524135035518 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null] 2018-04-19 13:50:35,537+03 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Exception: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@fa6e6e5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@5b050fc3 connection handles=0 lastReturned=1524135035527 lastValidated=1524134463813 lastCheckedOut=1524135035518 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@123909eb mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@62d662ff[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl@1a6a5b19[connectionListener=fa6e6e5 connectionManager=653865e8 warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null] (Failed with error ENGINE and code 5001) 2018-04-19 13:50:35,550+03 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] Host installation failed for host 'ff2dbd5a-908c-4120-856d-45a85cccdaba', 'trk-ovhv-4': Failed to configure management network on the host 2018-04-19 13:50:35,610+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2068) [3b423c4c] EVENT_ID: VDS_INSTALL_FAILED(505), Host trk-ovhv-4 installation failed. Failed to configure management network on the host. and host deploy going to status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE' Expected results: Additional info: vdsm not create interfaces: tree /var/lib/vdsm/persistence/ /var/lib/vdsm/persistence/ ├── netconf -> /var/lib/vdsm/persistence/netconf.TAv2Xhos └── netconf.TAv2Xhos ├── bonds └── nets
Created attachment 1424027 [details] supervsdm log from deployed host
More dangerous than the problem with adding a host seems to be the JDBC errors. When do they start? About the problem with adding a host, I miss a line in engine.log which explains the reason. Can you please share a larger snipped of engine.log? The vdsm.log from the host might be helpful, too. Would you share this file? What is the switch type of the cluster you try to add the host?
Hi,Dominik So, I'm not shure, JDBC errors in engine log is starts after upgarade ovirt from 4.2.1 to 4.2.2 in April, because in log before upgrade I don't saw this ERROR. vdsm.log an full engine.log in attachment. I'm new in Ovirt an I can't clear answere aboute switch type in cluster, I'm use default network installs in web.
Created attachment 1424525 [details] complete engine log
Created attachment 1424526 [details] vdsm log
On worked node what I was setup in Ovirt 4.2.1 I have an a network tree: tree /var/lib/vdsm/persistence/ /var/lib/vdsm/persistence/ ├── netconf -> /var/lib/vdsm/persistence/netconf.B1ilzZvK └── netconf.B1ilzZvK ├── bonds │ ├── bond0 │ └── bond1 └── nets ├── ovirtmgmt ├── vlan10 └── vlan11
Please add relevant PG log as well
Hi, I can't found postgresql logs on engine host. How can I make start this log file?
*** This bug has been marked as a duplicate of bug 1570388 ***