Description of problem: Testing vdsm3 vdsm3-4.14.5-1.el7ev.x86_64 on rhel7 with rhev 3.4 host-deploy finishes successfully. vdsm3 daemon is up and running but still the engine fails to communicate with the host and make it operational.: 2014-03-23 12:26:21,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetE xception
Are you sure the vdsmd service is up and running? vdsm.log suggests it crashes repeatedly 12:25:42,825::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) Please try out again with http://gerrit.ovirt.org/#/c/25408/ applied.
from engine log, =================== 2014-03-23 12:26:56,508 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] Host installation failed for host e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, cinteg19.ci.lab.tlv.redhat.com.: org.ovirt.engine.core.bll.InstallVdsCommand$VdsInstallException: Network error during communication with the host at org.ovirt.engine.core.bll.InstallVdsCommand.configureManagementNetwork(InstallVdsCommand.java:313) [bll.jar:] at org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:238) [bll.jar:] at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:140) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1123) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1208) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1884) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1228) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:351) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:413) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:392) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:634) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.3.2.Final-redhat-2] at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:163) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:97) [utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] 2014-03-23 12:26:56,514 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] START, SetVdsStatusVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 47cb07 2014-03-23 12:26:56,522 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] FINISH, SetVdsStatusVDSCommand, log id: 47cb07 2014-03-23 12:26:56,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-4) [58432328] Correlation ID: 58432328, Job ID: 9139ea77-34b6-4b4b-8e96-dcadaf8d71b4, Call Stack: null, Custom Event ID: -1, Message: Host cinteg19.ci.lab.tlv.redhat.com installation failed. Network error during communication with the host. 2014-03-23 12:26:56,576 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] Lock freed to object EngineLock [exclusiveLocks= key: e6ea63b6-2a90-4622-aa50-bd25f8fffd7a value: VDS , sharedLocks= ] 2014-03-23 12:26:59,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:02,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:05,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:08,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:11,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:14,509 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:17,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:20,581 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:23,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:26,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:29,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:32,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:35,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:38,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:41,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:44,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:47,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:50,799 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:53,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:56,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:27:59,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:02,895 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:05,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:08,942 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:11,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:14,982 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:17,996 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:21,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:24,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:27,048 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:30,073 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:33,091 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:36,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:39,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:42,139 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:45,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:48,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:51,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:54,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:28:57,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:00,266 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:03,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:06,301 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:09,326 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:12,342 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:15,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:18,376 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:21,391 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:24,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:27,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:30,446 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host 2014-03-23 12:29:33,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host ================================ it looks like the host deploy failed
We have an error when calling vdsm-tool when initializing storage: MainThread::DEBUG::2014-03-23 12:08:49,246::multipath::178::Storage.Misc.excCmd::(setupMultipath) FAILED: <err> = 'Traceback (most recent call last):\n File "/usr/bin/vdsm-tool", line 145, in <module>\n sys.exit(main())\n File "/usr/bin/vdsm-tool", line 142, in main\n return tool_command[cmd]["command"](*args[1:])\n File "/usr/lib64/python2.7/site-packages/vdsm/tool/service.py", line 398, in service_reload\n return _runAlts(_srvReloadAlts, srvName)\n File "/usr/lib64/python2.7/site-packages/vdsm/tool/service.py", line 350, in _runAlts\n "%s failed" % alt.func_name, out, err)\nvdsm.tool.service.ServiceOperationError: ServiceOperationError: _systemctlReload failed\nJob for multipathd.service failed. See \'systemctl status multipathd.service\' and \'journalctl -xn\' for details.\n\n'; <rc> = 1 MainThread::ERROR::2014-03-23 12:08:49,247::utils::1107::root::(panic) Panic: Error initializing IRS Traceback (most recent call last): File "/usr/share/vdsm/vdsm", line 72, in serve_clients irs = Dispatcher(HSM()) File "/usr/share/vdsm/storage/hsm.py", line 358, in __init__ multipath.setupMultipath() File "/usr/share/vdsm/storage/multipath.py", line 180, in setupMultipath raise se.MultipathReloadError() MultipathReloadError: Multipath service reload error: () We proabbly need another bug for this.
It means we cannot test RHEL 7 as hypervisor for 3.4. Putting as urgent for Compute team.
Not worked after update: [root@blue-vdsc ~]# systemctl status -l vdsmd vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled) Active: active (running) since Thu 2014-03-27 09:41:42 IST; 1min 43s ago Process: 13321 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 13356 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 13420 (vdsm) CGroup: /system.slice/vdsmd.service └─13420 /usr/bin/python /usr/share/vdsm/vdsm Mar 27 09:42:57 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:00 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:03 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:06 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:09 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:12 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:15 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:18 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:22 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 Mar 27 09:43:25 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start self.server.handle_request() File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request self._handle_request_noblock() File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock request, client_address = self.get_request() File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request return self.socket.accept() File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept raise SSL.SSLError("%s, client %s" % (e, address[0])) SSLError: sslv3 alert certificate unknown, client 10.35.163.99 [root@blue-vdsc ~]#
Nikolai, these long broken dumps of exception logs are not too readable here in bugzilla. Please try to trim them out. It seems as if you have a problem with the key/certificate deployment. What is openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -noout -text|grep Issuer
Hi Nikolai, I see you had a previous vdsm installed in the system. I recommend you to do a full re-deploy of your host via RHEV-M and not using yum install. Just for the record (from your system): # openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 1395749036 (0x533170ac) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=VDSM Certificate Authority Validity Not Before: Mar 25 12:03:56 2014 GMT Not After : Mar 25 12:03:56 2015 GMT Subject: CN=VDSM Certificate Authority <snip>
Removed all vdsm packages using yum remove and then tried to add via engine, not worked: Error received via engine's WEBUI. 2014-Mar-30, 17:51 Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Failed to execute stage 'Package installation': [u'vdsm3-4.14.6-0.el7ev.x86_64 requires sanlock >= 2.8', u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock >= 2.4']. 2014-Mar-30, 17:51 Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Yum [u'vdsm3-4.14.6-0.el7ev.x86_64 requires sanlock >= 2.8', u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock >= 2.4']. 2014-Mar-30, 17:51 Host blue-vdsc.qa.lab.tlv.redhat.com installation failed. Command returned failure code 1 during SSH session 'root.lab.tlv.redhat.com'.
(In reply to Nikolai Sednev from comment #14) > Removed all vdsm packages using yum remove and then tried to add via engine, > not worked: > Error received via engine's WEBUI. > > > 2014-Mar-30, 17:51 > > Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Failed to execute > stage 'Package installation': [u'vdsm3-4.14.6-0.el7ev.x86_64 requires > sanlock >= 2.8', u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock > >= 2.4']. > > 2014-Mar-30, 17:51 > > Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Yum > [u'vdsm3-4.14.6-0.el7ev.x86_64 requires sanlock >= 2.8', > u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock >= 2.4']. > > 2014-Mar-30, 17:51 > > Host blue-vdsc.qa.lab.tlv.redhat.com installation failed. Command returned > failure code 1 during SSH session 'root.lab.tlv.redhat.com'. Looks like RHEL7 repo is not updated with brew. In that case, you might want to update your sanlock manually.
I'm guessing that it's all about repos now, because I've just succeeded installing VDSM3 with all of it's components, while used locally placed packages and made xml for them myself, so when rhevm (engine) started the whole process of adding the host, VDSM3 was found via locally stored rpms linked from local repos: [local-vdsm3latest] Name=vdsm3 enabled=1 baseurl=file:///tmp/vdsm3latest/ gpgcheck=0 [local-libvirt] Name=libvirt enabled=1 baseurl=file:///tmp/nsednev/libvirt/ gpgcheck=0 . I was using these repos on my systems: epel.repo [epel] name=Extra Packages for Enterprise Linux 7 - $basearch #baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch failovermethod=priority enabled=1 gpgcheck=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 [epel-debuginfo] name=Extra Packages for Enterprise Linux 7 - $basearch - Debug #baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=1 [epel-source] name=Extra Packages for Enterprise Linux 7 - $basearch - Source #baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=1 epel.repo.rpmsave [epel] Name=epel baseurl=http://download.fedoraproject.org/pub/epel/beta/7/x86_64/ enabled=1 gpgcheck=0 epel-testing.repo [epel-testing] name=Extra Packages for Enterprise Linux 7 - Testing - $basearch #baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch failovermethod=priority enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 [epel-testing-debuginfo] name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Debug #baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch/debug mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-debug-epel7&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=1 [epel-testing-source] name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Source #baseurl=http://download.fedoraproject.org/pub/epel/testing/7/SRPMS mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-source-epel7&arch=$basearch failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=1 rhel-optional.repo [rhel-optional] Name=RHEL Optional enabled=1 baseurl=http://download.eng.tlv.redhat.com/pub/rhel/rel-eng/latest-RHEL-7/compose/Server-optional/x86_64/os/ gpgcheck=0 rhel-server.repo [rhel-server] Name=RHEL server enabled=1 baseurl=http://download.eng.tlv.redhat.com/pub/rhel/rel-eng/latest-RHEL-7/compose/Server/x86_64/os/ gpgcheck=0 [irhel-latest] Name=RHEL server enabled=1 baseurl=http://bob.eng.lab.tlv.redhat.com/builds/latest_av_dev/el7/ gpgcheck=0 [local-vdsm3latest] Name=vdsm3 enabled=1 baseurl=file:///tmp/vdsm3latest/ gpgcheck=0 [local-libvirt] Name=libvirt enabled=1 baseurl=file:///tmp/nsednev/libvirt/ gpgcheck=0 Errors received on engine while trying to add the "blue-vdsc.qa.lab.tlv.redhat.com", before I added local repos: 2014-Mar-31, 17:45 Host blue-vdsc.qa.lab.tlv.redhat.com installation failed. Command returned failure code 1 during SSH session 'root.lab.tlv.redhat.com'. 2014-Mar-31, 17:45 Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Failed to execute stage 'Package installation': [u'unbound-libs-1.4.21-1.el6.x86_64 requires libevent-1.4.so.2()(64bit)']. 2014-Mar-31, 17:45 Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Yum [u'unbound-libs-1.4.21-1.el6.x86_64 requires libevent-1.4.so.2()(64bit)']. Please provide me with relevant repos, so there will no need on copying files with rpms and adding local repos.
Works for me with: vdsm3-4.14.6-0.el7ev.x86_64 sanlock-3.1.0-2.el7.x86_64 libvirt-1.1.1-29.el7.x86_64 qemu-kvm-rhev-1.5.3-58.el7ev.x86_64
The problem is not that it's impossible to install vdsm3 the problem is that after host deploy/installation and running the service, the engine is failing to poll the vdsm service
Ohad, please add more information. Last problem that we have solved was a mismatch between Vdsm's certificate and Engine's. Is that the problem again? Does Vdsm run? Does it respond to vdsClient -s 0 getVdsCaps ?
yes the command responds. let me run the latest brew build on automation and I'll reply again
now supervdsm is failing for me Apr 07 15:33:19 cinteg19.ci.lab.tlv.redhat.com systemd[1]: Failed to start "Auxiliary vdsm service for running helper functions as root". -- Subject: Unit supervdsmd.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit supervdsmd.service has failed. -- -- The result is failed.
Created attachment 883614 [details] supervdsm log
Thre's a lot to be understood in supervdsm.log, but it is crashing because it is missing a requirement # rpm -V vdsm3 Unsatisfied dependencies for vdsm3-4.14.6-0.el7ev.x86_64: python-cpopen >= 1.3 is needed by (installed) vdsm3-4.14.6-0.el7ev.x86_64 S.5....T. c /etc/vdsm/vdsm.conf Why is it not installed on your host?
I don't know. I'll install it and retry. it's supposed to be a dependency. how come it let me install vdsm3 without this package?
the automation framework removes vdsm3\* and python-cpopen before each run vdsm3 and all its dependencies should be fetched with ovirt-host-deploy regular yum transaction does install python-cpopen running it with miniyum also installs it but using host deploy to install it does not do it Alon can you please throw a look? I'll attach the logs
I have removed vdsm3 and python-cpopen from the host and now host deploy successfully installs it. and supervdsm and vdsm are up but the host is still not in "up" status
2014-04-08 14:04:45,015 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = cac2907e-dac7-4b96-824a-e1a3d1b66c8d, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host Is the 54321 port open on the firewall (iptables -L)? Is `vdsClient -s 0 getVdsCaps` working locally? note that AGAIN, something evil has removed python-cpopen while keeping vdsm3 behind. Do you have a script doing `rpm -e --force` somewhere?
Thank you Dan, I got the problem (automation teardown bug) I see on iptables that the port is not open [root@cinteg19 log]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT icmp -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination there is a missing rule.
host is up thank you dan
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0504.html