Bug 1079707 - vdsm3: PollVDSCommand invocation target exception
Summary: vdsm3: PollVDSCommand invocation target exception
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 3.4.0
Assignee: Douglas Schilling Landgraf
QA Contact: Nikolai Sednev
URL:
Whiteboard: infra
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-23 11:21 UTC by Ohad Basan
Modified: 2016-02-10 19:02 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, initializing hosts that had been newly added to a Red Hat Enterprise Virtualization environment would fail under certain conditions with a PollVDSCommand exception. This was caused by the logical volume manager in Fedora 19 and Fedora 20 reading metadata stored in the lvmetad daemon instead of the storage for the host. Now, use of the lvmetad daemon has been disabled in the private logical volume manager configuration used by VDSM, forcing the logical volume manager to read metadata from the storage for a given host and allowing that host to be initialized correctly.
Clone Of:
Environment:
Last Closed: 2014-06-09 13:29:47 UTC
oVirt Team: Infra


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:0504 normal SHIPPED_LIVE vdsm 3.4.0 bug fix and enhancement update 2014-06-09 17:21:35 UTC

Description Ohad Basan 2014-03-23 11:21:54 UTC
Description of problem:
Testing vdsm3 vdsm3-4.14.5-1.el7ev.x86_64 on rhel7 with rhev 3.4
host-deploy finishes successfully. vdsm3 daemon is up and running but still the engine fails to communicate with the host and make it operational.:

2014-03-23 12:26:21,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetE
xception

Comment 4 Dan Kenigsberg 2014-03-24 13:43:10 UTC
Are you sure the vdsmd service is up and running? vdsm.log suggests it crashes repeatedly

 12:25:42,825::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)


Please try out again with http://gerrit.ovirt.org/#/c/25408/ applied.

Comment 5 Barak 2014-03-24 13:46:03 UTC
from engine log,

===================
2014-03-23 12:26:56,508 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] Host installation failed for host e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, cinteg19.ci.lab.tlv.redhat.com.: org.ovirt.engine.core.bll.InstallVdsCommand$VdsInstallException: Network error during communication with the host
	at org.ovirt.engine.core.bll.InstallVdsCommand.configureManagementNetwork(InstallVdsCommand.java:313) [bll.jar:]
	at org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:238) [bll.jar:]
	at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:140) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1123) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1208) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1884) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1228) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:351) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:413) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:392) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:634) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.3.2.Final-redhat-2]
	at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [bll.jar:]
	at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:163) [bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:97) [utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

2014-03-23 12:26:56,514 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] START, SetVdsStatusVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 47cb07
2014-03-23 12:26:56,522 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] FINISH, SetVdsStatusVDSCommand, log id: 47cb07
2014-03-23 12:26:56,558 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-4) [58432328] Correlation ID: 58432328, Job ID: 9139ea77-34b6-4b4b-8e96-dcadaf8d71b4, Call Stack: null, Custom Event ID: -1, Message: Host cinteg19.ci.lab.tlv.redhat.com installation failed. Network error during communication with the host.
2014-03-23 12:26:56,576 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (org.ovirt.thread.pool-4-thread-4) [58432328] Lock freed to object EngineLock [exclusiveLocks= key: e6ea63b6-2a90-4622-aa50-bd25f8fffd7a value: VDS
, sharedLocks= ]
2014-03-23 12:26:59,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:02,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:05,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:08,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:11,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:14,509 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:17,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:20,581 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:23,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:26,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:29,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:32,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:35,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:38,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:41,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:44,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:47,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:50,799 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:53,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:56,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:27:59,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:02,895 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:05,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:08,942 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:11,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:14,982 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:17,996 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:21,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:24,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:27,048 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:30,073 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:33,091 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:36,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:39,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:42,139 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:45,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:48,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:51,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:54,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:28:57,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:00,266 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:03,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:06,301 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:09,326 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:12,342 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:15,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:18,376 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:21,391 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:24,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:27,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:30,446 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2014-03-23 12:29:33,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = e6ea63b6-2a90-4622-aa50-bd25f8fffd7a, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host


================================

it looks like the host deploy failed

Comment 7 Nir Soffer 2014-03-25 07:50:51 UTC
We have an error when calling vdsm-tool when initializing storage:

MainThread::DEBUG::2014-03-23 12:08:49,246::multipath::178::Storage.Misc.excCmd::(setupMultipath) FAILED: <err> = 'Traceback (most recent call last):\n  File "/usr/bin/vdsm-tool", line 145, in <module>\n    sys.exit(main())\n  File "/usr/bin/vdsm-tool", line 142, in main\n    return tool_command[cmd]["command"](*args[1:])\n  File "/usr/lib64/python2.7/site-packages/vdsm/tool/service.py", line 398, in service_reload\n    return _runAlts(_srvReloadAlts, srvName)\n  File "/usr/lib64/python2.7/site-packages/vdsm/tool/service.py", line 350, in _runAlts\n    "%s failed" % alt.func_name, out, err)\nvdsm.tool.service.ServiceOperationError: ServiceOperationError: _systemctlReload failed\nJob for multipathd.service failed. See \'systemctl status multipathd.service\' and \'journalctl -xn\' for details.\n\n'; <rc> = 1
MainThread::ERROR::2014-03-23 12:08:49,247::utils::1107::root::(panic) Panic: Error initializing IRS
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsm", line 72, in serve_clients
    irs = Dispatcher(HSM())
  File "/usr/share/vdsm/storage/hsm.py", line 358, in __init__
    multipath.setupMultipath()
  File "/usr/share/vdsm/storage/multipath.py", line 180, in setupMultipath
    raise se.MultipathReloadError()
MultipathReloadError: Multipath service reload error: ()

We proabbly need another bug for this.

Comment 9 Nikolai Sednev 2014-03-26 13:01:02 UTC
It means we cannot test RHEL 7 as hypervisor for 3.4.
Putting as urgent for Compute team.

Comment 11 Nikolai Sednev 2014-03-27 07:44:26 UTC
Not worked after update:
[root@blue-vdsc ~]# systemctl status -l  vdsmd                                                                                       
vdsmd.service - Virtual Desktop Server Manager                                                                                       
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)                                                                   
   Active: active (running) since Thu 2014-03-27 09:41:42 IST; 1min 43s ago                                                          
  Process: 13321 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)                     
  Process: 13356 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)                     
 Main PID: 13420 (vdsm)                                                                                                              
   CGroup: /system.slice/vdsmd.service                                                                                               
           └─13420 /usr/bin/python /usr/share/vdsm/vdsm                                                                              

Mar 27 09:42:57 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception
                                                             Traceback (most recent call last):      
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start
                                                                 self.server.handle_request()                                     
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request
                                                                 self._handle_request_noblock()                                        
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock
                                                                 request, client_address = self.get_request()                                   
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request            
                                                                 return self.socket.accept()                                                    
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))                                  
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99                             
Mar 27 09:43:00 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception                                                   
                                                             Traceback (most recent call last):                                                         
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start                      
                                                                 self.server.handle_request()                                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request                 
                                                                 self._handle_request_noblock()                                                         
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock        
                                                                 request, client_address = self.get_request()                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request                    
                                                                 return self.socket.accept()                                                            
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))                                  
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99                             
Mar 27 09:43:03 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception                                                   
                                                             Traceback (most recent call last):                                                         
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start                      
                                                                 self.server.handle_request()                                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request                 
                                                                 self._handle_request_noblock()                                                         
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock        
                                                                 request, client_address = self.get_request()                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request                    
                                                                 return self.socket.accept()                                                            
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))                                  
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99                             
Mar 27 09:43:06 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception                                                   
                                                             Traceback (most recent call last):                                                         
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start                      
                                                                 self.server.handle_request()                                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request                 
                                                                 self._handle_request_noblock()                                                         
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock        
                                                                 request, client_address = self.get_request()                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request                    
                                                                 return self.socket.accept()                                                            
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))                                  
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99                             
Mar 27 09:43:09 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception                                                   
                                                             Traceback (most recent call last):                                                         
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start                      
                                                                 self.server.handle_request()                                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request                 
                                                                 self._handle_request_noblock()                                                         
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock        
                                                                 request, client_address = self.get_request()                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request                    
                                                                 return self.socket.accept()                                                            
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))                                  
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99                             
Mar 27 09:43:12 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception                                                   
                                                             Traceback (most recent call last):                                                         
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start                      
                                                                 self.server.handle_request()                                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request                 
                                                                 self._handle_request_noblock()                                                         
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock        
                                                                 request, client_address = self.get_request()                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request                    
                                                                 return self.socket.accept()                                                            
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))                                  
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99                             
Mar 27 09:43:15 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception                                                   
                                                             Traceback (most recent call last):                                                         
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start                      
                                                                 self.server.handle_request()                                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request                 
                                                                 self._handle_request_noblock()                                                         
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock        
                                                                 request, client_address = self.get_request()                                           
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request
                                                                 return self.socket.accept()
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99
Mar 27 09:43:18 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception
                                                             Traceback (most recent call last):
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start
                                                                 self.server.handle_request()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request
                                                                 self._handle_request_noblock()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock
                                                                 request, client_address = self.get_request()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request
                                                                 return self.socket.accept()
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99
Mar 27 09:43:22 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception
                                                             Traceback (most recent call last):
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start
                                                                 self.server.handle_request()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request
                                                                 self._handle_request_noblock()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock
                                                                 request, client_address = self.get_request()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request
                                                                 return self.socket.accept()
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99
Mar 27 09:43:25 blue-vdsc.qa.lab.tlv.redhat.com vdsm[13420]: vdsm vds ERROR xml-rpc handler exception
                                                             Traceback (most recent call last):
                                                               File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start
                                                                 self.server.handle_request()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 280, in handle_request
                                                                 self._handle_request_noblock()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 290, in _handle_request_noblock
                                                                 request, client_address = self.get_request()
                                                               File "/usr/lib64/python2.7/SocketServer.py", line 463, in get_request
                                                                 return self.socket.accept()
                                                               File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept
                                                                 raise SSL.SSLError("%s, client %s" % (e, address[0]))
                                                             SSLError: sslv3 alert certificate unknown, client 10.35.163.99
[root@blue-vdsc ~]#

Comment 12 Dan Kenigsberg 2014-03-27 08:51:29 UTC
Nikolai, these long broken dumps of exception logs are not too readable here in bugzilla. Please try to trim them out.

It seems as if you have a problem with the key/certificate deployment.

What is

  openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -noout -text|grep Issuer

Comment 13 Douglas Schilling Landgraf 2014-03-28 14:30:46 UTC
Hi Nikolai,

I see you had a previous vdsm installed in the system. I recommend you to do a full re-deploy of your host via RHEV-M and not using yum install.

Just for the record (from your system):
# openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1395749036 (0x533170ac)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=VDSM Certificate Authority
        Validity
            Not Before: Mar 25 12:03:56 2014 GMT
            Not After : Mar 25 12:03:56 2015 GMT
        Subject: CN=VDSM Certificate Authority
<snip>

Comment 14 Nikolai Sednev 2014-03-30 14:55:04 UTC
Removed all vdsm packages using yum remove and then tried to add via engine, not worked:
Error received via engine's WEBUI.	

	
2014-Mar-30, 17:51
	
Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Failed to execute stage 'Package installation': [u'vdsm3-4.14.6-0.el7ev.x86_64 requires sanlock >= 2.8', u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock >= 2.4'].
	
2014-Mar-30, 17:51
	
Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Yum [u'vdsm3-4.14.6-0.el7ev.x86_64 requires sanlock >= 2.8', u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock >= 2.4'].

2014-Mar-30, 17:51
	
Host blue-vdsc.qa.lab.tlv.redhat.com installation failed. Command returned failure code 1 during SSH session 'root@blue-vdsc.qa.lab.tlv.redhat.com'.

Comment 15 Douglas Schilling Landgraf 2014-03-30 19:12:39 UTC
(In reply to Nikolai Sednev from comment #14)
> Removed all vdsm packages using yum remove and then tried to add via engine,
> not worked:
> Error received via engine's WEBUI.	
> 
> 	
> 2014-Mar-30, 17:51
> 	
> Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Failed to execute
> stage 'Package installation': [u'vdsm3-4.14.6-0.el7ev.x86_64 requires
> sanlock >= 2.8', u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock
> >= 2.4'].
> 	
> 2014-Mar-30, 17:51
> 	
> Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Yum
> [u'vdsm3-4.14.6-0.el7ev.x86_64 requires sanlock >= 2.8',
> u'libvirt-lock-sanlock-1.1.1-29.el7.x86_64 requires sanlock >= 2.4'].
> 
> 2014-Mar-30, 17:51
> 	
> Host blue-vdsc.qa.lab.tlv.redhat.com installation failed. Command returned
> failure code 1 during SSH session 'root@blue-vdsc.qa.lab.tlv.redhat.com'.

Looks like RHEL7 repo is not updated with brew. In that case, you might want to update your sanlock manually.

Comment 17 Nikolai Sednev 2014-04-01 14:03:04 UTC
I'm guessing that it's all about repos now, because I've just succeeded installing VDSM3 with all of it's components, while used locally placed packages and made xml for them myself, so when rhevm (engine) started the whole process of adding the host, VDSM3 was found via locally stored rpms linked from local repos:
[local-vdsm3latest]
Name=vdsm3
enabled=1
baseurl=file:///tmp/vdsm3latest/
gpgcheck=0

[local-libvirt]
Name=libvirt
enabled=1
baseurl=file:///tmp/nsednev/libvirt/
gpgcheck=0
.

I was using these repos on my systems:

epel.repo   
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1



epel.repo.rpmsave
[epel]
Name=epel
baseurl=http://download.fedoraproject.org/pub/epel/beta/7/x86_64/
enabled=1
gpgcheck=0




epel-testing.repo  
[epel-testing]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-testing-debuginfo]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch/debug
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-debug-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-testing-source]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-source-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1



rhel-optional.repo
[rhel-optional]
Name=RHEL Optional
enabled=1
baseurl=http://download.eng.tlv.redhat.com/pub/rhel/rel-eng/latest-RHEL-7/compose/Server-optional/x86_64/os/
gpgcheck=0



rhel-server.repo


[rhel-server]
Name=RHEL server
enabled=1
baseurl=http://download.eng.tlv.redhat.com/pub/rhel/rel-eng/latest-RHEL-7/compose/Server/x86_64/os/
gpgcheck=0

[irhel-latest]
Name=RHEL server
enabled=1
baseurl=http://bob.eng.lab.tlv.redhat.com/builds/latest_av_dev/el7/
gpgcheck=0

[local-vdsm3latest]
Name=vdsm3
enabled=1
baseurl=file:///tmp/vdsm3latest/
gpgcheck=0

[local-libvirt]
Name=libvirt
enabled=1
baseurl=file:///tmp/nsednev/libvirt/
gpgcheck=0




Errors received on engine while trying to add the "blue-vdsc.qa.lab.tlv.redhat.com", before I added local repos:
	
2014-Mar-31, 17:45
	
Host blue-vdsc.qa.lab.tlv.redhat.com installation failed. Command returned failure code 1 during SSH session 'root@blue-vdsc.qa.lab.tlv.redhat.com'.

	
2014-Mar-31, 17:45
	
Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Failed to execute stage 'Package installation': [u'unbound-libs-1.4.21-1.el6.x86_64 requires libevent-1.4.so.2()(64bit)'].
	
2014-Mar-31, 17:45
	
Failed to install Host blue-vdsc.qa.lab.tlv.redhat.com. Yum [u'unbound-libs-1.4.21-1.el6.x86_64 requires libevent-1.4.so.2()(64bit)'].



Please provide me with relevant repos, so there will no need on copying files with rpms and adding local repos.

Comment 19 Nikolai Sednev 2014-04-01 14:15:05 UTC
Works for me with:
vdsm3-4.14.6-0.el7ev.x86_64
sanlock-3.1.0-2.el7.x86_64
libvirt-1.1.1-29.el7.x86_64
qemu-kvm-rhev-1.5.3-58.el7ev.x86_64

Comment 22 Ohad Basan 2014-04-07 11:09:09 UTC
The problem is not that it's impossible to install vdsm3
the problem is that after host deploy/installation and running the service, the engine is failing to poll the vdsm service

Comment 23 Dan Kenigsberg 2014-04-07 11:27:26 UTC
Ohad, please add more information. Last problem that we have solved was a mismatch between Vdsm's certificate and Engine's.

Is that the problem again? Does Vdsm run? Does it respond to
  vdsClient -s 0 getVdsCaps
?

Comment 24 Ohad Basan 2014-04-07 11:57:43 UTC
yes the command responds.
let me run the latest brew build on automation and I'll reply again

Comment 25 Ohad Basan 2014-04-07 12:34:53 UTC
now supervdsm is failing for me
Apr 07 15:33:19 cinteg19.ci.lab.tlv.redhat.com systemd[1]: Failed to start "Auxiliary vdsm service for running helper functions as root".
-- Subject: Unit supervdsmd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit supervdsmd.service has failed.
-- 
-- The result is failed.

Comment 26 Ohad Basan 2014-04-07 12:43:39 UTC
Created attachment 883614 [details]
supervdsm log

Comment 27 Dan Kenigsberg 2014-04-07 15:26:44 UTC
Thre's a lot to be understood in supervdsm.log, but it is crashing because it is missing a requirement

# rpm -V vdsm3
Unsatisfied dependencies for vdsm3-4.14.6-0.el7ev.x86_64:
	python-cpopen >= 1.3 is needed by (installed) vdsm3-4.14.6-0.el7ev.x86_64
S.5....T.  c /etc/vdsm/vdsm.conf


Why is it not installed on your host?

Comment 28 Ohad Basan 2014-04-08 07:28:12 UTC
I don't know. 
I'll install it and retry.
it's supposed to be a dependency.
how come it let me install vdsm3 without this package?

Comment 29 Ohad Basan 2014-04-08 10:09:37 UTC
the automation framework removes vdsm3\* and python-cpopen before each run
vdsm3 and all its dependencies should be fetched with ovirt-host-deploy
regular yum transaction does install python-cpopen
running it with miniyum also installs it
but using host deploy to install it does not do it
Alon can you please throw a look?
I'll attach the logs

Comment 31 Ohad Basan 2014-04-08 12:14:30 UTC
I have removed vdsm3 and python-cpopen from the host and now host deploy successfully installs it.
and supervdsm and vdsm are up
but the host is still not in "up" status

Comment 33 Dan Kenigsberg 2014-04-08 12:52:11 UTC
2014-04-08 14:04:45,015 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = cinteg19.ci.lab.tlv.redhat.com, HostId = cac2907e-dac7-4b96-824a-e1a3d1b66c8d, vds=Host[cinteg19.ci.lab.tlv.redhat.com]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host

Is the 54321 port open on the firewall (iptables -L)?
Is `vdsClient -s 0 getVdsCaps` working locally?

note that AGAIN, something evil has removed python-cpopen while keeping vdsm3 behind. Do you have a script doing `rpm -e --force` somewhere?

Comment 34 Ohad Basan 2014-04-08 13:15:54 UTC
Thank you Dan,
I got the problem (automation teardown bug)
I see on iptables that the port is not open

[root@cinteg19 log]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

there is a missing rule.

Comment 35 Ohad Basan 2014-04-08 13:33:24 UTC
host is up
thank you dan

Comment 36 errata-xmlrpc 2014-06-09 13:29:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0504.html


Note You need to log in before you can comment on or make changes to this bug.