Bug 979766 - Failed add second and and subsequent host with identical UUID - Failed to initiate vdsm-id request on host
Failed add second and and subsequent host with identical UUID - Failed to i...
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.3.0
x86_64 Linux
unspecified Severity medium
: ---
: 3.3.0
Assigned To: nobody nobody
vvyazmin@redhat.com
infra
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-30 07:39 EDT by vvyazmin@redhat.com
Modified: 2016-02-10 14:17 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-15 03:26:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
## Logs engine, server (17.64 KB, application/x-gzip)
2013-06-30 07:39 EDT, vvyazmin@redhat.com
no flags Details

  None (edit)
Description vvyazmin@redhat.com 2013-06-30 07:39:26 EDT
Created attachment 767039 [details]
## Logs engine, server

Description of problem: 
Failed add second and and subsequent host with  identical UUID -  Failed to initiate vdsm-id request on host


Version-Release number of selected component (if applicable):
RHEVM 3.3 - IS3 environment:

RHEVM: rhevm-3.3.0-0.5.master.el6ev.noarch
VDSM: vdsm-4.10.2-23.0.el6ev.x86_64
LIBVIRT: libvirt-0.10.2-18.el6_4.8.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Add 2 or more host (with identical UUID) via UI 
  
Actual results:
Failed add second and and subsequent host 

Expected results:
Get pop-up with a explanation and solution to this issue.

Impact on user:

Workaround:
On all hosts run following command:
uuidgen > /etc/vdsm/vdsm.id

Additional info:

/var/log/ovirt-engine/engine.log

2013-06-30 11:54:47,426 WARN  [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] Failed to initiate vdsm-id request on host: java.io.IOException: Command returned failure code 1 during SSH session 'root@tigris02.scl.lab.tlv.redhat.com'
	at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:507) [utils.jar:]
	at org.ovirt.engine.core.bll.AddVdsCommand.getInstalledVdsIdIfExists(AddVdsCommand.java:386) [bll.jar:]
	at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:408) [bll.jar:]
	at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:334) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:681) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:312) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:348) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:301) [bll.jar:]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) [:1.7.0_25]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view8.RunAction(Unknown Source) [common.jar:]
	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.RunAction(GenericApiGWTServiceImpl.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
	at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196)
	at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161)
	at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:59) [utils.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:465)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488)
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

2013-06-30 11:54:47,449 INFO  [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] Running command: AddVdsCommand internal: false. Entities affected :  ID: 6c18c719-138a-48ad-9f23-fabcb8ce2e1d Type: VdsGroups
2013-06-30 11:54:47,462 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] Running command: AddVdsSpmIdCommand internal: true. Entities affected :  ID: ff64aa3d-4151-4e2b-96bb-ac31d009f363 Type: VDS
2013-06-30 11:54:47,473 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] Lock freed to object EngineLock [exclusiveLocks= key: e79e4a05-2e22-4ecd-903f-caf7c55fa15b value: REGISTER_VDS
, sharedLocks= ]
2013-06-30 11:54:47,473 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [29a0225e] Cannot get vdsManager for vdsid=ff64aa3d-4151-4e2b-96bb-ac31d009f363
2013-06-30 11:54:47,474 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] START, RemoveVdsVDSCommand(HostName = tigris02.scl.lab.tlv.redhat.com, HostId = ff64aa3d-4151-4e2b-96bb-ac31d009f363), log id: 1598f558
2013-06-30 11:54:47,475 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [29a0225e] Cannot get vdsManager for vdsid=ff64aa3d-4151-4e2b-96bb-ac31d009f363
2013-06-30 11:54:47,475 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] FINISH, RemoveVdsVDSCommand, log id: 1598f558
2013-06-30 11:54:47,475 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [29a0225e] Cannot get vdsManager for vdsid=ff64aa3d-4151-4e2b-96bb-ac31d009f363
2013-06-30 11:54:47,477 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] START, AddVdsVDSCommand(HostName = tigris02.scl.lab.tlv.redhat.com, HostId = ff64aa3d-4151-4e2b-96bb-ac31d009f363), log id: 3068248b
2013-06-30 11:54:47,477 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] AddVds - entered , starting logic to add VDS ff64aa3d-4151-4e2b-96bb-ac31d009f363
2013-06-30 11:54:47,478 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] AddVds - VDS ff64aa3d-4151-4e2b-96bb-ac31d009f363 was added, will try to add it to the resource manager
2013-06-30 11:54:47,478 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-1) [29a0225e] Entered VdsManager constructor
2013-06-30 11:54:47,479 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-1) [29a0225e] Initialize vdsBroker (tigris02.scl.lab.tlv.redhat.com,54,321)
2013-06-30 11:54:47,480 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [29a0225e] VDS ff64aa3d-4151-4e2b-96bb-ac31d009f363 was added to the Resource Manager
2013-06-30 11:54:47,480 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [29a0225e] FINISH, AddVdsVDSCommand, log id: 3068248b
2013-06-30 11:54:47,520 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-5-thread-7) [29a0225e] Running command: InstallVdsCommand internal: true. Entities affected :  ID: ff64aa3d-4151-4e2b-96bb-ac31d009f363 Type: VDS
2013-06-30 11:54:47,521 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-5-thread-7) [29a0225e] Before Installation host ff64aa3d-4151-4e2b-96bb-ac31d009f363, tigris02.scl.lab.tlv.redhat.com
2013-06-30 11:54:47,522 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-5-thread-7) [29a0225e] START, SetVdsStatusVDSCommand(HostName = tigris02.scl.lab.tlv.redhat.com, HostId = ff64aa3d-4151-4e2b-96bb-ac31d009f363, status=Installing, nonOperationalReason=NONE), log id: 3f4beee9
2013-06-30 11:54:47,526 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-5-thread-7) [29a0225e] FINISH, SetVdsStatusVDSCommand, log id: 3f4beee9
2013-06-30 11:54:47,596 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (pool-5-thread-7) [29a0225e] Installation tigris02.scl.lab.tlv.redhat.com: Connected to host tigris02.scl.lab.tlv.redhat.com with SSH key fingerprint: e9:3d:6f:b1:43:38:4f:61:e7:d4:43:02:57:f4:b9:04
2013-06-30 11:54:47,622 INFO  [org.ovirt.engine.core.bll.VdsDeploy] (pool-5-thread-7) [29a0225e] Installation of tigris02.scl.lab.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar
2013-06-30 11:54:47,626 INFO  [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-5-thread-7) SSH execute root@tigris02.scl.lab.tlv.redhat.com 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x &&  "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True'

/var/log/vdsm/vdsm.log

Note You need to log in before you can comment on or make changes to this bug.