Bug 1414696

Summary: The engine should wait for HostedEngineConfigFetcher to successfully complete before letting the user add an additonal hosted-engine host
Product: [oVirt] ovirt-engine Reporter: Martin Perina <mperina>
Component: BLL.HostedEngineAssignee: Doron Fediuck <dfediuck>
Status: CLOSED WORKSFORME QA Contact: meital avital <mavital>
Severity: medium Docs Contact:
Priority: unspecified    
Version: ---CC: bugs, mperina, msivak, sbonazzo, stirabos
Target Milestone: ovirt-4.2.0Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: sla
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-02 18:43:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1412547, 1414805    
Bug Blocks:    
Attachments:
Description Flags
Hosted engine VM logs
none
2nd host logs
none
1st host logs
none
2nd host etc
none
1st host etc none

Description Martin Perina 2017-01-19 09:32:32 UTC 
Created attachment 1242418 [details]
Hosted engine VM logs

Description of problem:

I've installed hosted-engine on Centos 7.3 host, installation was fine, I've added data storage domain after installation and good. But I have an issue when trying to add 2nd Centos 7.3 host using webadmin (hosted engine deployment action set to deploy), installation is successful, host become up, but "Hosted engine HA" in Host details tab reports "Not active". When checking status by "systemctl status ovirt-ha-agent -l", following error appears:

Jan 19 10:11:32 dev-21.rhev.lab.eng.brq.redhat.com ovirt-ha-agent[15294]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: ''Configuration value not found: file=/etc/ovirt-hosted-engine/hosted-engine.conf, key=gateway'' - trying to restart agent

This error is repeating every 5 seconds.

Version-Release number of selected component (if applicable):

ovirt-hosted-engine-ha-2.2.0-0.0.master.20170112134931.20170112134927.git657f109.el7.centos.noarch
ovirt-hosted-engine-setup-2.2.0-0.0.master.20170113125907.git39955c3.el7.centos.noarch

The same error I get when put host to maintenance and try to reinstall. And the same happens when completely remove the host from engine, reinstall Centos from scratch and try to install it from webadmin.

How reproducible:

100%


Steps to Reproduce:
1. Install hosted-engine, add storage domain
2. Try to add 2nd host with hosted engine deployment action set to deploy
3.

Actual results:

Host is added successfully, but not available as hosted engine host

Expected results:

Host will be available as hosted engine host after successful installation

Additional info:
Comment 1 Martin Perina 2017-01-19 09:34:34 UTC 
Created attachment 1242419 [details]
2nd host logs
Comment 2 Martin Perina 2017-01-19 09:36:22 UTC 
Created attachment 1242420 [details]
1st host logs
Comment 3 Martin Perina 2017-01-19 09:39:39 UTC 
Created attachment 1242422 [details]
2nd host etc
Comment 4 Martin Perina 2017-01-19 09:40:47 UTC 
Created attachment 1242423 [details]
1st host etc
Comment 5 Sandro Bonazzola 2017-01-19 09:41:33 UTC 
Simone can you check if 4.1 is affected too?
Comment 6 Martin Perina 2017-01-19 09:52:27 UTC 
Even adding missing configuration value didn't help, copying the whole hosted-engine.conf file from host1 didn't help and after reinstall the host from UI again the hosted-engine.conf contains only following:

ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem
host_id=2
Comment 7 Sandro Bonazzola 2017-01-19 10:00:45 UTC 
Martin, have you checked this on 4.1?
Comment 8 Martin Perina 2017-01-19 10:13:14 UTC 
(In reply to Sandro Bonazzola from comment #7)
> Martin, have you checked this on 4.1?

Not yet, I will try to install hosted engine from yesterdays RC1 build
Comment 9 Martin Perina 2017-01-19 10:14:11 UTC 
Restoring needinfo on Simone
Comment 10 Sandro Bonazzola 2017-01-19 10:42:15 UTC 
In the meanwhile moving to enging bll, since the config is not written by ovirt-host-deploy because the engine is not sending it.
Comment 11 Simone Tiraboschi 2017-01-19 10:48:13 UTC 
Host-deploy will write to /etc/ovirt-hosted-engine/hosted-engine.conf all the value received from the engine where the key prefix is 'HOSTED_ENGINE_CONFIG/'.

The issue, looking at host-deploy logs for the additional host ( ovirt-host-deploy-20170119003958-10.34.63.152-8e4f0cdd-9054-4c25-9418-c52a6dc729c8 ), is that now the engine passes only 

2017-01-19 01:38:55 DEBUG otopi.context context.dumpEnvironment:770 ENV HOSTED_ENGINE_CONFIG/host_id=str:'2'

everything else is missing there.
Comment 12 Simone Tiraboschi 2017-01-19 13:07:35 UTC 
The engine failed building the set of hosted-engine configuration parameters probably due to this: 

2017-01-19 08:57:29,203Z ERROR [org.ovirt.engine.core.bll.hostedengine.HostedEngineConfigFetcher] (default task-33) [78cf6710] Failed to fetch the hosted engine config disk due to {}: java.util.concurrent.ExecutionException: org.ovirt.engine.core.common.errors.EngineException: EngineException: Failed to download the HE configuration disk (Failed with error ENGINE and code 5001)
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_111]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) [rt.jar:1.8.0_111]
	at org.ovirt.engine.core.bll.hostedengine.HostedEngineConfigFetcher.fetch(HostedEngineConfigFetcher.java:95) [bll.jar:]
	at org.ovirt.engine.core.bll.hostedengine.HostedEngineHelper.createVdsDeployParams(HostedEngineHelper.java:88) [bll.jar:]
	at org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand.executeCommand(AddVdsCommand.java:169) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1251) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1391) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2055) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1451) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:397) [bll.jar:]
	at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) [bll.jar:]
	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) [:1.8.0_111]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111]
	at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
	at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]
	at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source) [:1.8.0_111]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111]
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
	at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final]
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
	at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view5.runAction(Unknown Source) [common.jar:]
	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runAction(GenericApiGWTServiceImpl.java:171)
	at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source) [:1.8.0_111]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111]
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:265)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:305)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]
	at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:77)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) [branding.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
	at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
	at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
	at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
Caused by: org.ovirt.engine.core.common.errors.EngineException: EngineException: Failed to download the HE configuration disk (Failed with error ENGINE and code 5001)
	at org.ovirt.engine.core.bll.hostedengine.HostedEngineConfigFetcher.downloadDisk(HostedEngineConfigFetcher.java:146) [bll.jar:]
	at org.ovirt.engine.core.bll.hostedengine.HostedEngineConfigFetcher.lambda$fetchPromise$0(HostedEngineConfigFetcher.java:77) [bll.jar:]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [rt.jar:1.8.0_111]
	at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) [rt.jar:1.8.0_111]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [rt.jar:1.8.0_111]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [rt.jar:1.8.0_111]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [rt.jar:1.8.0_111]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [rt.jar:1.8.0_111]
Comment 13 Simone Tiraboschi 2017-01-19 13:08:41 UTC 
Which is probably caused by:
2017-01-19 02:11:02,379Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler10) [7fd81570] Command 'UploadStreamVDSCommand(HostName = localhost.localdomain, UploadStreamVDSCommandParameters:{runAsync='true', hostId='c6b76520-28fa-48cc-a0f4-e207a4841437'})' execution failed: java.lang.IllegalArgumentException: TLS
Comment 14 Simone Tiraboschi 2017-01-19 13:30:03 UTC 
It could be related to https://gerrit.ovirt.org/#/c/70559/1 https://gerrit.ovirt.org/#/c/70038/6
Comment 15 Simone Tiraboschi 2017-01-19 13:44:54 UTC 
Martin, maybe it's also worth to hide the deploy hosted-engine dialog (or block it somewhere else also for the API) if, as in this case, for any reason the engine failed getting all the required information from the first host.

From the log we can see that that the engine is trying to fetch the required information (hosted-engine configuration) from the first host in a loop (but it always fails due to this java.lang.IllegalArgumentException: TLS issue) but nothing is preventing from trying to deploy the additional hosted-engine host if the import from the first didn't correctly completed. This can also hide a race conditions trying to add multiple hosts in a short time frame.

Worst than that, host-deploy and so the engine reports that the second host got correctly deployed which is not true in this case.
Comment 16 Martin Perina 2017-01-19 14:23:56 UTC 
I was not able to reproduce the issue with latest 4.1 RC build, adding 2nd host on 4.1 works correctly so removing blocks to 1413928
Comment 17 Yaniv Kaul 2017-10-02 14:21:11 UTC 
If it's not reproducible, can we close it?