Bug 997180 - Failed open RHEVM URL after upgrade - HTTP Status 500 - java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
Failed open RHEVM URL after upgrade - HTTP Status 500 - java.lang.IllegalStat...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-webadmin-portal (Show other bugs)
3.3.0
x86_64 Linux
unspecified Severity medium
: ---
: 3.3.0
Assigned To: Mooli Tayer
vvyazmin@redhat.com
infra
: Regression, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-14 16:52 EDT by vvyazmin@redhat.com
Modified: 2016-02-10 14:36 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-22 11:16:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
## Logs rhevm, vdsm, libvirt, thread dump, superVdsm (3.14 MB, application/x-gzip)
2013-08-14 16:52 EDT, vvyazmin@redhat.com
no flags Details

  None (edit)
Description vvyazmin@redhat.com 2013-08-14 16:52:35 EDT
Created attachment 786700 [details]
## Logs rhevm, vdsm, libvirt, thread dump, superVdsm

Description of problem:
Failed open RHEVM URL after upgrade - HTTP Status 500 - java.lang.IllegalStateException: JBAS011048: Failed to construct component instance

Version-Release number of selected component (if applicable):
RHEVM 3.3 - IS10 environment:

RHEVM:  rhevm-3.3.0-0.15.master.el6ev.noarch
PythonSDK:  rhevm-sdk-python-3.3.0.10-1.el6ev.noarch
VDSM:  vdsm-4.12.0-61.git8178ec2.el6ev.x86_64
LIBVIRT:  libvirt-0.10.2-18.el6_4.9.x86_64
QEMU & KVM:  qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64
SANLOCK:  sanlock-2.8-1.el6.x86_64

How reproducible:
unknown

Steps to Reproduce:
Upgrade RHEVM 3.3 from IS9.1 to IS10 

Actual results:
Upgrade process worked OK, but failed open RHEVM URL - HTTP Status 500 - java.lang.IllegalStateException: JBAS011048: Failed to construct component instance

Expected results:
Succeed open RHEVM

Impact on user:
Failed RHEVM URL

Workaround:
Reboot RHEVM machine

Additional info:

/var/log/ovirt-engine/engine.log
2013-08-14 23:17:58,609 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-10) Failed to decrypt value for property AttestationTruststorePass 
will be used encrypted value: javax.crypto.BadPaddingException: Data must start with zero
        at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:325) [rt.jar:1.7.0_25]
        at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:272) [rt.jar:1.7.0_25]
        at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:356) [sunjce_provider.jar:1.7.0_25]
        at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:382) [sunjce_provider.jar:1.7.0_25]
        at javax.crypto.Cipher.doFinal(Cipher.java:1922) [jce.jar:1.7.0_25]
        at org.ovirt.engine.core.utils.crypt.EngineEncryptionUtils.decrypt(EngineEncryptionUtils.java:169) [utils.jar:]
        at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.GetValue(DBConfigUtils.java:109) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.updateOption(DBConfigUtils.java:277) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.refreshVdcOptionCache(DBConfigUtils.java:36) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:168) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:154) [dal.jar:]
        at org.ovirt.engine.core.bll.Backend.Initialize(Backend.java:182) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.create(Backend.java:129) [bll.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMeth
odInterceptorFactory.java:130) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.ja
r:7.2.0.Final-redhat-8]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
…
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
        at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view30.RunPublicQuery(Unknown Source) [common.jar:]
        at org.ovirt.engine.ui.frontend.server.gwt.GwtDynamicHostPageServlet.runPublicQuery(GwtDynamicHostPageServlet.java:209)
        at org.ovirt.engine.ui.frontend.server.gwt.WebAdminHostPageServlet.getApplicationMode(WebAdminHostPageServlet.java:80)
        at org.ovirt.engine.ui.frontend.server.gwt.WebAdminHostPageServlet.doGet(WebAdminHostPageServlet.java:56)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
        at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:465)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488)
        at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
        at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

2013-08-14 23:17:58,680 INFO  [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-10) Running ovirt-engine 3.3.0-0.15.master.el6ev
2013-08-14 23:17:58,680 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-10) Start initializing dictionaries
2013-08-14 23:17:58,681 INFO  [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-10) Finished initializing dictionaries

--------------------------------------------------------------------------------
JBWEB000065: HTTP Status 500 - java.lang.IllegalStateException: JBAS011048: Failed to construct component instance

JBWEB000309: type JBWEB000066: Exception report

JBWEB000068: message java.lang.IllegalStateException: JBAS011048: Failed to construct component instance

JBWEB000069: description JBWEB000145: The server encountered an internal error that prevented it from fulfilling this request.

JBWEB000070: exception

javax.ejb.EJBException: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
	org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:191)
	org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:237)
	org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
	org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
	org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
	org.ovirt.engine.core.common.interfaces.BackendLocal$$$view30.RunPublicQuery(Unknown Source)
	org.ovirt.engine.core.WelcomeServlet.doGet(WelcomeServlet.java:104)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64)

JBWEB000071: root cause

java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
	org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
	org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
	org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:126)
	org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:47)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235)
	org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374)
	org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
	org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)

/var/log/vdsm/vdsm.log
Comment 3 Doron Fediuck 2013-08-20 10:26:38 EDT
The decryption error is not relevant, and happens for other values as well.

This seems to be the error flow, based on server_kipi-rhevm.log:

2013-08-14 23:07:01,756 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
	at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57) [jboss-as-ee.jar:7.2.0.Final-redhat-8]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_25]

.....
.....

Caused by: java.lang.ClassCastException: org.ovirt.engine.core.utils.timer.SchedulerUtil$$$view2 cannot be cast to org.ovirt.engine.core.utils.timer.SchedulerUtil
	at org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl.getInstance(SchedulerUtilQuartzImpl.java:108)
	at org.ovirt.engine.core.bll.AuditLogCleanupManager.<init>(AuditLogCleanupManager.java:35)
	at org.ovirt.engine.core.bll.AuditLogCleanupManager.<clinit>(AuditLogCleanupManager.java:19)
	... 39 more


Since both AuditLogCleanupManager.java and SchedulerUtilQuartzImpl.java were not changed for several months,
this may be due to a jboss change or something similar.
Moving to infra for upgrade investigation.
Comment 4 Eli Mesika 2013-08-21 02:57:16 EDT
I had found this thread from the nogah days in jboss forum. Seems related 
https://community.jboss.org/thread/172462
(See Livnat summary at the last comment...)

Oved , please advice
Comment 5 Oved Ourfali 2013-08-21 03:51:40 EDT
Found this JIRA:
https://issues.jboss.org/browse/AS7-1658

They suggest to move to remote EJB interfaces, but I guess that might have performance issues. Added a comment there regarding that.
Is that new? Does it reproduce? Does it happen only after upgrade?
Comment 7 vvyazmin@redhat.com 2013-08-22 04:36:29 EDT
(In reply to Oved Ourfali from comment #5)
> Found this JIRA:
> https://issues.jboss.org/browse/AS7-1658
> 
> They suggest to move to remote EJB interfaces, but I guess that might have
> performance issues. Added a comment there regarding that.
> Is that new? Does it reproduce? Does it happen only after upgrade?

Is that new? - Yes
Does it reproduce? - No, only in upgrade flow, from RHEVM 3.3 IS9.1 to IS10 
Does it happen only after upgrade? - Yes

My workaround was, install all RHEVM 3.3 IS10 environment from scratch.
Comment 8 Juan Hernández 2013-08-26 08:00:50 EDT
This shouldn't happen unless we have the SchedulerUtil class in two different places, which we shouldn't. It should be only in the scheduler.jar file, in /usr/share/java/ovirt-engine/scheduler.jar. I would suggest to check, after the upgrade, that this class isn't in any other file, specially in files inside /usr/share/ovirt-engine/engine.ear/*.jar and /usr/share/ovirt-engine/engine.ear/lib/*.jar.
Comment 9 Barak 2013-08-26 10:21:04 EDT
Vlad,

Do you still have this environment ?
We need to take a look in it per comment #8.

if you don't have it - we need to take a look at a reproduced environment.
Comment 10 vvyazmin@redhat.com 2013-09-22 09:48:13 EDT
No, I haven’t environment. In latest versions RHEVM 3.3 - IS 15, this bug not reproduced.
Comment 11 Barak 2013-09-22 11:16:59 EDT
Per comment #10 this can not be reproduced on IS15,
Hence moving to CLOSE NOTABUG

Note You need to log in before you can comment on or make changes to this bug.