Created attachment 786700 [details] ## Logs rhevm, vdsm, libvirt, thread dump, superVdsm Description of problem: Failed open RHEVM URL after upgrade - HTTP Status 500 - java.lang.IllegalStateException: JBAS011048: Failed to construct component instance Version-Release number of selected component (if applicable): RHEVM 3.3 - IS10 environment: RHEVM: rhevm-3.3.0-0.15.master.el6ev.noarch PythonSDK: rhevm-sdk-python-3.3.0.10-1.el6ev.noarch VDSM: vdsm-4.12.0-61.git8178ec2.el6ev.x86_64 LIBVIRT: libvirt-0.10.2-18.el6_4.9.x86_64 QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.355.el6_4.5.x86_64 SANLOCK: sanlock-2.8-1.el6.x86_64 How reproducible: unknown Steps to Reproduce: Upgrade RHEVM 3.3 from IS9.1 to IS10 Actual results: Upgrade process worked OK, but failed open RHEVM URL - HTTP Status 500 - java.lang.IllegalStateException: JBAS011048: Failed to construct component instance Expected results: Succeed open RHEVM Impact on user: Failed RHEVM URL Workaround: Reboot RHEVM machine Additional info: /var/log/ovirt-engine/engine.log 2013-08-14 23:17:58,609 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-10) Failed to decrypt value for property AttestationTruststorePass will be used encrypted value: javax.crypto.BadPaddingException: Data must start with zero at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:325) [rt.jar:1.7.0_25] at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:272) [rt.jar:1.7.0_25] at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:356) [sunjce_provider.jar:1.7.0_25] at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:382) [sunjce_provider.jar:1.7.0_25] at javax.crypto.Cipher.doFinal(Cipher.java:1922) [jce.jar:1.7.0_25] at org.ovirt.engine.core.utils.crypt.EngineEncryptionUtils.decrypt(EngineEncryptionUtils.java:169) [utils.jar:] at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.GetValue(DBConfigUtils.java:109) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.updateOption(DBConfigUtils.java:277) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.refreshVdcOptionCache(DBConfigUtils.java:36) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:168) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:154) [dal.jar:] at org.ovirt.engine.core.bll.Backend.Initialize(Backend.java:182) [bll.jar:] at org.ovirt.engine.core.bll.Backend.create(Backend.java:129) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMeth odInterceptorFactory.java:130) [jboss-as-ee.jar:7.2.0.Final-redhat-8] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.ja r:7.2.0.Final-redhat-8] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.2.0.Final-redhat-8] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.2.0.Final-redhat-8] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8] at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.2.0.Final-redhat-8] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.2.0.Final-redhat-8] … at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.2.0.Final-redhat-8] at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view30.RunPublicQuery(Unknown Source) [common.jar:] at org.ovirt.engine.ui.frontend.server.gwt.GwtDynamicHostPageServlet.runPublicQuery(GwtDynamicHostPageServlet.java:209) at org.ovirt.engine.ui.frontend.server.gwt.WebAdminHostPageServlet.getApplicationMode(WebAdminHostPageServlet.java:80) at org.ovirt.engine.ui.frontend.server.gwt.WebAdminHostPageServlet.doGet(WebAdminHostPageServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:465) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] 2013-08-14 23:17:58,680 INFO [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8702-10) Running ovirt-engine 3.3.0-0.15.master.el6ev 2013-08-14 23:17:58,680 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-10) Start initializing dictionaries 2013-08-14 23:17:58,681 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ajp-/127.0.0.1:8702-10) Finished initializing dictionaries -------------------------------------------------------------------------------- JBWEB000065: HTTP Status 500 - java.lang.IllegalStateException: JBAS011048: Failed to construct component instance JBWEB000309: type JBWEB000066: Exception report JBWEB000068: message java.lang.IllegalStateException: JBAS011048: Failed to construct component instance JBWEB000069: description JBWEB000145: The server encountered an internal error that prevented it from fulfilling this request. JBWEB000070: exception javax.ejb.EJBException: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:191) org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:237) org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) org.ovirt.engine.core.common.interfaces.BackendLocal$$$view30.RunPublicQuery(Unknown Source) org.ovirt.engine.core.WelcomeServlet.doGet(WelcomeServlet.java:104) javax.servlet.http.HttpServlet.service(HttpServlet.java:734) javax.servlet.http.HttpServlet.service(HttpServlet.java:847) org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) JBWEB000071: root cause java.lang.IllegalStateException: JBAS011048: Failed to construct component instance org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163) org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85) org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:126) org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:47) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235) org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:218) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) /var/log/vdsm/vdsm.log
The decryption error is not relevant, and happens for other values as well. This seems to be the error flow, based on server_kipi-rhevm.log: 2013-08-14 23:07:01,756 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57) [jboss-as-ee.jar:7.2.0.Final-redhat-8] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_25] ..... ..... Caused by: java.lang.ClassCastException: org.ovirt.engine.core.utils.timer.SchedulerUtil$$$view2 cannot be cast to org.ovirt.engine.core.utils.timer.SchedulerUtil at org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl.getInstance(SchedulerUtilQuartzImpl.java:108) at org.ovirt.engine.core.bll.AuditLogCleanupManager.<init>(AuditLogCleanupManager.java:35) at org.ovirt.engine.core.bll.AuditLogCleanupManager.<clinit>(AuditLogCleanupManager.java:19) ... 39 more Since both AuditLogCleanupManager.java and SchedulerUtilQuartzImpl.java were not changed for several months, this may be due to a jboss change or something similar. Moving to infra for upgrade investigation.
I had found this thread from the nogah days in jboss forum. Seems related https://community.jboss.org/thread/172462 (See Livnat summary at the last comment...) Oved , please advice
Found this JIRA: https://issues.jboss.org/browse/AS7-1658 They suggest to move to remote EJB interfaces, but I guess that might have performance issues. Added a comment there regarding that. Is that new? Does it reproduce? Does it happen only after upgrade?
(In reply to Oved Ourfali from comment #5) > Found this JIRA: > https://issues.jboss.org/browse/AS7-1658 > > They suggest to move to remote EJB interfaces, but I guess that might have > performance issues. Added a comment there regarding that. > Is that new? Does it reproduce? Does it happen only after upgrade? Is that new? - Yes Does it reproduce? - No, only in upgrade flow, from RHEVM 3.3 IS9.1 to IS10 Does it happen only after upgrade? - Yes My workaround was, install all RHEVM 3.3 IS10 environment from scratch.
This shouldn't happen unless we have the SchedulerUtil class in two different places, which we shouldn't. It should be only in the scheduler.jar file, in /usr/share/java/ovirt-engine/scheduler.jar. I would suggest to check, after the upgrade, that this class isn't in any other file, specially in files inside /usr/share/ovirt-engine/engine.ear/*.jar and /usr/share/ovirt-engine/engine.ear/lib/*.jar.
Vlad, Do you still have this environment ? We need to take a look in it per comment #8. if you don't have it - we need to take a look at a reproduced environment.
No, I haven’t environment. In latest versions RHEVM 3.3 - IS 15, this bug not reproduced.
Per comment #10 this can not be reproduced on IS15, Hence moving to CLOSE NOTABUG