Bug 1117424

Summary: Dashbuilder log4j classloading conflict when CA SiteMinder TAI is installed in WAS 8
Product: [Retired] JBoss BPMS Platform 6 Reporter: Josh West <jowest>
Component: BAMAssignee: David Gutierrez <dgutierr>
Status: CLOSED WONTFIX QA Contact: Radovan Synek <rsynek>
Severity: unspecified Docs Contact:
Priority: high    
Version: 6.0.2CC: alazarot, kverlaen, mwinkler
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1119512 (view as bug list) Environment:
Last Closed: 2014-07-22 17:30:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1119512    
Bug Blocks: 1115481    
Attachments:
Description Flags
Dashbuilder boot with SSO and parent *First* classloading
none
Dashbuilder boot with SSO and parent *Last* classloading none

Description Josh West 2014-07-08 16:36:35 UTC
Description of problem:

Certain scenarios in dashbuilder causes a classloading error (java.lang.NoSuchFieldError: org/apache/log4j/Level.TRACE) to arise due to a missing/mismatch log4j file. WAS8 contains a different log4j version than the BPMS application is compiled against.

To work around the issue
a) copy the log4j file 
     from `./business-central.war/WEB-INF/lib/log4j-1.2.16-redhat-2.jar` 
     to `./dashbuilder.war/WEB-INF/lib/log4j-1.2.16-redhat-2.jar`
b) configure the Dashbuilder.war file to have parent last classloading


Version-Release number of selected component (if applicable):

JBoss BPMS 6.0.2 GA on WebSphere Application Server 8.5

How reproducible:

100%

Steps to Reproduce:
one scenario is a misconfigured datasource (jdbc/dashbuilder) but other scenarios are likely.
1. Deploy Dasbuilder on top of WAS 8
2. Do not create the jdbc/dashbuilder datasource
3. Start the application container / app
4. log into dashbuilder

** NOTE this happened in a environment with SSO configured. This should be reproducable through the steps above alone, but alternative reproduction methods may need to be found.

Actual results:

Exception in "additional info" arises.

Expected results:

No exception mentioning missing class.

Additional info:

/opt/middleware/jbpmwas/jbpmwas_Runtime/profiles/jbpmwasCell/jbpmwasProfile/installedApps/jbpmwasCell/dashbuilder_war.ear/dashbuilder.war/./WEB-INF/etc/factory
[7/8/14 11:16:30:117 EDT] 0000009d Factory       I org.jboss.dashboard.factory.Factory addDescriptorFiles Factory configured.
[7/8/14 11:16:30:237 EDT] 0000009d SystemErr     R   log4j:WARN No appenders could be found for logger (org.jboss.logging).
[7/8/14 11:16:30:237 EDT] 0000009d SystemErr     R   log4j:WARN Please initialize the log4j system properly.
[7/8/14 11:16:30:496 EDT] 0000009d StartableProc E org.jboss.dashboard.annotation.StartableProcessor wakeUpStartableBeans Error starting bean org.jboss.dashboard.database.hibernate.HibernateInitializer_$$_javassist_42
                                 java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:607)
	at org.apache.webbeans.intercept.InterceptorHandler.invoke(InterceptorHandler.java:297)
	at org.apache.webbeans.intercept.NormalScopedBeanInterceptorHandler.invoke(NormalScopedBeanInterceptorHandler.java:98)
	at org.jboss.dashboard.database.hibernate.HibernateInitializer_$$_javassist_42.start(HibernateInitializer_$$_javassist_42.java)
	at org.jboss.dashboard.annotation.StartableProcessor.wakeUpStartableBeans(StartableProcessor.java:48)
	at org.jboss.dashboard.Application.start(Application.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:607)
	at org.apache.webbeans.intercept.InterceptorHandler.invoke(InterceptorHandler.java:297)
	at org.apache.webbeans.intercept.NormalScopedBeanInterceptorHandler.invoke(NormalScopedBeanInterceptorHandler.java:98)
	at org.jboss.dashboard.Application_$$_javassist_29.start(Application_$$_javassist_29.java)
	at org.jboss.dashboard.ui.controller.ControllerServlet$1.doWork(ControllerServlet.java:73)
	at org.jboss.dashboard.factory.Factory.runWork(Factory.java:295)
	at org.jboss.dashboard.factory.Factory.doWork(Factory.java:321)
	at org.jboss.dashboard.ui.controller.ControllerServlet.init(ControllerServlet.java:70)
	at javax.servlet.GenericServlet.init(GenericServlet.java:161)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:342)
	at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:168)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1366)
	at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:610)
	at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:580)
	at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:425)
	at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
	at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)
	at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)
	at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)
	at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)
	at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)
	at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1175)
	at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)
	at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
	at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)
	at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:774)
	at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2182)
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)
	at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)
	at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
	at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
Caused by: java.lang.NoSuchFieldError: org/apache/log4j/Level.TRACE
	at org.jboss.logging.Log4jLogger.translate(Log4jLogger.java:60)
	at org.jboss.logging.Log4jLogger.isEnabled(Log4jLogger.java:35)
	at org.jboss.logging.Logger.logv(Logger.java:1950)
	at org.jboss.logging.DelegatingBasicLogger.tracev(DelegatingBasicLogger.java:89)
	at org.hibernate.cfg.EJB3DTDEntityResolver.resolveEntity(EJB3DTDEntityResolver.java:58)
	at org.apache.xerces.util.EntityResolverWrapper.resolveEntity(Unknown Source)
	at org.apache.xerces.impl.XMLEntityManager.resolveEntity(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentScannerImpl$DTDDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at org.dom4j.io.SAXReader.read(SAXReader.java:465)
	at org.hibernate.cfg.Configuration.doConfigure(Configuration.java:2027)
	at org.hibernate.cfg.Configuration.configure(Configuration.java:2005)
	at org.jboss.dashboard.database.hibernate.HibernateInitializer.start(HibernateInitializer.java:117)
	... 46 more

Comment 2 Radovan Synek 2014-07-09 08:31:47 UTC
> To work around the issue
> a) copy the log4j file 
>      from `./business-central.war/WEB-INF/lib/log4j-1.2.16-redhat-2.jar` 
>      to `./dashbuilder.war/WEB-INF/lib/log4j-1.2.16-redhat-2.jar`
> b) configure the Dashbuilder.war file to have parent last classloading

Josh, I am not sure that changing classloading to "parent last" is a suitable workaround, this setting is not advised to change. Please see bug 1095391 and corresponding bug 1095363, comment 6

Comment 3 Josh West 2014-07-09 15:11:39 UTC
You are right. We just did more testing and the ProxyObject class cast exception comes up.  The issue still remains, but the workaround is mentioned is no good.

Comment 4 Josh West 2014-07-09 15:40:19 UTC
This customer has two test environments. One basic vanilla WAS installtion, and a second configured for SSO with CA SiteMinder TAI.  The Log4j trace exception is only happening in the SSO environment.

Comment 5 Josh West 2014-07-09 15:43:57 UTC
Created attachment 916867 [details]
Dashbuilder boot with SSO and parent *First* classloading

Comment 6 Josh West 2014-07-09 15:44:27 UTC
Created attachment 916868 [details]
Dashbuilder boot with SSO and parent *Last* classloading

Comment 7 Josh West 2014-07-10 17:22:58 UTC
Additional findings from analysis/testing/troubleshooting with the customer. 

The Log4j exception is being caused only in an environment with CA SiteMinder TAI (SMTAI) installed.  SMTAI includes an old version of log4j at a higher level of the classpath than the dashbuilder.war that does not have the Level.TRACE enum field defined.  Since log4j is initialized by SMTAI Hibernate's jboss logging detects it and uses attempts to use it - causing the initialization of a hibernate session to fail.  Hibernate is intended to use the "jdk" logger instead of log4j.  

The fix for this issue is to set the java system property "org.jboss.logging.provider" to "jdk" in WebSphere. This forces hibernate within dashbuilder to not use log4j.  

Testing on the customer side confirms that Dashbuilder boots in environments with CA Siteminder TAI installed. 

For this BZ we should consider:
a) adding the java system property update to the installation directions, or
b) setting the `org.jboss.logging.provider` property through a property file embedded in the WAS8 WAR file that we ship.

Comment 9 David Gutierrez 2014-07-22 16:46:57 UTC
This has raised the following documentacion BZ https://bugzilla.redhat.com/show_bug.cgi?id=1119512

As the fix did not require any source code modifications I'm closing this BZ.