Bug 780840 (SOA-3291)

Summary: java.lang.NoClassDefFoundError: sun/reflect/ConstructorAccessorImpl after redeploying bpm_orchestration quickstart multiple times
Product: [JBoss] JBoss Enterprise SOA Platform 5 Reporter: Martin Weiler <mweiler>
Component: JBPM - within SOAAssignee: Martin Weiler <mweiler>
Status: CLOSED DEFERRED QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 5.1.0 GACC: kevin.conner
Target Milestone: ---   
Target Release: 5.2.0 GA   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/SOA-3291
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-09-01 10:57:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Martin Weiler 2011-08-23 11:12:52 UTC
Help Desk Ticket Reference: https://na7.salesforce.com/500A0000007rDjY
Steps to Reproduce: 1. In bpm_orchestration1: ant deployProcess
2. ant deploy
3. ant startProcess

repeat steps 2 and 3 a couple of times
Workaround: Workaround Exists
Workaround Description: Before (re)deploying the ESB deployment, first run {{ant undeploy}}

In short, when repeating deployments, follow the following steps: 
1. ant deploy
2. ant deployProcess
3. ant startProcess
4. ant undeploy

Obviously, please wait for each step to complete before going on to the next step or restarting the cycle. 
project_key: SOA

After calling 'ant deploy' and 'ant startProcess' multiple times, process execution fails with the following error:

{code}
13:01:11,490 ERROR [JmsServerSession] (WorkManager(2)-75:) Unexpected error delivering message delegator->JBossMessage[5382542423785572]:PERSISTENT, deliveryId=38
java.lang.NoClassDefFoundError: sun/reflect/ConstructorAccessorImpl
	at sun.misc.Unsafe.defineClass(Native Method)
	at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
	at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
	at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:94)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:48)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
	at java.lang.Class.newInstance0(Class.java:372)
	at java.lang.Class.newInstance(Class.java:325)
	at org.jbpm.instantiation.FieldInstantiator.newInstance(FieldInstantiator.java:115)
	at org.jbpm.instantiation.FieldInstantiator.instantiate(FieldInstantiator.java:53)
	at org.jbpm.instantiation.Delegation.instantiate(Delegation.java:147)
	at org.jbpm.instantiation.Delegation.getInstance(Delegation.java:106)
	at sun.reflect.GeneratedMethodAccessor456.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
	at org.jbpm.instantiation.Delegation_$$_javassist_77.getInstance(Delegation_$$_javassist_77.java)
	at org.jbpm.graph.def.Action.execute(Action.java:115)
	at org.jbpm.graph.def.GraphElement.executeActionImpl(GraphElement.java:300)
	at org.jbpm.graph.def.GraphElement.executeAction(GraphElement.java:267)
	at org.jbpm.graph.def.GraphElement.executeActions(GraphElement.java:240)
	at org.jbpm.graph.def.GraphElement.fireAndPropagateEvent(GraphElement.java:206)
	at org.jbpm.graph.def.GraphElement.fireEvent(GraphElement.java:188)
	at org.jbpm.graph.def.Node.enter(Node.java:368)
	at sun.reflect.GeneratedMethodAccessor492.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
	at org.jbpm.graph.def.Node_$$_javassist_148.enter(Node_$$_javassist_148.java)
	at org.jbpm.graph.def.Transition.take(Transition.java:162)
	at org.jbpm.graph.def.Node.leave(Node.java:450)
	at sun.reflect.GeneratedMethodAccessor488.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
	at org.jbpm.graph.def.Node_$$_javassist_148.leave(Node_$$_javassist_148.java)
	at org.jbpm.graph.exe.Token.signal(Token.java:210)
	at org.jbpm.graph.exe.Token.signal(Token.java:144)
	at sun.reflect.GeneratedMethodAccessor486.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
	at org.jbpm.graph.exe.Token_$$_javassist_80.signal(Token_$$_javassist_80.java)
	at org.jbpm.command.SignalCommand.execute(SignalCommand.java:89)
	at org.jboss.soa.esb.services.jbpm.cmd.AsyncProcessSignal$AsyncSignalAction.execute(AsyncProcessSignal.java:300)
	at org.jbpm.graph.def.Action.execute(Action.java:117)
	at sun.reflect.GeneratedMethodAccessor469.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
	at org.jbpm.graph.def.Action_$$_javassist_100.execute(Action_$$_javassist_100.java)
	at org.jbpm.graph.def.GraphElement.executeActionImpl(GraphElement.java:300)
	at org.jbpm.graph.def.GraphElement.executeAction(GraphElement.java:274)
	at sun.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
	at org.jbpm.graph.def.Node_$$_javassist_148.executeAction(Node_$$_javassist_148.java)
	at org.jbpm.job.ExecuteActionJob.execute(ExecuteActionJob.java:30)
	at sun.reflect.GeneratedMethodAccessor481.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
	at org.jbpm.job.Job_$$_javassist_152.execute(Job_$$_javassist_152.java)
	at org.jboss.soa.esb.services.jbpm.integration.job.ExecuteJobCommand.executeJob(ExecuteJobCommand.java:124)
	at org.jboss.soa.esb.services.jbpm.integration.job.ExecuteJobCommand.execute(ExecuteJobCommand.java:92)
	at org.jboss.soa.esb.services.jbpm.integration.command.AbstractMessageListener.onMessage(AbstractMessageListener.java:66)
	at org.jboss.soa.esb.services.jbpm.integration.command.JobListener.onMessage(JobListener.java:16)
	at sun.reflect.GeneratedMethodAccessor468.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.jboss.internal.soa.esb.dependencies.JCAInflowAdapter$1.invoke(JCAInflowAdapter.java:276)
	at org.jboss.soa.esb.listeners.jca.EndpointProxy.delivery(EndpointProxy.java:258)
	at org.jboss.soa.esb.listeners.jca.EndpointProxy.invoke(EndpointProxy.java:150)
	at $Proxy302.onMessage(Unknown Source)
	at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:179)
	at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
	at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
	at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_1744231521.invoke(SessionAspect_z_handleRun_1744231521.java)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
	at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
	at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:236)
	at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
	at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.ClassNotFoundException: Class not found: sun.reflect.ConstructorAccessorImpl
	at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:873)
	at org.jboss.classloader.spi.base.BaseClassLoader.doLoadClass(BaseClassLoader.java:502)
	at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:447)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
	... 94 more
Caused by: java.lang.IllegalStateException: BaseClassLoader@4ec99007{vfsfile:/data/jboss/work/jboss-soa-p-5.1.0/jboss-as/server/test/deploy/Quickstart_bpm_orchestration1.esb/} classLoader is not connected to a domain (probably undeployed?) for class sun.reflect.ConstructorAccessorImpl
	at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:857)
	... 97 more
{code}

Comment 1 (please assign to mrietvel@redhat.com) 2011-08-30 16:50:43 UTC
Link: Added: This issue is related to JBESB-3582


Comment 2 (please assign to mrietvel@redhat.com) 2011-09-01 10:43:41 UTC
Things tried: 
- adding jboss-classloader.xml to the .esb deployable
-- a. with a top-level classloader as described in [1]
-- b. with isolation with a specific parent (9.9 in [2])

The problem with b. is that the par/jbpm process deployment can't share the same subdomain as the classloader specified for the ESB classloader, so that when the process is processed by jBPM, it can't find the classses deployed in the ESB deployment. 

See, for more info: 
[1] http://community.jboss.org/wiki/useJBossWebClassLoaderinJBoss5
[2] [Microcontainer Classloading Documentation|http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Web_Platform/5/html/JBoss_Microcontainer_User_Guide/sect-JBoss_Microcontainer_User_Guide-The_ClassLoading_Layer-ClassLoading.html] (includes explanation of jboss-classloader.xml)

Comment 3 (please assign to mrietvel@redhat.com) 2011-09-01 10:47:05 UTC
(BTW, a. above was just to see what would happen.. and re b. above -- .par deployments don't look at jboss-classloading.xml files)

The solution was actually fairly simple: undeploy the ESB and redploy the process. I'm currently running the following bash script, and it's already at iteration 13 -- most of the time this problem showed up by the 8th or 9th redployment. 

{code}
d=0
while date
  do ant deploy
  s=10
  echo -e "---\ndeploy $[++d] ($s)\n---" &&  sleep $s
  ant deployProcess
  s=5
  echo -e "---\nprocess $d ($s)\n---" && sleep $s
  ant startProcess
  s=15
  echo -e "---\nstart $d ($s)\n---" && sleep $s
  ant undeploy
  s=30
  echo -e "---\nundeploy $d ($s)\n---" && sleep $s
done
{code}


Comment 4 (please assign to mrietvel@redhat.com) 2011-09-01 10:52:12 UTC
Argh, not being able to edit my own comments is frustrating: I mean undeploy the ESB _deployment_ and redploy the ESB deployment and the process. 

Comment 5 (please assign to mrietvel@redhat.com) 2011-09-01 10:56:29 UTC
Workaround Description: Added: Before (re)deploying the ESB deployment, first run {{ant undeploy}}

In short, when repeating deployments, follow the following steps: 
1. ant deploy
2. ant deployProcess
3. ant startProcess
4. ant undeploy

Obviously, please wait for each step to complete before going on to the next step or restarting the cycle. 
Workaround: Added: [Workaround Exists]


Comment 6 (please assign to mrietvel@redhat.com) 2011-09-01 10:57:57 UTC
Workaround found: run ant undeploy before redeployment. See workaround part of issue, above. 

Comment 7 (please assign to mrietvel@redhat.com) 2011-09-01 11:08:23 UTC
FYI: The script ran 33 iterations (without any exceptions) before I stopped it. 

Comment 8 Kevin Conner 2011-09-01 14:46:59 UTC
Link: Removed: This issue is related to JBESB-3582 


Comment 9 Kevin Conner 2011-09-01 14:54:18 UTC
JBESB-3582 is not the same issue, that is related to multiple deployments sharing classes with one of them being undeployed.

It looks like a jbpm job is executing a task using a classloader that has been undeployed, so this will be some sort of race.


Comment 10 David Le Sage 2011-09-01 23:26:32 UTC
Release Notes Docs Status: Added: Documented as Resolved Issue
Writer: Added: dlesage
Release Notes Text: Added: https://issues.jboss.org/browse/SOA-3291

If the user redeploys the bpm_orchestration quick start multiple times, a race condition could occur, which lead to an java.lang.NoClassDefFoundError. A code fix has now been applied to address this.  As a result, users will no longer see this error.


Comment 11 (please assign to mrietvel@redhat.com) 2011-09-06 09:30:03 UTC
The MyESBActionHandler class, referenced in the process but deployed in the ESB deployable, is the class that causes the classloading err

Stack when the class is initially loaded: 
{code}
1. ClassLoadingTask$ThreadTask.run() line: 453	
   ClassLoaderManager.nextTask(Thread, ClassLoadingTask) line: 251	
   ClassLoaderManager.process(Thread, ClassLoadingTask) line: 150	
2. ClassLoaderDomain(BaseClassLoaderDomain).loadClass(BaseClassLoader, String, boolean) line: 265	
   ClassLoaderDomain(BaseClassLoaderDomain).loadClass(BaseClassLoader, String) line: 1119	
3. BaseClassLoader.loadClassFromDomain(String, boolean) line: 798	
   BaseClassLoader.loadClass(String, boolean) line: 441	
4. ProcessClassLoader(ClassLoader).loadClass(String, boolean) line: 296	
   ProcessClassLoader(ClassLoader).loadClass(String) line: 248	
   Class<T>.forName0(String, boolean, ClassLoader) line: not available [native method]	
   Class<T>.forName(String, boolean, ClassLoader) line: 247	
   Delegation.instantiate() line: 145	
{code}

{noformat}
BaseClassLoader  [3] --> ClassLoaderDomain [2] --> FilteredDelegateLoader [1]
       |
ProcessClassLoader [4]
{noformat}

4 delegates to 3, which then finds 2 and creates 1 to load the class.

Comment 12 (please assign to mrietvel@redhat.com) 2011-09-06 10:02:08 UTC
Link: Added: This issue is related to JBPM-2428