Bug 779175 (SOA-1570)

Summary: Rule Agent used by a BusinessRulesProcessor still works even after esb archive gets undeployed
Product: [JBoss] JBoss Enterprise SOA Platform 5 Reporter: Jaroslaw Kijanowski <kijanowski>
Component: JBoss RulesAssignee: Kevin Conner <kevin.conner>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 5.0.0 ER1   
Target Milestone: ---   
Target Release: 5.0.0 GA, 5.0.0 ER5   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/SOA-1570
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-12-11 10:28:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
server.log
none
server.log.zip none

Description Jaroslaw Kijanowski 2009-10-27 22:30:32 UTC
Date of First Response: 2009-11-30 11:22:10
project_key: SOA

I have a quickstart that uses a rule agent to grab rules from the BRMS.
Details can be found at
https://docspace.corp.redhat.com/clearspace/docs/DOC-27544

There are issues when I undeploy the BRMS:

1) start SOA-P with BRMS
2) import repository (brms-repo/repository.xml)
3) deploy QS:
ant deploy -Dsoa.ip=<SOA_IP>

4) run QS:
ant run-test -Dsoa.ip=<SOA_IP>

5) undeploy QS:
ant undeploy

6) undeploy BRMS - a lot of errors show up (see attached server.log).
Quite interesting is this part, looks like the agent is still trying to scan the BRMS URL, however the quickstart that used it has been undeployed:

18:20:07,332 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Was an error contacting http://10.16.93.36:8080/jboss-brms/org.drools.guvnor.Guvnor/package/soa/LATEST. Reponse header: {null=[HTTP/1.1 500 Internal Server Error], Transfer-Encoding=[chunked], Date=[Tue, 27 Oct 2009 22:20:07 GMT], Content-Type=[text/html;charset=utf-8], Connection=[close], Server=[Apache-Coyote/1.1]}
18:20:07,345 ERROR [LogAgentEventListener] AgentName = 'insuranceconfig' Exception: Was unable to reach server.
java.io.IOException: Was unable to reach server.
	at org.drools.agent.URLScanner.hasChanged(URLScanner.java:155)
	at org.drools.agent.URLScanner.getChangeSet(URLScanner.java:119)
	at org.drools.agent.URLScanner.loadPackageChanges(URLScanner.java:96)
	at org.drools.agent.RuleAgent.checkForChanges(RuleAgent.java:410)
	at org.drools.agent.RuleAgent.refreshRuleBase(RuleAgent.java:362)
	at org.drools.agent.RuleAgent$2.run(RuleAgent.java:516)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
18:20:07,346 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Falling back to local cache.
18:20:07,370 INFO  [LogAgentEventListener] AgentName = 'insuranceconfig' Applying changes to the rulebase.
18:20:07,371 INFO  [LogAgentEventListener] AgentName = 'insuranceconfig' Creating a new rulebase as per settings.
18:20:07,371 INFO  [LogAgentEventListener] AgentName = 'insuranceconfig' Adding package called soa
18:20:07,373 ERROR [LogAgentEventListener] AgentName = 'insuranceconfig' Exception: java.lang.IllegalStateException: BaseClassLoader@e9cfa3{vfsfile:/qa/home/jkijanowski/SOA-P-5/er1/jboss-soa-p.5.0.0/jboss-as/server/production/deploy/Quickstart_brms.esb/} classLoader is not connected to a domain (probably undeployed?) for class soa.450
org.drools.RuntimeDroolsException: java.lang.IllegalStateException: BaseClassLoader@e9cfa3{vfsfile:/qa/home/jkijanowski/SOA-P-5/er1/jboss-soa-p.5.0.0/jboss-as/server/production/deploy/Quickstart_brms.esb/} classLoader is not connected to a domain (probably undeployed?) for class soa.450
	at org.drools.agent.PackageProvider.applyChanges(PackageProvider.java:87)
	at org.drools.agent.PackageProvider.applyChanges(PackageProvider.java:62)
	at org.drools.agent.RuleAgent.refreshRuleBase(RuleAgent.java:392)
	at org.drools.agent.RuleAgent$2.run(RuleAgent.java:516)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)

Comment 1 Jaroslaw Kijanowski 2009-10-27 22:30:56 UTC
Attachment: Added: server.log


Comment 2 Jaroslaw Kijanowski 2009-11-30 15:58:04 UTC
Link: Added: This issue is a dependency of JBESB-3011


Comment 3 Kevin Conner 2009-11-30 16:22:10 UTC
As far as I can remember we have never done any integration testing with BRMS, nor has it been part of the current requirements.  Can you please tell me where you are getting the requirement from?

Comment 4 Kevin Conner 2009-11-30 16:28:54 UTC
Found JBESB-643 which added it into 4.4.

Comment 5 Kevin Conner 2009-11-30 16:40:02 UTC
Just out of interest, which version of BRMS are you running?  JBESB-643 was done a long time back and there has been no further work to upgrade that integration.


Comment 6 Jaroslaw Kijanowski 2009-11-30 18:54:43 UTC
Hey Kevin,
 it's in the ERD, 4.1.19 - BRMS Support
The PRD has more details:
======
Delivery of proper version of a rule package via the ruleagent to any ESB node.  ESB containers allow for hot deployment/reload of the changed rules without component restart and no loss of transactions/messages.
This diagram shows three different ESB nodes, each running unique services that use JBoss Rules.  The RuleAgent is responsible for polling the BRMS/Repository and reloading any changed rule packages while the ESB is responsible for insuring no messages are lost during the rule reload.
======

For this purpose I use the BRMS-5.0.1 deployed in EAP-4.3 CP06

Comment 7 Kevin Conner 2009-11-30 19:04:46 UTC
We already support that functionality through the current integration.  What seems to have been missed in this is the upgrade to another version of BRMS.

I am chasing this up through other means at the moment, will update you once we get something.

Comment 8 Kevin Conner 2009-11-30 19:35:30 UTC
I have downloaded BRMS 5, will go through this and see if there are other issues to be addressed.

Comment 9 trev 2009-12-02 11:31:48 UTC
jar sent to jarek for testing

Comment 10 Jaroslaw Kijanowski 2009-12-02 19:05:18 UTC
Used the new jar (md5sum):
jbossesb-jbrules.jar 128e3a91ddbd8f0924f3f0acde01943b


Now it's worse (zipped server log attached), when I run the QS for the first time I get:

======================================
======================================

2009-12-02 13:48:44,691 DEBUG [org.jboss.soa.esb.lifecycle.LifecycleResource] (pool-26-thread-1) Creating resource using factory: org.jboss.internal.soa.esb.services.rules.DroolsRuleService$LifecycleRuleBaseFactory@3a6012 with identity ID-7
2009-12-02 13:48:44,691 INFO  [org.jboss.internal.soa.esb.services.rules.DroolsRuleService] (pool-26-thread-1) destroying stateful session for ID-7
2009-12-02 13:48:44,691 DEBUG [org.jboss.soa.esb.listeners.message.ActionProcessingPipeline] (pool-26-thread-1) Unexpected exception caught while processing the action pipeline: header: [ To: JMSEpr [ PortReference < <wsa:Address jms:10.16.94.134:1099#queue/quickstart_brms_Request_ESB/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 10.16.94.134:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:destination-name : queue/quickstart_brms_Request_ESB/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] MessageID: e341b748-e997-4af3-a33c-bd662f498782 RelatesTo: jms:correlationID#e341b748-e997-4af3-a33c-bd662f498782 ]
org.jboss.soa.esb.actions.ActionProcessingException: RuleAgent could not get the RuleBase. null
	at org.jboss.soa.esb.actions.BusinessRulesProcessor.process(BusinessRulesProcessor.java:100)
	at org.jboss.soa.esb.listeners.message.ActionProcessingPipeline.processPipeline(ActionProcessingPipeline.java:634)
	at org.jboss.soa.esb.listeners.message.ActionProcessingPipeline.processPipeline(ActionProcessingPipeline.java:586)
	at org.jboss.soa.esb.listeners.message.ActionProcessingPipeline.process(ActionProcessingPipeline.java:420)
	at org.jboss.soa.esb.listeners.message.MessageAwareListener$TransactionalRunner.run(MessageAwareListener.java:540)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: org.jboss.internal.soa.esb.services.rules.RuleServiceException: RuleAgent could not get the RuleBase. null
	at org.jboss.internal.soa.esb.services.rules.DroolsRuleService.executeStatelessRulesFromRuleAgent(DroolsRuleService.java:247)
	at org.jboss.internal.soa.esb.services.rules.RuleServiceCallHelper.executeStateless(RuleServiceCallHelper.java:176)
	at org.jboss.internal.soa.esb.services.rules.RuleServiceCallHelper.executeRulesService(RuleServiceCallHelper.java:160)
	at org.jboss.soa.esb.actions.BusinessRulesProcessor.executeRulesService(BusinessRulesProcessor.java:111)
	at org.jboss.soa.esb.actions.BusinessRulesProcessor.process(BusinessRulesProcessor.java:92)
	... 7 more
Caused by: java.lang.NullPointerException
	at org.jboss.internal.soa.esb.services.rules.DroolsRuleService.getRuleBaseForRuleAgent(DroolsRuleService.java:829)
	at org.jboss.internal.soa.esb.services.rules.DroolsRuleService.executeStatelessRulesFromRuleAgent(DroolsRuleService.java:238)
	... 11 more
2009-12-02 13:48:44,703 WARN  [org.jboss.soa.esb.listeners.message.ActionProcessingPipeline] (pool-26-thread-1) No fault address defined for fault message! To: JMSEpr [ PortReference < <wsa:Address jms:10.16.94.134:1099#queue/quickstart_brms_Request_ESB/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 10.16.94.134:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:destination-name : queue/quickstart_brms_Request_ESB/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] MessageID: e341b748-e997-4af3-a33c-bd662f498782 RelatesTo: jms:correlationID#e341b748-e997-4af3-a33c-bd662f498782
2009-12-02 13:48:44,794 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] (pool-26-thread-1) Going to query the registry for name pattern [DeadLetterService]
2009-12-02 13:48:44,854 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] (pool-26-thread-1) Service name: DeadLetterService

======================================
======================================

it works fine when I run it for the second time...

Now when I undeploy the QS and *then* stop the BRMS, Im flooded with these:

======================================
======================================

13:59:43,812 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Was an error contacting http://10.16.94.36:8080/jboss-brms/org.drools.guvnor.Guvnor/package/soa/LATEST. Reponse header: {null=[HTTP/1.1 503 Service Unavailable], Date=[Wed, 02 Dec 2009 18:59:42 GMT], Content-Length=[1090], Content-Type=[text/html;charset=utf-8], Connection=[close], Server=[Apache-Coyote/1.1]}
13:59:43,812 ERROR [LogAgentEventListener] AgentName = 'insuranceconfig' Exception: Was unable to reach server.
java.io.IOException: Was unable to reach server.
	at org.drools.agent.URLScanner.hasChanged(URLScanner.java:155)
	at org.drools.agent.URLScanner.getChangeSet(URLScanner.java:119)
	at org.drools.agent.URLScanner.loadPackageChanges(URLScanner.java:96)
	at org.drools.agent.RuleAgent.checkForChanges(RuleAgent.java:410)
	at org.drools.agent.RuleAgent.refreshRuleBase(RuleAgent.java:362)
	at org.drools.agent.RuleAgent$2.run(RuleAgent.java:516)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
13:59:43,812 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Falling back to local cache.
13:59:43,826 INFO  [LogAgentEventListener] AgentName = 'insuranceconfig' Applying changes to the rulebase.
13:59:43,826 INFO  [LogAgentEventListener] AgentName = 'insuranceconfig' Creating a new rulebase as per settings.
13:59:43,827 INFO  [LogAgentEventListener] AgentName = 'insuranceconfig' Adding package called soa
13:59:43,841 ERROR [LogAgentEventListener] AgentName = 'insuranceconfig' Exception: java.lang.IllegalStateException: BaseClassLoader@35ebee{vfsfile:/qa/services/hudson/jkijanow/jboss-soa-p.5.0.0/jboss-as/server/production/deploy/Quickstart_brms.esb/} classLoader is not connected to a domain (probably undeployed?) for class soa.450
org.drools.RuntimeDroolsException: java.lang.IllegalStateException: BaseClassLoader@35ebee{vfsfile:/qa/services/hudson/jkijanow/jboss-soa-p.5.0.0/jboss-as/server/production/deploy/Quickstart_brms.esb/} classLoader is not connected to a domain (probably undeployed?) for class soa.450
	at org.drools.agent.PackageProvider.applyChanges(PackageProvider.java:87)
	at org.drools.agent.PackageProvider.applyChanges(PackageProvider.java:62)
	at org.drools.agent.RuleAgent.refreshRuleBase(RuleAgent.java:392)
	at org.drools.agent.RuleAgent$2.run(RuleAgent.java:516)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
Caused by: java.lang.IllegalStateException: BaseClassLoader@35ebee{vfsfile:/qa/services/hudson/jkijanow/jboss-soa-p.5.0.0/jboss-as/server/production/deploy/Quickstart_brms.esb/} classLoader is not connected to a domain (probably undeployed?) for class soa.450
	at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:793)
	at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.drools.rule.CompositeClassLoader.loadClass(CompositeClassLoader.java:61)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.mvel2.ParserConfiguration.checkForDynamicImport(ParserConfiguration.java:112)
	at org.mvel2.ParserConfiguration.hasImport(ParserConfiguration.java:142)
	at org.mvel2.ParserContext.hasImport(ParserContext.java:264)
	at org.mvel2.compiler.AbstractParser.createPropertyToken(AbstractParser.java:1213)
	at org.mvel2.compiler.AbstractParser.nextToken(AbstractParser.java:767)
	at org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:113)
	at org.mvel2.compiler.PropertyVerifier.getMethod(PropertyVerifier.java:357)
	at org.mvel2.compiler.PropertyVerifier.analyze(PropertyVerifier.java:99)
	at org.mvel2.compiler.ExpressionCompiler.verify(ExpressionCompiler.java:364)
	at org.mvel2.compiler.ExpressionCompiler._compile(ExpressionCompiler.java:258)
	at org.mvel2.compiler.ExpressionCompiler.compile(ExpressionCompiler.java:61)
	at org.drools.base.mvel.MVELCompilationUnit.compile(MVELCompilationUnit.java:365)
	at org.drools.base.mvel.MVELCompilationUnit.getCompiledExpression(MVELCompilationUnit.java:288)
	at org.drools.base.mvel.MVELConsequence.compile(MVELConsequence.java:49)
	at org.drools.rule.MVELDialectRuntimeData.onBeforeExecute(MVELDialectRuntimeData.java:103)
	at org.drools.rule.DialectRuntimeRegistry.onBeforeExecute(DialectRuntimeRegistry.java:113)
	at org.drools.common.AbstractRuleBase.addPackages(AbstractRuleBase.java:468)
	at org.drools.reteoo.ReteooRuleBase.addPackage(ReteooRuleBase.java:432)
	at org.drools.agent.PackageProvider.applyChanges(PackageProvider.java:85)
	... 5 more
13:59:44,807 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Was an error contacting http://10.16.94.36:8080/jboss-brms/org.drools.guvnor.Guvnor/package/soa/LATEST. Reponse header: {null=[HTTP/1.1 503 Service Unavailable], Date=[Wed, 02 Dec 2009 18:59:44 GMT], Content-Length=[1090], Content-Type=[text/html;charset=utf-8], Connection=[close], Server=[Apache-Coyote/1.1]}
13:59:44,807 ERROR [LogAgentEventListener] AgentName = 'insuranceconfig' Exception: Was unable to reach server.
java.io.IOException: Was unable to reach server.
	at org.drools.agent.URLScanner.hasChanged(URLScanner.java:155)
	at org.drools.agent.URLScanner.getChangeSet(URLScanner.java:119)
	at org.drools.agent.URLScanner.loadPackageChanges(URLScanner.java:96)
	at org.drools.agent.RuleAgent.checkForChanges(RuleAgent.java:410)
	at org.drools.agent.RuleAgent.refreshRuleBase(RuleAgent.java:362)
	at org.drools.agent.RuleAgent$2.run(RuleAgent.java:516)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
13:59:44,807 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Falling back to local cache.
13:59:45,809 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Was an error contacting http://10.16.94.36:8080/jboss-brms/org.drools.guvnor.Guvnor/package/soa/LATEST. Reponse header: {null=[HTTP/1.1 404 Not Found], Date=[Wed, 02 Dec 2009 18:59:44 GMT], Content-Length=[1117], Content-Type=[text/html;charset=utf-8], Server=[Apache-Coyote/1.1], X-Powered-By=[Servlet 2.4; JBoss-4.3.0.GA_CP06 (build: SVNTag=JBPAPP_4_3_0_GA_CP06 date=200907141446)/JBossWeb-2.0]}
13:59:45,810 ERROR [LogAgentEventListener] AgentName = 'insuranceconfig' Exception: Was unable to reach server.
java.io.IOException: Was unable to reach server.
	at org.drools.agent.URLScanner.hasChanged(URLScanner.java:155)
	at org.drools.agent.URLScanner.getChangeSet(URLScanner.java:119)
	at org.drools.agent.URLScanner.loadPackageChanges(URLScanner.java:96)
	at org.drools.agent.RuleAgent.checkForChanges(RuleAgent.java:410)
	at org.drools.agent.RuleAgent.refreshRuleBase(RuleAgent.java:362)
	at org.drools.agent.RuleAgent$2.run(RuleAgent.java:516)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
13:59:45,810 WARN  [LogAgentEventListener] AgentName = 'insuranceconfig' Falling back to local cache.

======================================
======================================

Comment 11 Jaroslaw Kijanowski 2009-12-02 19:05:46 UTC
Attachment: Added: server.log.zip


Comment 12 Kevin Conner 2009-12-03 09:36:49 UTC
The second is likely to be a consequence of the first.

Comment 13 Kevin Conner 2009-12-04 17:30:16 UTC
Fixed in the project, will be in the next merge.

Comment 14 Jaroslaw Kijanowski 2009-12-11 10:28:32 UTC
Verified in ER5