Bug 780334 (SOA-2743) - Camel Gateway - loading of type converters doesn't work when esb package is deployed after long time of server idling
Summary: Camel Gateway - loading of type converters doesn't work when esb package is d...
Status: VERIFIED
Alias: SOA-2743
Product: JBoss Enterprise SOA Platform 5
Classification: JBoss
Component: 3rd Party
Version: 5.1.0.ER6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ER1
: 5.3.1
Assignee: tcunning
QA Contact: Matej Melko
URL: http://jira.jboss.org/jira/browse/SOA...
Whiteboard:
Keywords: Reopened
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-01-05 15:58 UTC by Robert Balent
Modified: 2018-03-29 21:46 UTC (History)
5 users (show)

(edit)
There is an issue with VFS caching and the deployment of non-core Camel components where the server has been running for more than thirty minutes. After the cache lifetime expires (30 minutes), if you deploy a non-core Camel component, an error is thrown. To work around this, deploy your application before starting the server or before this 30 minute limit, or raise the   boss.vfs.cache.TimedPolicyCaching.lifetime property to a higher value. (No issue is present if you deploy a non-core Camel component before booting the server, or before the cache lifetime expires.)
Clone Of:
(edit)
Last Closed: 2012-02-07 15:32:58 UTC


Attachments (Terms of Use)
Quickstart_camel_http.esb (51.14 KB, application/octet-stream)
2011-01-05 15:59 UTC, Robert Balent
no flags Details
failed_deployment.log (56.62 KB, text/plain)
2011-01-05 15:59 UTC, Robert Balent
no flags Details
successful_deployment.log (32.44 KB, text/plain)
2011-01-05 15:59 UTC, Robert Balent
no flags Details
camel-http-2.4.0.jar (57.50 KB, application/x-java-archive)
2012-03-09 15:43 UTC, Robert Balent
no flags Details
New_Quickstart_camel_http.esb (4.04 KB, application/octet-stream)
2012-03-09 15:44 UTC, Robert Balent
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker JBESB-3855 Major Closed IterableTimedCache expiry causes deployment issues with Camel non-core components 2015-04-27 12:40:33 UTC
JBoss Issue Tracker SOA-2743 Major Closed Camel Gateway - loading of type converters doesn't work when esb package is deployed after long time of server idling 2015-04-27 12:40:33 UTC

Description Robert Balent 2011-01-05 15:58:49 UTC
Workaround: Workaround Exists
Workaround Description: Use a different VFSCache implementation, or set the default lifetime to something really high.

JAVA_OPTS="$JAVA_OPTS -Djboss.vfs.cache=org.jboss.virtual.plugins.cache.IterableTimedCache -Djboss.vfs.cache.TimedPolicyCaching.lifetime=140000"

project_key: SOA

There's a problem with deployment of esb application, where the camel gateway is used .

If is the application deployed right after the server start, there's no problem. But if you wait about 30-60 minutes, error occurs, and about 100 of this error message are shown:

DEBUG [JBossPackageScanClassResolver] Cannot find class 'apache/camel/converter/stream/InputStreamCache.class' in any classloaders: [BaseClassLoader@2444569{vfszip:/home/rbalent/qa/tests/quickstarts/tests/output/lib/Quickstart_camel_http.esb/}]

To see the messages you must set log threshold to DEBUG.

To reproduce it, just start server, deploy attached Quickstart_camel_http.esb, then you will see the correct message in server log:

BEFORE**
Camel HTTP Hello World!
AFTER**

Lines "BEFORE**", and "AFTER**" are added to find the message body in server log easier.

then undeploy Quickstart_camel_http.esb and wait about 30-60 minutes. After this time, deploy the esb package again. You will see the mentioned lines in server log, and incorrect message body:
The received message will look like this:

BEFORE**
java.io.ByteArrayInputStream@6c515139AFTER**

After server restart it works again.

Quickstart_camel_http.esb, server log of successful deployment and failed deployment are attached.

Comment 1 Robert Balent 2011-01-05 15:59:34 UTC
files attached

Comment 2 Robert Balent 2011-01-05 15:59:34 UTC
Attachment: Added: Quickstart_camel_http.esb
Attachment: Added: failed_deployment.log
Attachment: Added: successful_deployment.log


Comment 3 Kevin Conner 2011-01-05 17:53:39 UTC
This sounds suspiciously like a classloader leak, will do some investigation over the next day or so.

Comment 4 tcunning 2011-08-15 13:52:14 UTC
Link: Added: This issue is related to SOA-3210


Comment 5 tcunning 2011-08-15 13:52:59 UTC
Link: Removed: This issue is related to SOA-3210 


Comment 6 tcunning 2011-09-22 20:05:08 UTC
This seems related to JBVFS-134.   Notice in the bug description in the DEBUG message that the first part of the package name of the classes are truncated.    That's happening because IterableTimedVFSCache.findContext(uri) is returning a different context after the 30 minutes (the default lifetime of the cache).      The incorrect context affects the way the class name is parsed, and we end up with the incorrectly parsed classnames which won't resolve with any classloader because they are wrong.

I am not sure whether the code creating the context is wrong, or whether there's a ClassLoader leak here which is making it behave incorrectly - if I have time I'll do some additional research into that.   There is an easy workaround here though - just use a different VFSCache implementation. 

Another thing that should be noted is that the .esb archive posted here goes against the directions on using third party Camel components on the Wiki.      The wiki article specifies that these should be installed into esb.deployer/lib rather than within the ESB archive.  

Comment 7 tcunning 2011-09-22 20:05:41 UTC
Link: Added: This issue is related to JBVFS-134


Comment 8 tcunning 2011-09-22 20:07:14 UTC
Workaround Description: Added: Use a different VFSCache implementation, or set the default lifetime to something really high.

JAVA_OPTS="$JAVA_OPTS -Djboss.vfs.cache=org.jboss.virtual.plugins.cache.IterableTimedCache -Djboss.vfs.cache.TimedPolicyCaching.lifetime=140000"

Workaround: Added: [Workaround Exists]


Comment 9 Len DiMaggio 2011-10-04 16:30:36 UTC
Oct 4, 2011 bug triage - move Fix Version to Future


Comment 10 tcunning 2012-02-07 15:32:58 UTC
Workaround exists, should follow directions for third party Camel components.   Closing as won't fix.

Comment 11 Ken Johnson 2012-02-28 20:46:24 UTC
Two questions:
1.  Does this occur only after undeploy or even on initial deployment after server idle?
2. What are the (side) effects of the workaround approaches for other components/deployments?

Comment 12 Robert Balent 2012-02-29 15:37:09 UTC
1. Yes, it occurs for initial deployment too. Just run server, wait jboss.vfs.cache.TimedPolicyCaching.lifetime and hotdeploy application.

Comment 13 Robert Balent 2012-03-09 15:41:57 UTC
I'm not sure if "VFS cache change" workaround works correctly. Can somebody try to reproduce it with me?

Reproducing the original reported bug:

1. Copy camel-http-2.4.0.jar to deployers/esb.deployer/lib as Tom recommended.
2. Set logging same as in all configuration to see output.
3. Run server with -Djboss.vfs.cache.TimedPolicyCaching.lifetime=180 so you can easily wait for cache to expire.
4. Wait 3 minutes for cache expiration
5. Copy New_Quickstart_camel_http.esb to deploy folder. You will see in server log: 

BEFORE**
java.io.ByteArrayInputStream@6c515139AFTER**

6. Correct output is:

BEFORE**
Camel HTTP Hello World!
AFTER**


Try same with the 3 minutes cache expiration and append -Djboss.vfs.cache=org.jboss.virtual.plugins.cache.IterableTimedCache

You will probably see same incorrect output after 3 minutes. If you deploy esb application before 3 minutes expires, You will see the correct output.


Will attach camel-http-2.4.0.jar and New_Quickstart_camel_http.esb files.

Comment 14 Robert Balent 2012-03-09 15:43:14 UTC
Created attachment 568948 [details]
camel-http-2.4.0.jar

Comment 15 Robert Balent 2012-03-09 15:44:49 UTC
Created attachment 568950 [details]
New_Quickstart_camel_http.esb

Comment 16 Robert Balent 2012-03-09 17:22:47 UTC
I've created JBPAPP-8411 issue in jira.

Comment 18 Suz 2012-06-12 04:14:30 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
If you deploy an ESB application with a non-core Camel gateway, it will stop working after about thirty minutes. After this time, an error is thrown. To work around this, deploy your application before this limit or raise the  jboss.vfs.cache.TimedPolicyCaching.lifetime property to a higher value.

Comment 19 tcunning 2012-06-12 13:34:42 UTC
That description isn't really accurate.      It should read something like :

There is an issue with VFS caching and the deployment of non-core Camel components after a server has been booted/started and running for more than thirty minutes.   No issue is present if you deploy a non-core Camel component before boot.     After the thirty minutes, if you deploy a non-core Camel component, an error is thrown. To work around this, deploy your application before this 30 minute limit or raise the  jboss.vfs.cache.TimedPolicyCaching.lifetime property to a higher value.

Comment 20 tcunning 2012-06-12 13:44:57 UTC
Take 2 - I think this is even a little more clear :

There is an issue with VFS caching and the deployment of non-core Camel components in the a situation where the server has been booted/started and running for more than thirty minutes.    No issue is present if you deploy a non-core Camel component before booting the server, or before the cache lifetime expires (30 minutes).     After the cache lifetime expires (30 minutes), if you deploy a non-core Camel component, an error is thrown.     To work around this, deploy your application before starting the server or before this 30 minute limit or raise the  jboss.vfs.cache.TimedPolicyCaching.lifetime property to a higher value.

Comment 21 Suz 2012-06-12 23:26:21 UTC
Thanks very much for that Tom. Have altered the note.

Comment 22 Suz 2012-06-12 23:26:21 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-If you deploy an ESB application with a non-core Camel gateway, it will stop working after about thirty minutes. After this time, an error is thrown. To work around this, deploy your application before this limit or raise the  jboss.vfs.cache.TimedPolicyCaching.lifetime property to a higher value.+There is an issue with VFS caching and the deployment of non-core Camel components where the server has been running for more than thirty minutes. After the cache lifetime expires (30 minutes), if you deploy a non-core Camel component, an error is thrown. To work around this, deploy your application before starting the server or before this 30 minute limit, or raise the   boss.vfs.cache.TimedPolicyCaching.lifetime property to a higher value. (No issue is present if you deploy a non-core Camel component before booting the server, or before the cache lifetime expires.)

Comment 23 Robert Balent 2012-06-18 16:29:59 UTC
Hello, I was just looking a little at this problem today and maybe I found the solution. In class:

org.jboss.soa.esb.listeners.gateway.camel.JBossPackageScanClassResolver$MatchingClassVisitor

method: "visit(VirtualFile file)"

it could look like this. I compiled the class, tested and can't reproduce the problem anymore. Can somebody review my code?


public void visit(VirtualFile file) {
    if (file.getName().endsWith(".class")) {
        String fqn;
        try {
            fqn = file.toURL().toString();
        } catch (Exception ex) {
            fqn = file.getName();
        }
        String qn;

        if (fqn.indexOf("jar/") != -1) {
            qn = fqn.substring(fqn.indexOf("jar/") + 4);
        } else {
            qn = fqn;
        }

        addIfMatching(filter, qn, classes);
    }
}

Comment 24 tcunning 2012-10-12 19:59:23 UTC
Reviewed and tested Robert's patch - I don't see any errors with it using the attached quickstart.    Committed to 4_11_CP branch.


Note You need to log in before you can comment on or make changes to this bug.