Bug 780551 (SOA-2985)

Summary: VDB deployment leaves with too many open files
Product: [JBoss] JBoss Enterprise SOA Platform 5 Reporter: Ramesh Reddy <rareddy>
Component: EDSAssignee: Van Halbert <vhalbert>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.1.0 GACC: rareddy, vhalbert
Target Milestone: ---   
Target Release: 5.2.0 ER1   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/SOA-2985
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-10-26 14:56:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Ramesh Reddy 2011-03-23 14:22:04 UTC
Steps to Reproduce: Use the admin api to deploy/undeploy a large number of times, for example in adminshell

for (i = 0; i < 300; i++) {
  deployVDB("foo.vdb");
  undeployVDB("foo", 1);
}

There will be well over 300 new open sockets at the end of the run.  Prior to the previous commit on this issue in 7.4, at most 64 of them can be buffer files that will get auto cleaned up, but the rest of them will be remoting serversockets that do not seem to be limited by the suggestion in JBAS-6665.
Workaround: Workaround Exists
Workaround Description: Use an alternative deployment mechanism, such as a local file drop into the deploy directory, or through RHQ.
project_key: SOA

After deploying a VDB using Admn API or AdminShell a temporary file handle is open and does not seem to be closed. After deploying the VDB run

lsof

Comment 1 Steven Hawkins 2011-03-23 14:22:05 UTC
Link: Added: This issue Cloned from TEIID-1511


Comment 2 Steven Hawkins 2011-03-23 15:36:28 UTC
Release Notes Docs Status: Added: Not Yet Documented
Release Notes Text: Added: VDB deployments through the AdminApi cause the application server to leak serversockets and listening threads, which can cause failures from too many open file handles.
Security: Added: Public


Comment 3 Steven Hawkins 2011-03-23 16:47:17 UTC
Note that the underlying issue appears to be with Teiid's usage of a local ProfileService instance.  We are obtaining a ProfileService by injecting it in our deployer with:

<bean name="RuntimeEngineDeployer" class="org.teiid.jboss.deployers.RuntimeEngineDeployer">
        <property name="jndiName">teiid/engine-deployer</property>
        <property name="profileService"><inject bean="ProfileService"/></property>
        ...

Any file deployment through that ProfileService then leaks a JBoss Remoting server socket.  Someone on the AS team could confirm if we should not be doing file deployments this way or if this is an issue with local ProfileService instances. 

Comment 4 Steven Hawkins 2011-03-23 16:47:17 UTC
Release Notes Text: Removed: VDB deployments through the AdminApi cause the application server to leak serversockets and listening threads, which can cause failures from too many open file handles. Added: VDB deployments through the AdminApi cause the application server to leak server sockets and listening threads, which can cause failures from too many open file handles.
Steps to Reproduce: Removed: Use the admin api to deploy/undeploy a large number of times, for example in adminshell

for (i = 0; i < 200; i++) {
  deployVDB("foo.vdb");
  undeployVDB("foo", 1);
}

There will be well over 300 new open sockets at the end of the run.  Prior to the previous commit on this issue in 7.4, at most 64 of them can be buffer files that will get auto cleaned up, but the rest of them will be remoting serversockets that do not seem to be limited by the suggestion in JBAS-6665. Added: Use the admin api to deploy/undeploy a large number of times, for example in adminshell

for (i = 0; i < 300; i++) {
  deployVDB("foo.vdb");
  undeployVDB("foo", 1);
}

There will be well over 300 new open sockets at the end of the run.  Prior to the previous commit on this issue in 7.4, at most 64 of them can be buffer files that will get auto cleaned up, but the rest of them will be remoting serversockets that do not seem to be limited by the suggestion in JBAS-6665.


Comment 5 Van Halbert 2011-04-17 01:57:54 UTC
Link: Added: This issue depends JBREM-1282


Comment 6 Van Halbert 2011-04-25 14:31:25 UTC
In order to resolve this problem, it will require a patch to EAP.

Comment 8 Van Halbert 2011-04-26 13:41:02 UTC
Link: Added: This issue depends JBREM-1284


Comment 10 Van Halbert 2011-06-02 20:04:03 UTC
The patch will be applied to EAP 5.1.1 for the SOA 5.2 release.

Comment 11 David Le Sage 2011-08-31 06:09:33 UTC
Release Notes Docs Status: Removed: Not Yet Documented Added: Documented as Resolved Issue
Writer: Added: dlesage
Release Notes Text: Removed: VDB deployments through the AdminApi cause the application server to leak server sockets and listening threads, which can cause failures from too many open file handles. Added: https://issues.jboss.org/browse/SOA-2985

VDB deployments through the AdminApi caused the application server to leak server sockets and listening threads, which could cause failures due to  too many  file handles being opened. A patch has been applied to fix this issue. As a result, server sockets and listening threads are no longer leaked so users will no longer encounter these failures.
Forum Reference: Removed: http://community.jboss.org/thread/163742 Added: http://community.jboss.org/thread/163742


Comment 12 Van Halbert 2011-10-11 15:34:36 UTC
Forum Reference: Removed: http://community.jboss.org/thread/163742 Added: http://community.jboss.org/thread/163742


Comment 13 Filip Nguyen 2011-10-26 14:56:35 UTC
Verified on 5.2 ER5 and 5.2 ER1. 

On 5.1 GA before: 8178 files in system. After 50 deploys/undeploys 8293 files.

In ER1 the numbers were 8207 before, 8191 after.
In ER5 8182 before, 8197 after.