Bug 772241

Summary: "java.net.SocketException: Too many open files" after refreshing Process Overview in the jBPM Console
Product: [JBoss] JBoss Enterprise BRMS Platform 5 Reporter: Jiri Locker <jlocker>
Component: jBPM ConsoleAssignee: Mark Proctor <mproctor>
Status: VERIFIED --- QA Contact: Lukáš Petrovický <lpetrovi>
Severity: high Docs Contact:
Priority: unspecified    
Version: BRMS 5.3.0.GACC: atangrin, kverlaen, lpetrovi, mproctor, rzhang, trikkola
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Release Note
Doc Text:
A bug in the Business Central console presented when the process overview was refreshed, which resulted in a large number of open file descriptors per process. The number of files could quickly reach the maximum number of allowed open file descriptors per process. The issue has been resolved by ensuring the open files are disposed of after the communication has closed.
 Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
16proc_repo.xml
none
openfiles.txt none

Description Jiri Locker 2012-01-06 14:56:08 UTC 
Description of problem:
For each process that is exposed by Guvnor repository, a large number of instances of jboss-brms.war/WEB-INF/classes/asseteditors.xml file is opened when the console fetches the list of processes. As the number of processes increases the number of open files may exceed the maximum number of open file descriptors per process. On my system the limit is 1024 files and I hit it with around 16 processes (which is not very high number). When the limit is reached, teh application server cannot open more files and stops working correctly. Sure, the limit can be increased, but this really doesn't sound like scalability.

Version-Release number of selected component (if applicable):
ER1

How reproducible:
depends on the maximum number of open file descriptors per process limit

Steps to Reproduce:
1. import the attached repository into Guvnor (it contains 16 processes)
2. build defaultPackage
3. start watching server log
4. open jBPM Console and go to Process Overview, if no processes show up there's a configuration problem (check other bugzillas)
5. if you can see java.net.SocketException: Too many open files in the log, you're done
6. hit Refresh and go to 5.
  
Actual results:


Expected results:
jBPM Console should scale with respect to the number of processes it manages. The number of times the asseteditors.xml file is open each time jBPM Console fetches processes should be optimized to a reasonable amount (1 per process or better, 1 for all)

Additional info:
on linux systems these commands can help:

> ulimit -n
to see the maximum open file descriptors

> lsof -p `jps -l|grep org.jboss.Main|sed "s/ .*//"` | wc -l
to see the number of open file descriptors by a JBoss AS process
Comment 1 Jiri Locker 2012-01-06 15:04:28 UTC 
Created attachment 551166 [details]
16proc_repo.xml
Comment 4 Kris Verlaenen 2012-02-28 23:54:52 UTC 
I actually had the same issue.  The Java process that is executing the application server is trying to open more than 1k file handles in that case.  Looking through the list of open files, it didn't seem however that this was a bug, there are just a large number of jars etc. that are loaded if you deploy all applications on the same server.

When looking at the list of open files, do you have any indication that there might be a leak somewhere (where we are opening file handles but they keep growing)?
Comment 5 Jiri Locker 2012-03-05 14:17:17 UTC 
Created attachment 567628 [details]
openfiles.txt

Yes, in this attachment, there is ~812 open file descriptors out of which 570 is for jboss-brms.war/WEB-INF/classes/asseteditors.xml, which is quite a big number.

I don't consider this a leak, since all the file descriptors for asseteditors.xml are closed eventually. The only trouble is that refreshing the process overview creates a large burst of file descriptors to be open, which may hit the system limit.

On the other hand, I've been forced to increase the limit on my system to 9k to get JBDS5 running recently and, since then, I haven't seen this issue.
Comment 7 Marco Rietveld 2012-04-08 10:49:03 UTC 
I've just merged Mauricio's pull request concerning this problem to master and 5.2.x. 

You'll have to ask Mauricio exactly what the impact is on this problem. :) 

See here for the pull request: https://github.com/droolsjbpm/jbpm/pull/73
Comment 8 Toni Rikkola 2012-04-17 06:44:21 UTC 
Hi, Tihomir. 
Do you know more about this issue?
Comment 9 Ryan Zhang 2012-04-23 07:37:23 UTC 
Update status to ON_QA. Please verify them against ER6.
Comment 10 Ryan Zhang 2012-04-23 07:55:22 UTC 
Confirmed that not picked in ER6. Change status back.
Comment 13 lcarlon 2012-06-08 03:42:34 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Refreshing the process overview in the Business Central Console results in a large number of files and can reach the maximum number of open file descriptors per process. The current workaround is to increase the maximum number of open file descriptors per process.
Comment 14 Zuzana Krejčová 2012-09-13 11:51:29 UTC 
VERIFIED
And just for 'fun', some quick numbers:

I get around 500 open file descriptors with 16 human task processes, each with a process image and 2 task forms.
With empty repo, the count is down to around 430. So a (simple) human task process might take about 4.5 open file descriptors.
If we start at 430 for an empty repo and leave the max count at 1024, we can have about 130 (simple) human task processes, not counting anything else in Guvnor or any other app running on the server that might influence this. Definitely better than 16.