Hide Forgot
Description of problem: For each process that is exposed by Guvnor repository, a large number of instances of jboss-brms.war/WEB-INF/classes/asseteditors.xml file is opened when the console fetches the list of processes. As the number of processes increases the number of open files may exceed the maximum number of open file descriptors per process. On my system the limit is 1024 files and I hit it with around 16 processes (which is not very high number). When the limit is reached, teh application server cannot open more files and stops working correctly. Sure, the limit can be increased, but this really doesn't sound like scalability. Version-Release number of selected component (if applicable): ER1 How reproducible: depends on the maximum number of open file descriptors per process limit Steps to Reproduce: 1. import the attached repository into Guvnor (it contains 16 processes) 2. build defaultPackage 3. start watching server log 4. open jBPM Console and go to Process Overview, if no processes show up there's a configuration problem (check other bugzillas) 5. if you can see java.net.SocketException: Too many open files in the log, you're done 6. hit Refresh and go to 5. Actual results: Expected results: jBPM Console should scale with respect to the number of processes it manages. The number of times the asseteditors.xml file is open each time jBPM Console fetches processes should be optimized to a reasonable amount (1 per process or better, 1 for all) Additional info: on linux systems these commands can help: > ulimit -n to see the maximum open file descriptors > lsof -p `jps -l|grep org.jboss.Main|sed "s/ .*//"` | wc -l to see the number of open file descriptors by a JBoss AS process
Created attachment 551166 [details] 16proc_repo.xml
I actually had the same issue. The Java process that is executing the application server is trying to open more than 1k file handles in that case. Looking through the list of open files, it didn't seem however that this was a bug, there are just a large number of jars etc. that are loaded if you deploy all applications on the same server. When looking at the list of open files, do you have any indication that there might be a leak somewhere (where we are opening file handles but they keep growing)?
Created attachment 567628 [details] openfiles.txt Yes, in this attachment, there is ~812 open file descriptors out of which 570 is for jboss-brms.war/WEB-INF/classes/asseteditors.xml, which is quite a big number. I don't consider this a leak, since all the file descriptors for asseteditors.xml are closed eventually. The only trouble is that refreshing the process overview creates a large burst of file descriptors to be open, which may hit the system limit. On the other hand, I've been forced to increase the limit on my system to 9k to get JBDS5 running recently and, since then, I haven't seen this issue.
I've just merged Mauricio's pull request concerning this problem to master and 5.2.x. You'll have to ask Mauricio exactly what the impact is on this problem. :) See here for the pull request: https://github.com/droolsjbpm/jbpm/pull/73
Hi, Tihomir. Do you know more about this issue?
Update status to ON_QA. Please verify them against ER6.
Confirmed that not picked in ER6. Change status back.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Refreshing the process overview in the Business Central Console results in a large number of files and can reach the maximum number of open file descriptors per process. The current workaround is to increase the maximum number of open file descriptors per process.
VERIFIED And just for 'fun', some quick numbers: I get around 500 open file descriptors with 16 human task processes, each with a process image and 2 task forms. With empty repo, the count is down to around 430. So a (simple) human task process might take about 4.5 open file descriptors. If we start at 430 for an empty repo and leave the max count at 1024, we can have about 130 (simple) human task processes, not counting anything else in Guvnor or any other app running on the server that might influence this. Definitely better than 16.