Hide Forgot
Date of First Response: 2009-06-23 10:25:48 project_key: SOA See linked issue.
Link: Added: This issue depends JBWS-2678
Please note that this issue breaks EBWS in SOA-P: 2009-06-12 14:15:42,140 ERROR [org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS] SOAP request exception org.jboss.ws.WSException: Cannot open: file:/home/mvecera/work/soa/43GACP01/jboss-as/server/oracle/data/wsdl/Quickstart_publish_as_webservice.esb/Quickstart_publish_as_webserv at org.jboss.ws.core.jaxws.handler.MessageContextJAXWS.setOperationMetaData(MessageContextJAXWS.java:143) at org.jboss.ws.core.server.ServiceEndpointInvoker.invoke(ServiceEndpointInvoker.java:177) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:424) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:287) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:197) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:123) at org.jboss.wsf.stack.jbws.EndpointServlet.service(EndpointServlet.java:84) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:619) Side effects: 2009-06-12 14:15:42,144 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310) at java.lang.Thread.run(Thread.java:619)
Instructions to reproduce by Paul Robinson (Arjuna) - they were able to reproduce the issue on Ubuntu where ulimit -n is 1024. Here's the steps: 1. cd to the publish_as_webservice quickstart 2. replace ./src/org/jboss/soa/esb/samples/quickstart/publishAsWebservice/test/SOAPTest.java with attached file. I have simply added a loop so as messages get sent one after another. 3. deploy the quickstart and start the server. 4. run the client for the quickstart. You should observe messages been sent one after another on both the client and the server's console. 5. Once you are happy that it's working for one client, open up 7 more terminals ad fire off the same client. 6. Leave the test running for 5-10 minutes. If after 10 minutes you habve not observed the bug, restart the server and start again. You will most likely get the error on the first try (if it exists), but I've had a run of 3 where it did not apear. Hence my origional statement that it was no longer happening in SOA-P 4.3.0.CP1.
Attachment: Added: SOAPTest.java
Changed priority to major, because the bad code invocation can be avoided on the client side - SOAPTest.java: < "http://127.0.0.1:8080/Quickstart_publish_as_webservice/ESBServiceSample/HelloWorldPubService?wsdl") > "http://127.0.0.1:8080/Quickstart_publish_as_webservice/ESBServiceSample/HelloWorldPubService")
Martin, It seems your suggestion to change the above URL does not have any effect on my system. Could you try running the following command "while true; do /usr/sbin/lsof | grep wsdl | wc -l;sleep 2;done" prior to follow Paul's instructions. When load is applied, I observer the amount of open WSDL files will increase until the server fails. Can you confirm?
You are right, the WSDL still being requested but not so often. The opened files are also being closed. Some of them immediately, some of them after a "timeout". I was not able to reach the point where there was more than 50 opened WSDL files. I suggest you to limit the number of messages or clients to get useful results. The issue is fixed in a newer WS version so it could be in 4.3.CP02 hopefully.
Ok, I can do that. I have tested with 4 concurrent clients and it seems to be stable for more than 20 minutes. 5 clients fail after a few minutes. I imagine these numbers are hardware dependant.
I can't edit my comments so I have to add a correction to to the above comment here. When I said "5 clients fail after a few minutes." I meant "Running the test with 5 clients applying load, I get the 'Too many open files' exceptions on the the server after a few minutes.". When I said "I imagine these numbers are hardware dependant.", I meant "I recognising that the amount of concurrent clients the system can handle is dependant on your test hardware".
To be retested with 4.3 CP05
Verified in SOA 4.3.5.ER2