Created attachment 1028055 [details] Direct memory report Problem: "java.lang.OutOfMemoryError: Direct buffer memory" was thrown during connector performance testing, see attaches 'exception.txt'. The problem was hidden by [1] which was fixed in 6.4.1.CP.CR2, therefore this is not a regression. Performance connectors testing * EAP is started with configuration standalone-ha.xml (see attachment) * please notice http connectors especialy, nio2 is set * EAP is requested from 4 other nodes by commands like (request are no-keep alive) ab -n 10000 -c 256 -Z ECDHE-RSA-DES-CBC3-SHA <https://...> This command is executed ~ 60 times from each node. * Exception was thrown after ~ 50minutes in described scenario * I tried to increate direct buffer memory to 8G ( -XX:MaxDirectMemorySize=8G) and exception was thrown later only (after ~ 150 minutes). * I monitored direct buffer memory during scenario with increased memory by [2]. For details see attached 'direct-buffer-memory-monitor.log' and 'server.log'. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1200276 [2] https://blogs.oracle.com/alanb/resource/MonBuffers.java [3] EAP JAVA_OPTS -XX:+UseCompressedOops -verbose:gc -Xloggc:"/tmp/hudson/jboss-eap/standalone/log/gc.log" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -server -Xms5g -Xmx5g -XX:MaxPermSize=512m -Xss1024k -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djgroups.udp.ip_ttl=1 -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dsf.pid=WvphqJBs824awiZM
Created attachment 1028056 [details] server.log
Created attachment 1028057 [details] server.log
Created attachment 1028058 [details] exception.txt
Created attachment 1028061 [details] standalone-ha.xml
From direct-buffer-memory-monitor.log these appear to be the results of GC: 09:54:01 PM 49720 561833612 561833612 11:09:01 PM 14595 178052887 178052887 12:28:01 AM 17866 216067099 216067099 01:48:01 AM 17717 211701469 211701469 03:07:01 AM 13474 170090946 170090946 04:23:01 AM 17971 219654123 219654123 I see no problem there.
Here, the SecureNioChannel does not need to use direct buffers, so it would be good to change that to regular ones and use that to determine there's a leak [I doubt it]. It looks like the JVM would have a much harder time GCing those direct buffers. Example of the problem: http://stackoverflow.com/questions/1854398/how-to-garbage-collect-a-direct-buffer-java
> x60 times? does this mean 4x60x256 concurrent calls or 4x256 concurrent calls and 60 repetitions. 4x256 concurrent calls and 60 repetitions. > Jan can you share JDK/OS pair? Oracle JDK7 / RHEL7 > So can you hack a bit SecureNioChannel and replace all ByteBuffer.allocateDirect ... If anyone provide patch to me I will execute the test over it. Testing time depends on load in lab.
I did testing with provided patch and exception did not appear anymore. I didn't notice performance drop - result of patched and not patched server are same basically.
r2617 in web.
great, thanks Remy.
Verified in 6.4.3.CR1. PASSED. Thank you.
Retroactively bulk-closing issues from released EAP 6.4 cummulative patches.