Bug 1455694 - Full GC hangs when using java-1.7.0-openjdk-1.7.0.141 and AggressiveHeap
Full GC hangs when using java-1.7.0-openjdk-1.7.0.141 and AggressiveHeap
Status: MODIFIED
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: java-1.7.0-openjdk (Show other bugs)
6.10
Unspecified Unspecified
urgent Severity urgent
: rc
: ---
Assigned To: Christine Flood
BaseOS QE - Apps
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-05-25 16:35 EDT by Robert Bost
Modified: 2017-08-21 11:53 EDT (History)
4 users (show)

See Also:
Fixed In Version: java-1.7.0-openjdk-1.7.0.151-2.6.11.1.el6
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Icedtea Bugzilla 3410 None None None 2017-06-27 21:33 EDT
Red Hat Knowledge Base (Solution) 3055841 None None None 2017-05-25 16:40 EDT

  None (edit)
Description Robert Bost 2017-05-25 16:35:17 EDT
Description of problem:


Version-Release number of selected component (if applicable):
tomcat6-6.0.24-105.el6_8.noarch
java-1.7.0-openjdk-1.7.0.141-2.6.10.1.el6_9.x86_64

How reproducible: Always


Steps to Reproduce:
1. yum install tomcat6 java-1.7.0-openjdk
2. echo JAVA_OPTS=\"-XX:+AggressiveHeap\" >> /etc/tomcat6/tomcat6.conf
3. service tomcat6 start 

Actual results:
Tomcat hangs on start and gc.log shows a Full GC occurring indefinitely (customer has Full GC showing for more than a day without completely and showing the "Times" output).

  0.788: [Full GC  <never finishes writing out rest of line>


Multiple Workarounds:

 - Use java-1.7.0-openjdk-1.7.0.121
 - Start tomcat without the -XX:+AggressiveHeap option
 - Keep AggressiveHeap option but include -XX:+DisableExplicitGC

Additional info:
Thread dump shows the GC occurring and not returning.

# jstack -F 5142
Attaching to process ID 5142, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.141-b02
Deadlock Detection:

No deadlocks found.

Thread 5158: (state = BLOCKED)
 - java.lang.Runtime.gc() @bci=0 (Interpreted frame)
 - java.lang.System.gc() @bci=3, line=984 (Interpreted frame)
 - sun.misc.GC$Daemon.run() @bci=38, line=109 (Interpreted frame)


Thread 5153: (state = BLOCKED)


Thread 5152: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)


Thread 5151: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Interpreted frame)


Thread 5145: (state = BLOCKED)
 - java.lang.Class.forName0(java.lang.String, boolean, java.lang.ClassLoader, java.lang.Class) @bci=0 (Interpreted frame)
 - java.lang.Class.forName(java.lang.String) @bci=11, line=195 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.getModelerSource(java.lang.String) @bci=37, line=921 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.load(java.lang.String, java.lang.Object, java.lang.String) @bci=208, line=753 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.loadDescriptors(java.lang.String, java.lang.Object, java.lang.String) @bci=4, line=866 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.findManagedBean(java.lang.Object, java.lang.Class, java.lang.String) @bci=147, line=651 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.findManagedBean(java.lang.Class, java.lang.String) @bci=4, line=963 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.registerComponent(java.lang.Object, javax.management.ObjectName, java.lang.String) @bci=84, line=794 (Interpreted frame)
 - org.apache.catalina.core.StandardServer.initialize() @bci=140, line=788 (Interpreted frame)
 - org.apache.catalina.startup.Catalina.load() @bci=328, line=540 (Interpreted frame)
 - org.apache.catalina.startup.Catalina.load(java.lang.String[]) @bci=9, line=560 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=57 (Interpreted frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.load(java.lang.String[]) @bci=109, line=261 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=131, line=413 (Interpreted frame)
Comment 2 Aaron Ogburn 2017-05-25 16:56:38 EDT
I'd think -XX:+DisableExplicitGC just avoids the issue for a time because it avoids the early forced System.GC.  Full GC might hang later when it occurs with AggressiveHeap set.  Are all Full GCs with AggressiveHeap now hanging (regardless of tomcat6 or some other java app being run)?
Comment 3 Robert Bost 2017-05-25 16:58:26 EDT
I've simplified this down to just:

# cat Test.java
public class Test {
  public static void main(String[] args){ 
    System.gc();
  }
}

# java -version
java version "1.7.0_141"
OpenJDK Runtime Environment (rhel-2.6.10.1.el6_9-x86_64 u141-b02)
OpenJDK 64-Bit Server VM (build 24.141-b02, mixed mode)
# javac Test.java
# java -XX:+AggressiveHeap Test
<hangs>

Now we can rule out tomcat6.
Comment 4 Deepak Bhole 2017-05-26 11:41:23 EDT
Thank you for the standalone test case.

Assigning to Christine to investigate.
Comment 5 Christine Flood 2017-06-01 13:38:11 EDT
I can give you a workaround for now, and I'm working on a patch.

The issue is that -XX:+AggressiveHeap is somehow asking to run parallel gc but setting parallelgcthreads to 0.  You can see this if you run with -XX:+PrintFlagsFinal.   Nothing happens and everyone hangs.

The workaround is to run:

java -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=n -XX:+AggressiveHeap Test 

which works just fine.
Comment 6 Christine Flood 2017-06-08 14:24:46 EDT
The bug was a regression that has been addressed upstream. (Thanks to Andrew Hughes for point thisout).

https://bugs.openjdk.java.net/browse/JDK-8179084
http://cr.openjdk.java.net/~kbarrett/8179084/hotspot.00/

This patch should be backported to JDK7.
Comment 7 Andrew John Hughes 2017-08-21 11:52:24 EDT
Fixed in IcedTea 2.6.11: http://bitly.com/it20611

Note You need to log in before you can comment on or make changes to this bug.