Bug 1455694

Summary: Full GC hangs when using java-1.7.0-openjdk-1.7.0.141 and AggressiveHeap
Product: Red Hat Enterprise Linux 6 Reporter: Robert Bost <rbost>
Component: java-1.7.0-openjdkAssignee: Christine Flood <chf>
Status: CLOSED CURRENTRELEASE QA Contact: BaseOS QE - Apps <qe-baseos-apps>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.10CC: ahughes, aogburn, dbhole, jvanek
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: java-1.7.0-openjdk-1.7.0.151-2.6.11.1.el6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-05 20:19:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1461138, 1503147    

Description Robert Bost 2017-05-25 20:35:17 UTC
Description of problem:


Version-Release number of selected component (if applicable):
tomcat6-6.0.24-105.el6_8.noarch
java-1.7.0-openjdk-1.7.0.141-2.6.10.1.el6_9.x86_64

How reproducible: Always


Steps to Reproduce:
1. yum install tomcat6 java-1.7.0-openjdk
2. echo JAVA_OPTS=\"-XX:+AggressiveHeap\" >> /etc/tomcat6/tomcat6.conf
3. service tomcat6 start 

Actual results:
Tomcat hangs on start and gc.log shows a Full GC occurring indefinitely (customer has Full GC showing for more than a day without completely and showing the "Times" output).

  0.788: [Full GC  <never finishes writing out rest of line>


Multiple Workarounds:

 - Use java-1.7.0-openjdk-1.7.0.121
 - Start tomcat without the -XX:+AggressiveHeap option
 - Keep AggressiveHeap option but include -XX:+DisableExplicitGC

Additional info:
Thread dump shows the GC occurring and not returning.

# jstack -F 5142
Attaching to process ID 5142, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.141-b02
Deadlock Detection:

No deadlocks found.

Thread 5158: (state = BLOCKED)
 - java.lang.Runtime.gc() @bci=0 (Interpreted frame)
 - java.lang.System.gc() @bci=3, line=984 (Interpreted frame)
 - sun.misc.GC$Daemon.run() @bci=38, line=109 (Interpreted frame)


Thread 5153: (state = BLOCKED)


Thread 5152: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)


Thread 5151: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Interpreted frame)


Thread 5145: (state = BLOCKED)
 - java.lang.Class.forName0(java.lang.String, boolean, java.lang.ClassLoader, java.lang.Class) @bci=0 (Interpreted frame)
 - java.lang.Class.forName(java.lang.String) @bci=11, line=195 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.getModelerSource(java.lang.String) @bci=37, line=921 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.load(java.lang.String, java.lang.Object, java.lang.String) @bci=208, line=753 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.loadDescriptors(java.lang.String, java.lang.Object, java.lang.String) @bci=4, line=866 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.findManagedBean(java.lang.Object, java.lang.Class, java.lang.String) @bci=147, line=651 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.findManagedBean(java.lang.Class, java.lang.String) @bci=4, line=963 (Interpreted frame)
 - org.apache.tomcat.util.modeler.Registry.registerComponent(java.lang.Object, javax.management.ObjectName, java.lang.String) @bci=84, line=794 (Interpreted frame)
 - org.apache.catalina.core.StandardServer.initialize() @bci=140, line=788 (Interpreted frame)
 - org.apache.catalina.startup.Catalina.load() @bci=328, line=540 (Interpreted frame)
 - org.apache.catalina.startup.Catalina.load(java.lang.String[]) @bci=9, line=560 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=57 (Interpreted frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.load(java.lang.String[]) @bci=109, line=261 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=131, line=413 (Interpreted frame)

Comment 2 Aaron Ogburn 2017-05-25 20:56:38 UTC
I'd think -XX:+DisableExplicitGC just avoids the issue for a time because it avoids the early forced System.GC.  Full GC might hang later when it occurs with AggressiveHeap set.  Are all Full GCs with AggressiveHeap now hanging (regardless of tomcat6 or some other java app being run)?

Comment 3 Robert Bost 2017-05-25 20:58:26 UTC
I've simplified this down to just:

# cat Test.java
public class Test {
  public static void main(String[] args){ 
    System.gc();
  }
}

# java -version
java version "1.7.0_141"
OpenJDK Runtime Environment (rhel-2.6.10.1.el6_9-x86_64 u141-b02)
OpenJDK 64-Bit Server VM (build 24.141-b02, mixed mode)
# javac Test.java
# java -XX:+AggressiveHeap Test
<hangs>

Now we can rule out tomcat6.

Comment 4 Deepak Bhole 2017-05-26 15:41:23 UTC
Thank you for the standalone test case.

Assigning to Christine to investigate.

Comment 5 Christine Flood 2017-06-01 17:38:11 UTC
I can give you a workaround for now, and I'm working on a patch.

The issue is that -XX:+AggressiveHeap is somehow asking to run parallel gc but setting parallelgcthreads to 0.  You can see this if you run with -XX:+PrintFlagsFinal.   Nothing happens and everyone hangs.

The workaround is to run:

java -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=n -XX:+AggressiveHeap Test 

which works just fine.

Comment 6 Christine Flood 2017-06-08 18:24:46 UTC
The bug was a regression that has been addressed upstream. (Thanks to Andrew Hughes for point thisout).

https://bugs.openjdk.java.net/browse/JDK-8179084
http://cr.openjdk.java.net/~kbarrett/8179084/hotspot.00/

This patch should be backported to JDK7.

Comment 7 Andrew John Hughes 2017-08-21 15:52:24 UTC
Fixed in IcedTea 2.6.11: http://bitly.com/it20611

Comment 12 Deepak Bhole 2018-01-05 20:19:30 UTC
Fixed as of java-1.7.0-openjdk-1.7.0.151-2.6.11.1.el6.