Bug 1010786

Summary: jmap and jstack are not working with openjdk
Product: Red Hat Enterprise Linux 7 Reporter: Liran Zelkha <lzelkha>
Component: java-1.7.0-openjdkAssignee: Severin Gehwolf <sgehwolf>
Status: CLOSED ERRATA QA Contact: OpenJDK QA <java-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.6CC: ahughes, aph, asah, bazulay, bela.pesics, dbhole, iweiss, jvanek, lakagwu, lzelkha, ophers, schaudha, sgehwolf, svanoort, tlavigne, zzambers
Target Milestone: rcKeywords: Reopened
Target Release: 7.7   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: java-1.7.0-openjdk-1.7.0.201-2.6.16.4.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1656996 1656997 (view as bug list) Environment:
Last Closed: 2019-08-06 13:02:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1657863    
Bug Blocks: 1644888, 1656996, 1656997    
Attachments:
Description Flags
Java test file none

Description Liran Zelkha 2013-09-23 05:11:22 UTC
Created attachment 801492 [details]
Java test file

Description of problem:
When running jstack and jmap, exceptions occur.

Version-Release number of selected component (if applicable):
1.7.0_25

How reproducible:


Steps to Reproduce:
1.java com.redhat.test.TestJMap (Attached to bug description)
2.At another command console, run the following command:
jstack -F <PID>
jmap -heap -F <PID>
3.

Actual results:
Exception

Expected results:
Stack trace and heap dump

Additional info:

Comment 2 Andrew John Hughes 2013-09-25 13:00:30 UTC
You don't give the exception thrown.  Is it the same as this?

Attaching to process ID 16089, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b56
Deadlock Detection:

java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc980001000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
	at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
	at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
	at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
	at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fc980001000
	at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
	at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
	... 15 more
Can't print deadlocks:Unable to deduce type of thread from address 0x00007fc980001000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)

Comment 3 Liran Zelkha 2013-10-01 06:45:20 UTC
Exception from jmap -heap:
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.tools.jmap.JMap.runTool(JMap.java:197)
	at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap
	at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:146)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:40)
	... 6 more


For jstack - it is as published above.

Comment 4 Andrew John Hughes 2013-10-01 21:26:52 UTC
Yes, jmap is as above for me too.

It's very odd.  I can't seem to replicate this on a local build, only with the system install so far.

$ jmap -heap -F 4231
Attaching to process ID 4231, please wait...
WARNING: Hotspot VM version 24.0-b56-r18ffb465e2ba does not match with SA version 24.0-b56. You may see unexpected results. 
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b56-r18ffb465e2ba

using thread-local object allocation.
Parallel GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2105540608 (2008.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 174063616 (166.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 33554432 (32.0MB)
   used     = 671200 (0.640106201171875MB)
   free     = 32883232 (31.359893798828125MB)
   2.0003318786621094% used
From Space:
   capacity = 5242880 (5.0MB)
   used     = 0 (0.0MB)
   free     = 5242880 (5.0MB)
   0.0% used
To Space:
   capacity = 5242880 (5.0MB)
   used     = 0 (0.0MB)
   free     = 5242880 (5.0MB)
   0.0% used
PS Old Generation
   capacity = 87556096 (83.5MB)
   used     = 0 (0.0MB)
   free     = 87556096 (83.5MB)
   0.0% used
PS Perm Generation
   capacity = 22020096 (21.0MB)
   used     = 2793200 (2.6638031005859375MB)
   free     = 19226896 (18.336196899414062MB)
   12.684776669456845% used

657 interned Strings occupying 42704 bytes.

Comment 5 Liran Zelkha 2013-10-08 10:14:27 UTC
Hate to nag - but any updates on this? We planned to release jmap and jstack scripts in ovirt 3.3.

Comment 7 Liran Zelkha 2013-10-08 11:22:26 UTC
Hi Lukas,

Concerning dates - I'm not sure. When is 6.5GA?
Concerning Errata - I don't have permission to the errata site. Any other place I can download this version from?

Comment 9 Andrew John Hughes 2013-10-08 15:39:29 UTC
I can't actually replicate this from a local build as I said, only with a system install of 2.4.2.  I'm afraid I won't have any more time to look at this right now as we have a security errata imminent.

Comment 10 Ingo Weiss 2013-10-09 08:37:24 UTC
This is affecting the latest RHEL 6.4, java-1.7.0-openjdk-1.7.0.25-2.3.10.4.el6_4.x86_64, and Fedora 19, java-1.7.0-openjdk-1.7.0.60-2.4.2.7.fc19.x86_64, packages as well.

$ /usr/lib/jvm/java-1.7.0-openjdk/bin/jmap -dump:format=b,file=jboss_test.hprof -F JBOSS_PID

Attaching to process ID JBOSS_PID, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b56
Dumping heap to jboss_test.hprof ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.tools.jmap.JMap.runTool(JMap.java:197)
        at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.utilities.AssertionFailure: Expecting GenCollectedHeap, G1CollectedHeap, or ParallelScavengeHeap, but got sun.jvm.hotspot.gc_interface.CollectedHeap
        at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
        at sun.jvm.hotspot.oops.ObjectHeap.collectLiveRegions(ObjectHeap.java:604)
        at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:244)
        at sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
        at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:416)
        at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
        ... 6 more

I could reproduce it on RHEL6.4 and F19, both 64-bits, when trying to take a heap dump of JBoss EAP.

I could not find any reference to this issue on KCS, Google and OpenJDK bug reporting system. The closest reference I could find was JDK-6966967.

I tried debugging, but it's way above my technical knowledge, the only think I could find is that it refer to collecting the live regions.

Comment 11 RHEL Program Management 2013-10-14 02:00:39 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 12 Bela Pesics 2014-01-03 11:55:40 UTC
looks like still the same with java-1.7.0-openjdk-1.7.0.45-2.4.3.4.el6_5.x86_64

Comment 13 Scott Dodson 2014-01-06 16:29:41 UTC
Sorry, I did not mean to unset needinfo, only adding myself to CC.

Comment 14 Lami Akagwu 2014-02-26 15:45:34 UTC
report of similar issue on OpenJDK Runtime Environment (IcedTea6 1.13.1) (rhel-3.1.13.1.el6_5-x86_64)

Is a new bugzilla required for 1.6?

Comment 15 Andrew Haley 2014-03-19 15:02:37 UTC
It's very odd that no-one mentioned the workaround.

jmap and jstack need debuginfo.  Install the -debuginfo openjdk package and it should work.

Comment 16 Lami Akagwu 2014-03-20 12:58:00 UTC
Cheers Andrew. I can confirm this worked for me.

Comment 19 Scott Dodson 2014-05-06 14:44:21 UTC
This was not addressed in 1.7.0_55? I've not been able to replicate the issue after having upgraded.

Comment 21 Deepak Bhole 2014-09-12 14:43:34 UTC
So is this issue reproducible any more?

Comment 22 Sam Van Oort 2014-11-17 23:52:20 UTC
I can confirm this still occurs with the latest versions of OpenJDK 7, and it still poses a problem.

On an Openshift Enterprise instance, I can see this when using JMap on a JBoss process, which SSH'd into the gear.

Running java -version reports version:
java version "1.7.0_71"
OpenJDK Runtime Environment (rhel-2.5.3.1.el6-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)


This is also reproducible on my dev laptop, running Fedora 20, testing JBoss   or IntelliJ IDEA process. 

RPM -qa reports the package is:
java-1.7.0-openjdk-1.7.0.71-2.5.3.0.fc20.x86_64.


In both cases, the error is the same: 

Heap Usage:
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.tools.jmap.JMap.runTool(JMap.java:197)
	at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap
	at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:146)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:40)

Unfortunately, for the OSE instances, installing the OpenJDK debuginfo package is probably not an option.

Comment 32 RHEL Program Management 2014-12-03 17:55:43 UTC
Development Management has reviewed and declined this request.
You may appeal this decision by reopening this request.

Comment 35 jiri vanek 2018-12-07 11:03:24 UTC
QAcking, but we do not have any nice test around

Comment 36 Severin Gehwolf 2018-12-07 11:07:24 UTC
Clearing need-info as this is going to get fixed anyway.

Comment 37 Severin Gehwolf 2018-12-07 11:18:19 UTC
(In reply to jiri vanek from comment #35)
> QAcking, but we do not have any nice test around

The test should be similar to the detailed NMT issue. Except for this test debuginfo *must not* be installed:

$ cat ~/HelloWait.java 
public class HelloWait {

    public static void main(String[] args) {
        System.out.println("Hello World");
        try {
            while (true) {
                Thread.sleep(200);
            }
        } catch (InterruptedException e) {
            // ignore
        }
    }

}
$ javac HelloWait.java
$ java HelloWait &
$ sleep 1 && jstack -F $(jps | grep HelloWait | cut -d' ' -f1) > jstack.out 2>&1
$ grep InvocationTargetException jstack.out && echo FAIL
$ jmap -dump:file=hellowait.hprof -F $(jps | grep HelloWait | cut -d' ' -f1) > jmap.out 2>&1
$ grep InvocationTargetException jmap.out && echo FAIL

Comment 38 jiri vanek 2018-12-07 11:30:02 UTC
I see. Thanx!

Comment 48 errata-xmlrpc 2019-08-06 13:02:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2005