Bug 1520879

Summary: Detailed Native Memory Tracking does not show stack traces with debug-info installed
Product: [Fedora] Fedora Reporter: Severin Gehwolf <sgehwolf>
Component: java-1.8.0-openjdkAssignee: jiri vanek <jvanek>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 27CC: dbhole, jerboaa, jvanek, msrb, mvala, omajid, zgu
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-28 09:34:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
detailed memory tracking of HelloWait with fixed JDK build. none

Description Severin Gehwolf 2017-12-05 11:30:12 UTC
Description of problem:
When running "jcmd VM.native_memory detail" on a java program which has the JVM option "-XX:NativeMemoryTracking=detail", the detail report does not contain stack traces. Only raw addresses are shown.

Version-Release number of selected component (if applicable):
java-1.8.0-openjdk-devel-1.8.0.151-1.b12.fc27.x86_64

How reproducible:
Always

Steps to Reproduce:
1. $ cat HelloWait.java
public class HelloWait {

    public static void main(String[] args) {
        System.out.println("Hello World");
        try {
            while (true) {
                Thread.sleep(200);
            }
        } catch (InterruptedException e) {
            // ignore
        }
    }

}
2. $ javac HelloWait.java
3. $ sudo debuginfo-install java-1.8.0-openjdk{,-devel,-headless}
4. $ java -XX:NativeMemoryTracking=detail HelloWait
Hello World
<sleeps>
5. In a separate terminal: $ jcmd HelloWait VM.native_memory detail

Actual results:
============================================================================
$ jcmd HelloWait VM.native_memory detail
8810:

Native Memory Tracking:

Total: reserved=5533723KB, committed=445951KB
-                 Java Heap (reserved=4028416KB, committed=251904KB)
                            (mmap: reserved=4028416KB, committed=251904KB) 
 
-                     Class (reserved=1066109KB, committed=14205KB)
                            (classes #411)
                            (malloc=9341KB #149) 
                            (mmap: reserved=1056768KB, committed=4864KB) 
 
-                    Thread (reserved=20608KB, committed=20608KB)
                            (thread #21)
                            (stack: reserved=20560KB, committed=20560KB)
                            (malloc=25KB #110) 
                            (arena=23KB #40)
 
-                      Code (reserved=249635KB, committed=2571KB)
                            (malloc=35KB #306) 
                            (mmap: reserved=249600KB, committed=2536KB) 
 
-                        GC (reserved=157567KB, committed=145275KB)
                            (malloc=10383KB #129) 
                            (mmap: reserved=147184KB, committed=134892KB) 
 
-                  Compiler (reserved=133KB, committed=133KB)
                            (malloc=2KB #25) 
                            (arena=131KB #3)
 
-                  Internal (reserved=9458KB, committed=9458KB)
                            (malloc=9426KB #1418) 
                            (mmap: reserved=32KB, committed=32KB) 
 
-                    Symbol (reserved=1478KB, committed=1478KB)
                            (malloc=958KB #94) 
                            (arena=520KB #1)
 
-    Native Memory Tracking (reserved=145KB, committed=145KB)
                            (malloc=88KB #1391) 
                            (tracking overhead=58KB)
 
-               Arena Chunk (reserved=175KB, committed=175KB)
                            (malloc=175KB) 
 
Virtual memory map:
 
[0x00000006ca200000 - 0x00000007c0000000] reserved 4028416KB for Java Heap from
    [0x00007f7a20cf9103]
    [0x00007f7a20cbf40f]
    [0x00007f7a20adb920]
    [0x00007f7a20cbfa8a]

	[0x00000006ca200000 - 0x00000006d4600000] committed 167936KB from
            [0x00007f7a20b30446]
            [0x00007f7a20b1fafc]
            [0x00007f7a20466d9e]
            [0x00007f7a20adba61]

	[0x000000076e100000 - 0x0000000773300000] committed 83968KB from
            [0x00007f7a20b30446]
            [0x00007f7a20b32496]
            [0x00007f7a20466d64]
            [0x00007f7a20adba61]
 
[0x00000007c0000000 - 0x0000000800000000] reserved 1048576KB for Class from
    [0x00007f7a20cf8491]
    [0x00007f7a20a2e3ff]
    [0x00007f7a20a31334]
    [0x00007f7a20cbfdd5]
[...]

Expected results:
=============================================================================
$ jcmd HelloWait VM.native_memory detail
8987:

Native Memory Tracking:

Total: reserved=5533650KB, committed=445878KB
-                 Java Heap (reserved=4028416KB, committed=251904KB)
                            (mmap: reserved=4028416KB, committed=251904KB) 
 
-                     Class (reserved=1066111KB, committed=14207KB)
                            (classes #404)
                            (malloc=9343KB #156) 
                            (mmap: reserved=1056768KB, committed=4864KB) 
 
-                    Thread (reserved=20646KB, committed=20646KB)
                            (thread #21)
                            (stack: reserved=20560KB, committed=20560KB)
                            (malloc=62KB #110) 
                            (arena=23KB #36)
 
-                      Code (reserved=249632KB, committed=2568KB)
                            (malloc=32KB #303) 
                            (mmap: reserved=249600KB, committed=2536KB) 
 
-                        GC (reserved=157567KB, committed=145275KB)
                            (malloc=10383KB #129) 
                            (mmap: reserved=147184KB, committed=134892KB) 
 
-                  Compiler (reserved=132KB, committed=132KB)
                            (malloc=2KB #24) 
                            (arena=131KB #7)
 
-                  Internal (reserved=9458KB, committed=9458KB)
                            (malloc=9426KB #1403) 
                            (mmap: reserved=32KB, committed=32KB) 
 
-                    Symbol (reserved=1358KB, committed=1358KB)
                            (malloc=902KB #88) 
                            (arena=456KB #1)
 
-    Native Memory Tracking (reserved=155KB, committed=155KB)
                            (malloc=98KB #1389) 
                            (tracking overhead=57KB)
 
-               Arena Chunk (reserved=175KB, committed=175KB)
                            (malloc=175KB) 
 
Virtual memory map:
 
[0x00000006ca200000 - 0x00000007c0000000] reserved 4028416KB for Java Heap from
    [0x00007faf42f483f4] ReservedHeapSpace::ReservedHeapSpace(unsigned long, unsigned long, bool, char*)+0xb4
    [0x00007faf42f0fa8f] Universe::reserve_heap(unsigned long, unsigned long)+0x38f
    [0x00007faf42db7da1] ParallelScavengeHeap::initialize()+0xe1
    [0x00007faf42f100aa] Universe::initialize_heap()+0x16a

	[0x00000006ca200000 - 0x00000006d4600000] committed 167936KB from
            [0x00007faf42e09926] PSVirtualSpace::expand_by(unsigned long)+0x56
            [0x00007faf42dfa2ac] PSOldGen::initialize(ReservedSpace, unsigned long, char const*, int)+0x8c
            [0x00007faf42784dee] AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, GenerationSizer*, unsigned long)+0x2fe
            [0x00007faf42db7ee2] ParallelScavengeHeap::initialize()+0x222

	[0x000000076e100000 - 0x0000000773300000] committed 83968KB from
            [0x00007faf42e09926] PSVirtualSpace::expand_by(unsigned long)+0x56
            [0x00007faf42e0b846] PSYoungGen::initialize(ReservedSpace, unsigned long)+0x96
            [0x00007faf42784db4] AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, GenerationSizer*, unsigned long)+0x2c4
            [0x00007faf42db7ee2] ParallelScavengeHeap::initialize()+0x222
 
[0x00000007c0000000 - 0x0000000800000000] reserved 1048576KB for Class from
    [0x00007faf42f47791] ReservedSpace::ReservedSpace(unsigned long, unsigned long, bool, char*, unsigned long)+0x91
    [0x00007faf42d139c0] Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned char*)+0x40
    [0x00007faf42d14404] Metaspace::global_initialize()+0x554
    [0x00007faf42f103b1] universe_init()+0x61

	[0x00000007c0060000 - 0x00000007c0080000] committed 128KB from
            [0x00007faf42f49390] VirtualSpace::expand_by(unsigned long, bool)+0x140
            [0x00007faf42d10d19] VirtualSpaceList::get_new_chunk(unsigned long, unsigned long, unsigned long)+0x1a9
            [0x00007faf42d120a5] Metaspace::Metaspace(Mutex*, Metaspace::MetaspaceType)+0x2f5
            [0x00007faf429064d6] ClassLoaderData::metaspace_non_null()+0xd6

	[0x00000007c0000000 - 0x00000007c0060000] committed 384KB from
            [0x00007faf42f49390] VirtualSpace::expand_by(unsigned long, bool)+0x140
            [0x00007faf42d10d19] VirtualSpaceList::get_new_chunk(unsigned long, unsigned long, unsigned long)+0x1a9
            [0x00007faf42d120a5] Metaspace::Metaspace(Mutex*, Metaspace::MetaspaceType)+0x2f5
            [0x00007faf429065a0] ClassLoaderData::metaspace_non_null()+0x1a0
 
[0x00007faef0510000 - 0x00007faef8000000] reserved 125888KB for GC from
    [0x00007faf42f478ad] ReservedSpace::ReservedSpace(unsigned long, unsigned long, bool, char*, unsigned long)+0x1ad
    [0x00007faf42dae3b9] ParMarkBitMap::initialize(MemRegion)+0xa9
    [0x00007faf42e01dc1] PSParallelCompact::initialize()+0x151
    [0x00007faf42db8095] ParallelScavengeHeap::initialize()+0x3d5
[...]

Additional info:
=============================================================================
Works fine with an upstream build. Expected results have been gathered that way.

Comment 1 Zhengyu 2017-12-05 14:54:57 UTC
Any changes on compiler? JVM decoder seems not happy with ELF sections of libjli.so and libjvm.so and tagged them invalid.

(gdb) p *$2->_opened_elf_files
$4 = {<CHeapObj<(MemoryType)7>> = {<No data fields>}, m_next = 0x7f7538005c10, 
  m_filepath = 0x7f75380054e0 "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.fc25.x86_64/jre/bin/../lib/amd64/jli/libjli.so", 
  m_file = 0x7f7538005550, m_elfHdr = {e_ident = "\177ELF\002\001\001\000\000\000\000\000\000\000\000", e_type = 3, e_machine = 62, 
    e_version = 1, e_entry = 9008, e_phoff = 64, e_shoff = 59536, e_flags = 0, e_ehsize = 64, e_phentsize = 56, e_phnum = 7, e_shentsize = 64, 
    e_shnum = 28, e_shstrndx = 27}, m_symbol_tables = 0x7f7538005790, m_string_tables = 0x7f7538005890, m_funcDesc_table = 0x0, 
  m_status = AbstractDecoder::file_invalid}
(gdb) p *$2->_opened_elf_files->m_next
$5 = {<CHeapObj<(MemoryType)7>> = {<No data fields>}, m_next = 0x0, 
  m_filepath = 0x7f7538005d00 "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.fc25.x86_64/jre/lib/amd64/server/libjvm.so", 
  m_file = 0x7f7538005dd0, m_elfHdr = {e_ident = "\177ELF\002\001\001\003\000\000\000\000\000\000\000", e_type = 3, e_machine = 62, 
    e_version = 1, e_entry = 2133024, e_phoff = 64, e_shoff = 15104992, e_flags = 0, e_ehsize = 64, e_phentsize = 56, e_phnum = 7, 
    e_shentsize = 64, e_shnum = 33, e_shstrndx = 32}, m_symbol_tables = 0x7f7538007020, m_string_tables = 0x7f753800c160, 
  m_funcDesc_table = 0x0, m_status = AbstractDecoder::file_invalid}

Comment 2 Severin Gehwolf 2017-12-05 15:30:44 UTC
(In reply to Zhengyu from comment #1)
> Any changes on compiler? JVM decoder seems not happy with ELF sections of
> libjli.so and libjvm.so and tagged them invalid.

The spec file determines the compiler options. Unfortunately the koji builders are down today[1] and I cannot point you to the logs.

FWIW, the build uses eu-readelf from elfutils package and that seems to have no problem with parsing it:

$ eu-readelf -s /usr/lib/jvm/java-1.8.0-openjdk/lib/amd64/jli/libjli.so 

Symbol table [ 4] '.dynsym' contains 99 entries:
 1 local symbol  String table: [ 5] '.dynstr'
  Num:            Value   Size Type    Bind   Vis          Ndx Name
    0: 0000000000000000      0 NOTYPE  LOCAL  DEFAULT    UNDEF 
    1: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __strcat_chk.4 (3)
    2: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF getenv.5 (4)
    3: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __snprintf_chk.4 (3)
    4: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF free.5 (4)
    5: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF putchar.5 (4)
    6: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF pthread_create.5 (5)
    7: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strcasecmp.5 (4)
    8: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __vfprintf_chk.4 (3)
    9: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __errno_location.5 (4)
   10: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strncpy.5 (4)
   11: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strncmp.5 (4)
   12: 0000000000000000      0 NOTYPE  WEAK   DEFAULT    UNDEF _ITM_deregisterTMCloneTable
   13: 0000000000000000      0 OBJECT  GLOBAL DEFAULT    UNDEF stdout.5 (4)
   14: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strcpy.5 (4)
   15: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF inflate
   16: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __read_chk (6)
   17: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF puts.5 (4)
   18: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF readlink.5 (4)
   19: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF getpid.5 (4)
   20: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF fclose.5 (4)
   21: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF opendir.5 (4)
   22: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF stpcpy.5 (4)
   23: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strlen.5 (4)
   24: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __stack_chk_fail (6)
   25: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF getuid.5 (4)
   26: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strchr.5 (4)
   27: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF snprintf.5 (4)
   28: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strrchr.5 (4)
   29: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF geteuid.5 (4)
   30: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF getcwd.5 (4)
   31: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF close.5 (4)
   32: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __realpath_chk (6)
   33: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strspn.5 (4)
   34: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF closedir.5 (4)
   35: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF fputc.5 (4)
   36: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strcspn.5 (4)
   37: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF read.5 (4)
   38: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF memcmp.5 (4)
   39: 0000000000000000      0 OBJECT  WEAK   DEFAULT    UNDEF _environ.5 (4)
   40: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF pthread_attr_init.5 (4)
   41: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF fgets.5 (4)
   42: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF execve.5 (4)
   43: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF putenv.5 (4)
   44: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strcmp.5 (4)
   45: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __memmove_chk.4 (3)
   46: 0000000000000000      0 NOTYPE  WEAK   DEFAULT    UNDEF __gmon_start__
   47: 0000000000000000      0 OBJECT  GLOBAL DEFAULT    UNDEF environ.5 (4)
   48: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF realpath (7)
   49: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF memcpy (8)
   50: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF inflateEnd
   51: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF getgid.5 (4)
   52: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __xstat.5 (4)
   53: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF readdir.5 (4)
   54: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF pthread_attr_setdetachstate.5 (4)
   55: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF dlopen.5 (9)
   56: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF malloc.5 (4)
   57: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF fflush.5 (4)
   58: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF sscanf.5 (4)
   59: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF getegid.5 (4)
   60: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strpbrk.5 (4)
   61: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF dlclose.5 (9)
   62: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF pthread_attr_setstacksize.5 (5)
   63: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF realloc.5 (4)
   64: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __printf_chk.4 (3)
   65: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF open.5 (4)
   66: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF access.5 (4)
   67: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF fopen.5 (4)
   68: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF inflateInit2_
   69: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF perror.5 (4)
   70: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strtok.5 (4)
   71: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF sysconf.5 (4)
   72: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF pthread_join.5 (5)
   73: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF pthread_attr_destroy.5 (4)
   74: 0000000000000000      0 OBJECT  GLOBAL DEFAULT    UNDEF __environ.5 (4)
   75: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF exit.5 (4)
   76: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF lseek64.5 (4)
   77: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __fprintf_chk.4 (3)
   78: 0000000000000000      0 NOTYPE  WEAK   DEFAULT    UNDEF _ITM_registerTMCloneTable
   79: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strdup.5 (4)
   80: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strerror.5 (4)
   81: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF dlsym.5 (9)
   82: 0000000000000000      0 FUNC    WEAK   DEFAULT    UNDEF __cxa_finalize.5 (4)
   83: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF execv.5 (4)
   84: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF strstr.5 (4)
   85: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF dlerror.5 (9)
   86: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __ctype_b_loc (7)
   87: 0000000000000000      0 OBJECT  GLOBAL DEFAULT    UNDEF stderr.5 (4)
   88: 0000000000000000      0 FUNC    GLOBAL DEFAULT    UNDEF __sprintf_chk.4 (3)
   89: 0000000000004c40    225 FUNC    GLOBAL DEFAULT       14 JLI_ReportMessage@@SUNWprivate_1.1
   90: 0000000000006e60    225 FUNC    GLOBAL DEFAULT       14 JLI_ReportErrorMessage@@SUNWprivate_1.1
   91: 0000000000009860    575 FUNC    GLOBAL DEFAULT       14 JLI_ManifestIterate@@SUNWprivate_1.1
   92: 0000000000007070     17 FUNC    GLOBAL DEFAULT       14 JLI_ReportExceptionDescription@@SUNWprivate_1.1
   93: 0000000000004d30   7301 FUNC    GLOBAL DEFAULT       14 JLI_Launch@@SUNWprivate_1.1
   94: 0000000000007740      8 FUNC    GLOBAL DEFAULT       14 JLI_GetStdArgc@@SUNWprivate_1.1
   95: 0000000000000000      0 OBJECT  GLOBAL DEFAULT      ABS SUNWprivate_1.1@@SUNWprivate_1.1
   96: 0000000000007730      8 FUNC    GLOBAL DEFAULT       14 JLI_GetStdArgs@@SUNWprivate_1.1
   97: 0000000000006f50    274 FUNC    GLOBAL DEFAULT       14 JLI_ReportErrorMessageSys@@SUNWprivate_1.1
   98: 0000000000008b30     58 FUNC    GLOBAL DEFAULT       14 JLI_SetTraceLauncher@@SUNWprivate_1.1

[1] https://status.fedoraproject.org/

Comment 3 Severin Gehwolf 2017-12-06 09:49:40 UTC
(In reply to Zhengyu from comment #1)
> Any changes on compiler? JVM decoder seems not happy with ELF sections of
> libjli.so and libjvm.so and tagged them invalid.

Build info of the build for my package was:
https://koji.fedoraproject.org/koji/buildinfo?buildID=989056

Logs for x86_64 for example (warning: it's a huge file), which should include compiler options:
https://kojipkgs.fedoraproject.org//packages/java-1.8.0-openjdk/1.8.0.151/1.b12.fc27/data/logs/x86_64/build.log

From the root.log: The JDK has been compiled with GCC version 7.2.1-2.fc27

Comment 4 Severin Gehwolf 2017-12-06 11:16:22 UTC
I'm pretty sure this has to do with debug-info stripping in the rpms. See [1]. It does not seem to be related to compiler options. If you build OpenJDK locally symbols show up.

$ fedpkg clone -a java-1.8.0-openjdk
$ cd java-1.8.0-openjdk
$ fedpkg switch-branch f27
$ fedpkg local

The image in 'java-1.8.0-openjdk-1.8.0.151-1.b12.fc27.x86_64/openjdk/build/jdk8.build/images/j2sdk-image' in my case has all relevant symbols. Yet, the RPM does not.

So the question is whether or not we could convince the JVM decoder to look at build-id notes and use the library with debug symbols instead.

$ eu-readelf -n /usr/lib/jvm/java-1.8.0-openjdk/lib/amd64/jli/libjli.so | grep -C3 build-id

Note section [ 1] '.note.gnu.build-id' of 36 bytes at offset 0x1c8:
  Owner          Data size  Type
  GNU                   20  GNU_BUILD_ID
    Build ID: 774094b24965e898a2dee839374d8c55a0e7dfe5

Thus, the library to look at would, in my case, be:
/usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.fc27.x86_64/jre/lib/amd64/jli/libjli.so-1.8.0.151-1.b12.fc27.x86_64.debug

That's after some symlink resolution via the build-id:
$ ls -l /usr/lib/debug/.build-id/77/4094b24965e898a2dee839374d8c55a0e7dfe5
lrwxrwxrwx. 1 root root 60 Oct 25 17:02 /usr/lib/debug/.build-id/77/4094b24965e898a2dee839374d8c55a0e7dfe5 -> ../../../.build-id/77/4094b24965e898a2dee839374d8c55a0e7dfe5

Same for libjvm.so

[1] https://fedoraproject.org/wiki/Packaging:Debuginfo?rd=Packaging/Debuginfo

Comment 5 Severin Gehwolf 2017-12-06 15:16:50 UTC
(In reply to Severin Gehwolf from comment #4)
> So the question is whether or not we could convince the JVM decoder to look
> at build-id notes and use the library with debug symbols instead.

I believe this was the commit adding support for this on the serviceability agent side:
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/rev/7de45b5044c3

Comment 6 Zhengyu 2017-12-13 14:36:13 UTC
jdk8u patch http://cr.openjdk.java.net/~zgu/8193318/8u/webrev.00/ has been verified by Severin.

Upstream work has to wait for jdk11.

Comment 7 Zhengyu 2018-03-28 17:11:54 UTC
Upstream effort stalled (http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2018-March/026992.html)

We are now considering not to strip debug symbols from OpenJDK bundles, as Oracle JDK bundles, which should be done during RPM build.

Comment 8 Zhengyu 2018-03-28 18:32:28 UTC
Reassigned to Jiri to make build change.

Comment 9 jiri vanek 2018-03-29 09:14:56 UTC
We agreed that no-strip at all is not an option. So what packaging changes do ou have in mind? Imho what is necessary to be achieved, cant be achieved via simple rpm changes.

Comment 10 Zhengyu 2018-03-29 12:14:49 UTC
Upstream jdk build already stripping symbols with -g option for none minimal VMs. Only thing we need to do is to prevent further stripping during RPM build.

Comment 11 jiri vanek 2018-03-29 12:49:47 UTC
(In reply to Zhengyu from comment #10)
> Upstream jdk build already stripping symbols with -g option for none minimal

This should be disabled during our rpm build.

> VMs. Only thing we need to do is to prevent further stripping during RPM
> build.

We cant strip all. That would lead to enormous binaries. And I dont know about any partial striping.

What led you to conclusion, rpm can handle this?

I remember our email discussion well, but i do not recall any final statement, that rpm can lower the stripping.  Can you be little bit more sepcific what is expected from rpm?  AFAIU it needs patch to rpm first

Comment 24 Ben Cotton 2018-11-27 16:25:03 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 25 Severin Gehwolf 2018-11-28 09:32:49 UTC
(In reply to jiri vanek from comment #21)
> So here is attempt: 
> https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/c/
> 57b83006b016e6f3e951bad719ee9b69b5921744?branch=master
> https://koji.fedoraproject.org/koji/taskinfo?taskID=26042483

This seems to have worked:

$ rpm -q java-1.8.0-openjdk
java-1.8.0-openjdk-1.8.0.191.b12-8.fc28.x86_64

Attaching the detailed native memory tracking output with the fixed JDK.

> Is jdk10 also affected?

Not sure about JDK 10, but I can reproduce on JDK 11. It needs a similar fix. I'll prepare a PR.

Comment 26 Severin Gehwolf 2018-11-28 09:33:37 UTC
Created attachment 1509420 [details]
detailed memory tracking of HelloWait with fixed JDK build.

Comment 27 Severin Gehwolf 2018-11-28 09:34:57 UTC
Closing with resolution CURRENTRELEASE per comment 25 and comment 26.

Comment 28 Severin Gehwolf 2018-11-28 09:43:21 UTC
(In reply to Severin Gehwolf from comment #25)
> (In reply to jiri vanek from comment #21)
> > So here is attempt: 
> > https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/c/
> > 57b83006b016e6f3e951bad719ee9b69b5921744?branch=master
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=26042483
> 
> This seems to have worked:
> 
> $ rpm -q java-1.8.0-openjdk
> java-1.8.0-openjdk-1.8.0.191.b12-8.fc28.x86_64
> 
> Attaching the detailed native memory tracking output with the fixed JDK.

I should mention that this now works without debuginfo packages being installed. This makes sense as the fix was to convince find-debuginfo.sh to use 'strip -g' instead of full strip of DSOs and EXEs.

Comment 29 Severin Gehwolf 2018-11-28 12:58:03 UTC
(In reply to Severin Gehwolf from comment #25)
> Not sure about JDK 10, but I can reproduce on JDK 11. It needs a similar
> fix. I'll prepare a PR.

Here it is:
https://src.fedoraproject.org/rpms/java-11-openjdk/pull-request/24