Bug 1520879
Summary: | Detailed Native Memory Tracking does not show stack traces with debug-info installed | ||||||
---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Severin Gehwolf <sgehwolf> | ||||
Component: | java-1.8.0-openjdk | Assignee: | jiri vanek <jvanek> | ||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 27 | CC: | dbhole, jerboaa, jvanek, msrb, mvala, omajid, zgu | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2018-11-28 09:34:57 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Severin Gehwolf
2017-12-05 11:30:12 UTC
Any changes on compiler? JVM decoder seems not happy with ELF sections of libjli.so and libjvm.so and tagged them invalid. (gdb) p *$2->_opened_elf_files $4 = {<CHeapObj<(MemoryType)7>> = {<No data fields>}, m_next = 0x7f7538005c10, m_filepath = 0x7f75380054e0 "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.fc25.x86_64/jre/bin/../lib/amd64/jli/libjli.so", m_file = 0x7f7538005550, m_elfHdr = {e_ident = "\177ELF\002\001\001\000\000\000\000\000\000\000\000", e_type = 3, e_machine = 62, e_version = 1, e_entry = 9008, e_phoff = 64, e_shoff = 59536, e_flags = 0, e_ehsize = 64, e_phentsize = 56, e_phnum = 7, e_shentsize = 64, e_shnum = 28, e_shstrndx = 27}, m_symbol_tables = 0x7f7538005790, m_string_tables = 0x7f7538005890, m_funcDesc_table = 0x0, m_status = AbstractDecoder::file_invalid} (gdb) p *$2->_opened_elf_files->m_next $5 = {<CHeapObj<(MemoryType)7>> = {<No data fields>}, m_next = 0x0, m_filepath = 0x7f7538005d00 "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.fc25.x86_64/jre/lib/amd64/server/libjvm.so", m_file = 0x7f7538005dd0, m_elfHdr = {e_ident = "\177ELF\002\001\001\003\000\000\000\000\000\000\000", e_type = 3, e_machine = 62, e_version = 1, e_entry = 2133024, e_phoff = 64, e_shoff = 15104992, e_flags = 0, e_ehsize = 64, e_phentsize = 56, e_phnum = 7, e_shentsize = 64, e_shnum = 33, e_shstrndx = 32}, m_symbol_tables = 0x7f7538007020, m_string_tables = 0x7f753800c160, m_funcDesc_table = 0x0, m_status = AbstractDecoder::file_invalid} (In reply to Zhengyu from comment #1) > Any changes on compiler? JVM decoder seems not happy with ELF sections of > libjli.so and libjvm.so and tagged them invalid. The spec file determines the compiler options. Unfortunately the koji builders are down today[1] and I cannot point you to the logs. FWIW, the build uses eu-readelf from elfutils package and that seems to have no problem with parsing it: $ eu-readelf -s /usr/lib/jvm/java-1.8.0-openjdk/lib/amd64/jli/libjli.so Symbol table [ 4] '.dynsym' contains 99 entries: 1 local symbol String table: [ 5] '.dynstr' Num: Value Size Type Bind Vis Ndx Name 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UNDEF 1: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __strcat_chk.4 (3) 2: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF getenv.5 (4) 3: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __snprintf_chk.4 (3) 4: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF free.5 (4) 5: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF putchar.5 (4) 6: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF pthread_create.5 (5) 7: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strcasecmp.5 (4) 8: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __vfprintf_chk.4 (3) 9: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __errno_location.5 (4) 10: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strncpy.5 (4) 11: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strncmp.5 (4) 12: 0000000000000000 0 NOTYPE WEAK DEFAULT UNDEF _ITM_deregisterTMCloneTable 13: 0000000000000000 0 OBJECT GLOBAL DEFAULT UNDEF stdout.5 (4) 14: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strcpy.5 (4) 15: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF inflate 16: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __read_chk (6) 17: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF puts.5 (4) 18: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF readlink.5 (4) 19: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF getpid.5 (4) 20: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF fclose.5 (4) 21: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF opendir.5 (4) 22: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF stpcpy.5 (4) 23: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strlen.5 (4) 24: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __stack_chk_fail (6) 25: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF getuid.5 (4) 26: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strchr.5 (4) 27: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF snprintf.5 (4) 28: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strrchr.5 (4) 29: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF geteuid.5 (4) 30: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF getcwd.5 (4) 31: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF close.5 (4) 32: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __realpath_chk (6) 33: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strspn.5 (4) 34: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF closedir.5 (4) 35: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF fputc.5 (4) 36: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strcspn.5 (4) 37: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF read.5 (4) 38: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF memcmp.5 (4) 39: 0000000000000000 0 OBJECT WEAK DEFAULT UNDEF _environ.5 (4) 40: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF pthread_attr_init.5 (4) 41: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF fgets.5 (4) 42: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF execve.5 (4) 43: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF putenv.5 (4) 44: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strcmp.5 (4) 45: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __memmove_chk.4 (3) 46: 0000000000000000 0 NOTYPE WEAK DEFAULT UNDEF __gmon_start__ 47: 0000000000000000 0 OBJECT GLOBAL DEFAULT UNDEF environ.5 (4) 48: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF realpath (7) 49: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF memcpy (8) 50: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF inflateEnd 51: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF getgid.5 (4) 52: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __xstat.5 (4) 53: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF readdir.5 (4) 54: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF pthread_attr_setdetachstate.5 (4) 55: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF dlopen.5 (9) 56: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF malloc.5 (4) 57: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF fflush.5 (4) 58: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF sscanf.5 (4) 59: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF getegid.5 (4) 60: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strpbrk.5 (4) 61: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF dlclose.5 (9) 62: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF pthread_attr_setstacksize.5 (5) 63: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF realloc.5 (4) 64: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __printf_chk.4 (3) 65: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF open.5 (4) 66: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF access.5 (4) 67: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF fopen.5 (4) 68: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF inflateInit2_ 69: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF perror.5 (4) 70: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strtok.5 (4) 71: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF sysconf.5 (4) 72: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF pthread_join.5 (5) 73: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF pthread_attr_destroy.5 (4) 74: 0000000000000000 0 OBJECT GLOBAL DEFAULT UNDEF __environ.5 (4) 75: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF exit.5 (4) 76: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF lseek64.5 (4) 77: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __fprintf_chk.4 (3) 78: 0000000000000000 0 NOTYPE WEAK DEFAULT UNDEF _ITM_registerTMCloneTable 79: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strdup.5 (4) 80: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strerror.5 (4) 81: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF dlsym.5 (9) 82: 0000000000000000 0 FUNC WEAK DEFAULT UNDEF __cxa_finalize.5 (4) 83: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF execv.5 (4) 84: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF strstr.5 (4) 85: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF dlerror.5 (9) 86: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __ctype_b_loc (7) 87: 0000000000000000 0 OBJECT GLOBAL DEFAULT UNDEF stderr.5 (4) 88: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __sprintf_chk.4 (3) 89: 0000000000004c40 225 FUNC GLOBAL DEFAULT 14 JLI_ReportMessage@@SUNWprivate_1.1 90: 0000000000006e60 225 FUNC GLOBAL DEFAULT 14 JLI_ReportErrorMessage@@SUNWprivate_1.1 91: 0000000000009860 575 FUNC GLOBAL DEFAULT 14 JLI_ManifestIterate@@SUNWprivate_1.1 92: 0000000000007070 17 FUNC GLOBAL DEFAULT 14 JLI_ReportExceptionDescription@@SUNWprivate_1.1 93: 0000000000004d30 7301 FUNC GLOBAL DEFAULT 14 JLI_Launch@@SUNWprivate_1.1 94: 0000000000007740 8 FUNC GLOBAL DEFAULT 14 JLI_GetStdArgc@@SUNWprivate_1.1 95: 0000000000000000 0 OBJECT GLOBAL DEFAULT ABS SUNWprivate_1.1@@SUNWprivate_1.1 96: 0000000000007730 8 FUNC GLOBAL DEFAULT 14 JLI_GetStdArgs@@SUNWprivate_1.1 97: 0000000000006f50 274 FUNC GLOBAL DEFAULT 14 JLI_ReportErrorMessageSys@@SUNWprivate_1.1 98: 0000000000008b30 58 FUNC GLOBAL DEFAULT 14 JLI_SetTraceLauncher@@SUNWprivate_1.1 [1] https://status.fedoraproject.org/ (In reply to Zhengyu from comment #1) > Any changes on compiler? JVM decoder seems not happy with ELF sections of > libjli.so and libjvm.so and tagged them invalid. Build info of the build for my package was: https://koji.fedoraproject.org/koji/buildinfo?buildID=989056 Logs for x86_64 for example (warning: it's a huge file), which should include compiler options: https://kojipkgs.fedoraproject.org//packages/java-1.8.0-openjdk/1.8.0.151/1.b12.fc27/data/logs/x86_64/build.log From the root.log: The JDK has been compiled with GCC version 7.2.1-2.fc27 I'm pretty sure this has to do with debug-info stripping in the rpms. See [1]. It does not seem to be related to compiler options. If you build OpenJDK locally symbols show up. $ fedpkg clone -a java-1.8.0-openjdk $ cd java-1.8.0-openjdk $ fedpkg switch-branch f27 $ fedpkg local The image in 'java-1.8.0-openjdk-1.8.0.151-1.b12.fc27.x86_64/openjdk/build/jdk8.build/images/j2sdk-image' in my case has all relevant symbols. Yet, the RPM does not. So the question is whether or not we could convince the JVM decoder to look at build-id notes and use the library with debug symbols instead. $ eu-readelf -n /usr/lib/jvm/java-1.8.0-openjdk/lib/amd64/jli/libjli.so | grep -C3 build-id Note section [ 1] '.note.gnu.build-id' of 36 bytes at offset 0x1c8: Owner Data size Type GNU 20 GNU_BUILD_ID Build ID: 774094b24965e898a2dee839374d8c55a0e7dfe5 Thus, the library to look at would, in my case, be: /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.fc27.x86_64/jre/lib/amd64/jli/libjli.so-1.8.0.151-1.b12.fc27.x86_64.debug That's after some symlink resolution via the build-id: $ ls -l /usr/lib/debug/.build-id/77/4094b24965e898a2dee839374d8c55a0e7dfe5 lrwxrwxrwx. 1 root root 60 Oct 25 17:02 /usr/lib/debug/.build-id/77/4094b24965e898a2dee839374d8c55a0e7dfe5 -> ../../../.build-id/77/4094b24965e898a2dee839374d8c55a0e7dfe5 Same for libjvm.so [1] https://fedoraproject.org/wiki/Packaging:Debuginfo?rd=Packaging/Debuginfo (In reply to Severin Gehwolf from comment #4) > So the question is whether or not we could convince the JVM decoder to look > at build-id notes and use the library with debug symbols instead. I believe this was the commit adding support for this on the serviceability agent side: http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/rev/7de45b5044c3 jdk8u patch http://cr.openjdk.java.net/~zgu/8193318/8u/webrev.00/ has been verified by Severin. Upstream work has to wait for jdk11. Upstream effort stalled (http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2018-March/026992.html) We are now considering not to strip debug symbols from OpenJDK bundles, as Oracle JDK bundles, which should be done during RPM build. Reassigned to Jiri to make build change. We agreed that no-strip at all is not an option. So what packaging changes do ou have in mind? Imho what is necessary to be achieved, cant be achieved via simple rpm changes. Upstream jdk build already stripping symbols with -g option for none minimal VMs. Only thing we need to do is to prevent further stripping during RPM build. (In reply to Zhengyu from comment #10) > Upstream jdk build already stripping symbols with -g option for none minimal This should be disabled during our rpm build. > VMs. Only thing we need to do is to prevent further stripping during RPM > build. We cant strip all. That would lead to enormous binaries. And I dont know about any partial striping. What led you to conclusion, rpm can handle this? I remember our email discussion well, but i do not recall any final statement, that rpm can lower the stripping. Can you be little bit more sepcific what is expected from rpm? AFAIU it needs patch to rpm first So here is attempt: https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/c/57b83006b016e6f3e951bad719ee9b69b5921744?branch=master https://koji.fedoraproject.org/koji/taskinfo?taskID=26042483 Is jdk10 also affected? (https://bugzilla.redhat.com/show_bug.cgi?id=1557371) This message is a reminder that Fedora 27 is nearing its end of life. On 2018-Nov-30 Fedora will stop maintaining and issuing updates for Fedora 27. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '27'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 27 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. (In reply to jiri vanek from comment #21) > So here is attempt: > https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/c/ > 57b83006b016e6f3e951bad719ee9b69b5921744?branch=master > https://koji.fedoraproject.org/koji/taskinfo?taskID=26042483 This seems to have worked: $ rpm -q java-1.8.0-openjdk java-1.8.0-openjdk-1.8.0.191.b12-8.fc28.x86_64 Attaching the detailed native memory tracking output with the fixed JDK. > Is jdk10 also affected? Not sure about JDK 10, but I can reproduce on JDK 11. It needs a similar fix. I'll prepare a PR. Created attachment 1509420 [details]
detailed memory tracking of HelloWait with fixed JDK build.
Closing with resolution CURRENTRELEASE per comment 25 and comment 26. (In reply to Severin Gehwolf from comment #25) > (In reply to jiri vanek from comment #21) > > So here is attempt: > > https://src.fedoraproject.org/rpms/java-1.8.0-openjdk/c/ > > 57b83006b016e6f3e951bad719ee9b69b5921744?branch=master > > https://koji.fedoraproject.org/koji/taskinfo?taskID=26042483 > > This seems to have worked: > > $ rpm -q java-1.8.0-openjdk > java-1.8.0-openjdk-1.8.0.191.b12-8.fc28.x86_64 > > Attaching the detailed native memory tracking output with the fixed JDK. I should mention that this now works without debuginfo packages being installed. This makes sense as the fix was to convince find-debuginfo.sh to use 'strip -g' instead of full strip of DSOs and EXEs. (In reply to Severin Gehwolf from comment #25) > Not sure about JDK 10, but I can reproduce on JDK 11. It needs a similar > fix. I'll prepare a PR. Here it is: https://src.fedoraproject.org/rpms/java-11-openjdk/pull-request/24 |