RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1180512 - Unable to profile jit'd JVM code when using static huge pages
Summary: Unable to profile jit'd JVM code when using static huge pages
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: oprofile
Version: 7.2
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: ---
Assignee: William Cohen
QA Contact: Michael Petlan
Lenka Špačková
URL:
Whiteboard:
Depends On:
Blocks: 1205796
TreeView+ depends on / blocked
 
Reported: 2015-01-09 10:36 UTC by John O'Hara
Modified: 2015-11-19 04:12 UTC (History)
6 users (show)

Fixed In Version: oprofile-0.9.9-13.el7
Doc Type: Bug Fix
Doc Text:
The operf tool now recognizes static huge page identifiers Previously, when profiling performance of Java just-in-time (JIT) compiled code with static huge pages enabled, OProfile's operf command recorded a large number of event samples to anonymous memory (in anon_hugepage) instead of to the appropriate Java method. With this update, operf recognizes the static huge page identifiers and correctly maps samples to Java methods when using statically allocated huge pages.
Clone Of:
Environment:
Last Closed: 2015-11-19 04:12:45 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2201 0 normal SHIPPED_LIVE oprofile bug fix and enhancement update 2015-11-19 08:04:05 UTC

Description John O'Hara 2015-01-09 10:36:09 UTC 
Description of problem:

When oprofile is used to profile JIT'd code where the java process has been started with huge page memory enabled for the process a large percentage of time is reported in anon_hugepage. Oprofile does not report to see JIT'd method calls in the perf report.

If I do not start the process with hugespages, we get the correct reporting of jvm method calls. 


How reproducible:


Steps to Reproduce:
1. Using the DaCapo benchmark: http://www.dacapobench.org/
2. profile jvm using: 'operf java -XX:+UseLargePages -agentlib:jvmti_oprofile -jar dacapo-9.12-bach.jar sunflow'
3. view captured data: 'opreport --symbols --threshold 1.0'

Actual results:

samples  %        image name               symbol name
582252   84.4164  anon_hugepage (deleted)  /anon_hugepage (deleted)
68864     9.9841  libjvm.so                /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el7_0.x86_64/jre/lib/amd64/server/libjvm.so
23543     3.4133  no-vmlinux               /no-vmlinux
11087     1.6074  libjava.so               /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el7_0.x86_64/jre/lib/amd64/libjava.so

Expected results:

samples  %        image name               symbol name
157817   21.8364  2838.jo                  void org.sunflow.core.accel.KDTree.intersect(org.sunflow.core.Ray, org.sunflow.core.IntersectionState)
147963   20.4729  2838.jo                  Interpreter
131928   18.2542  2838.jo                  void org.sunflow.core.accel.BoundingIntervalHierarchy.intersect(org.sunflow.core.Ray, org.sunflow.core.IntersectionState)~1
104371   14.4413  2838.jo                  void org.sunflow.core.primitive.TriangleMesh$WaldTriangle.intersect(org.sunflow.core.Ray, int, org.sunflow.core.IntersectionState)
79441    10.9919  libjvm.so                /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el7_0.x86_64/jre/lib/amd64/server/libjvm.so
37384     5.1726  2838.jo                  org.sunflow.image.Color org.sunflow.core.gi.InstantGI.getIrradiance(org.sunflow.core.ShadingState, org.sunflow.image.Color)~1
23981     3.3181  no-vmlinux               /no-vmlinux
11256     1.5574  libjava.so               /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el7_0.x86_64/jre/lib/amd64/libjava.so
9761      1.3506  2838.jo                  void org.sunflow.core.light.TriangleMeshLight$TriangleLight.getSamples(org.sunflow.core.ShadingState)

Additional info:
Comment 18 William Cohen 2015-03-16 18:30:25 UTC 
Can use the following patch from upstream oprofile that has been applied to rhel6.7 version of oprofile to address this issue:

commit 62ceb2731c2aee79fe4c588d492d6dffef347a75
Author: William Cohen <wcohen>
Date:   Fri Jan 9 16:44:09 2015 -0500

    Allow operf to track anon_hugepage mmap entries
    
    The perf mmap information for anon_huge pages has a different filename
    ("/anon_hugepage") than the mmap information for regions composed of
    normal sized pages ("//anon").  This results in opreport not being
    able to map samples collected by operf to Java methods when the Java
    VM uses statically allocated huge pages (rhbz1180512 and rhbz1180513).
    
    Signed-off-by: William Cohen <wcohen>
Comment 19 William Cohen 2015-04-07 20:01:10 UTC 
This has been addressed in oprofile-0.9.9-11.el7
Comment 21 Michael Petlan 2015-06-05 13:12:41 UTC 
Tested with oprofile-0.9.9 releases 7.el7 and 11.el7:

x86_64 OLD: XFAIL
x86_64 NEW: XPASS

ppc64 OLD: XFAIL
ppc64 NEW: FAIL ?? [1]

ppc64le OLD: XFAIL
ppc64le NEW: FAIL ?? [2]

aarch64 OLD: XFAIL
aarch64 NEW: XPASS


So x86_64 and aarch64 look fine, on ppc64 something changed since 7.el7, but the log from NEW seems to be not very useful, since the methods are not there.
On ppc64le the logs are the same in both cases (11.el7 compared to 7.ael7b.

William, do ppc64 and ppc64le deserve some additional fixing?


--------------------------------------------

[1] OLD and NEW on ppc64

Using /tmp/tmp.nEbwkCsIBB/oprofile_data/samples/ for samples directory.
WARNING: Lost samples detected! See /tmp/tmp.nEbwkCsIBB/oprofile_data/samples/operf.log for details.
warning: /anon_hugepage (deleted) could not be found.
warning: /no-vmlinux could not be found.
warning: [hypervisor_bucket] (tgid:40147 range:0x984-0x1523e4c) could not be found.
warning: [vdso] (tgid:40147 range:0x3fff9cb20000-0x3fff9cb3ffff) could not be found.
CPU: ppc64 POWER8, speed 3425 MHz (estimated)
Counted CYCLES events (Cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
1702329  88.6395  anon_hugepage (deleted)  /anon_hugepage (deleted)
155739    8.1093  libjvm.so                /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.ppc64/jre/lib/ppc64/server/libjvm.so
38574     2.0085  libjava.so               /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.ppc64/jre/lib/ppc64/libjava.so

--------

Using /tmp/tmp.vttA30sM1k/oprofile_data/samples/ for samples directory.
warning: /no-vmlinux could not be found.
warning: [hypervisor_bucket] (tgid:40462 range:0x980-0x152a694) could not be found.
warning: [vdso] (tgid:40462 range:0x3fff8a610000-0x3fff8a62ffff) could not be found.
CPU: ppc64 POWER8, speed 3425 MHz (estimated)
Counted CYCLES events (Cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
1726787  88.6889  anon (tgid:40462 range:0x3efff0000000-0x3effffffffff) anon (tgid:40462 range:0x3efff0000000-0x3effffffffff)
159630    8.1987  libjvm.so                /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.ppc64/jre/lib/ppc64/server/libjvm.so
38538     1.9793  libjava.so               /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.el7_1.ppc64/jre/lib/ppc64/libjava.so



[2] OLD and NEW on ppc64le

Using /tmp/tmp.jTG1ji1506/oprofile_data/samples/ for samples directory.
WARNING: Lost samples detected! See /tmp/tmp.jTG1ji1506/oprofile_data/samples/operf.log for details.
warning: /no-vmlinux could not be found.
warning: [hypervisor_bucket] (tgid:11941 range:0x508-0xaf4f4) could not be found.
warning: [vdso] (tgid:11941 range:0x3fffad990000-0x3fffad9affff) could not be found.
CPU: ppc64 POWER8, speed 3425 MHz (estimated)
Counted CYCLES events (Cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
555894   82.2849  anon (tgid:11941 range:0x3fff98000000-0x3fff983fffff) anon (tgid:11941 range:0x3fff98000000-0x3fff983fffff)
38583     5.7112  libjvm.so                BytecodeInterpreter::run(BytecodeInterpreter*)
16698     2.4717  libjava.so               __j__ieee754_sqrt
7181      1.0630  no-vmlinux               /no-vmlinux

--------

Using /tmp/tmp.o8bcTqAokR/oprofile_data/samples/ for samples directory.
WARNING: Lost samples detected! See /tmp/tmp.o8bcTqAokR/oprofile_data/samples/operf.log for details.
warning: /no-vmlinux could not be found.
warning: [hypervisor_bucket] (tgid:12250 range:0x530-0xaf59c) could not be found.
warning: [vdso] (tgid:12250 range:0x3fff81d60000-0x3fff81d7ffff) could not be found.
CPU: ppc64 POWER8, speed 3425 MHz (estimated)
Counted CYCLES events (Cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
532104   82.7045  anon (tgid:12250 range:0x3fff6c000000-0x3fff6c3fffff) anon (tgid:12250 range:0x3fff6c000000-0x3fff6c3fffff)
36758     5.7133  libjvm.so                BytecodeInterpreter::run(BytecodeInterpreter*)
16874     2.6227  libjava.so               __j__ieee754_sqrt



[3] OLD and NEW on aarch64

Using /tmp/tmp.6L81XnYVsI/oprofile_data/samples/ for samples directory.
warning: /no-vmlinux could not be found.
warning: [vdso] (tgid:27430 range:0x3ffa6160000-0x3ffa616ffff) could not be found.
CPU: APM X-Gene
Counted CPU_CYCLES events (Cycle) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
173704   72.3194  libjvm.so                /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.3.aa7a.aarch64/jre/lib/aarch64/server/libjvm.so
29525    12.2924  27430.jo                 Interpreter
15062     6.2709  libjava.so               /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.3.aa7a.aarch64/jre/lib/aarch64/libjava.so
6615      2.7541  no-vmlinux               /no-vmlinux
2720      1.1324  ld-2.17.so               _dl_tlsdesc_dynamic

--------

Using /tmp/tmp.waleRYyEJS/oprofile_data/samples/ for samples directory.
WARNING: Lost samples detected! See /tmp/tmp.waleRYyEJS/oprofile_data/samples/operf.log for details.
warning: /no-vmlinux could not be found.
warning: [vdso] (tgid:28808 range:0x3ff8ed50000-0x3ff8ed5ffff) could not be found.
CPU: APM X-Gene
Counted CPU_CYCLES events (Cycle) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
181752   53.0790  libjvm.so                /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.3.aa7a.aarch64/jre/lib/aarch64/server/libjvm.so
73325    21.4139  28808.jo                 Interpreter
34058     9.9463  28808.jo                 Lorg/sunflow/core/gi/InstantGI;getIrradiance(Lorg/sunflow/core/ShadingState;Lorg/sunflow/image/Color;)Lorg/sunflow/image/Color;%80
14624     4.2708  libjava.so               /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.3.aa7a.aarch64/jre/lib/aarch64/libjava.so
10440     3.0489  28808.jo                 void org.sunflow.core.light.TriangleMeshLight$TriangleLight.getSamples(org.sunflow.core.ShadingState)
8390      2.4502  28808.jo                 Lorg/sunflow/core/InstanceList;intersectPrimitive(Lorg/sunflow/core/Ray;ILorg/sunflow/core/IntersectionState;)V%72
5879      1.7169  no-vmlinux               /no-vmlinux
Comment 23 Michael Petlan 2015-06-05 16:10:22 UTC 
Since we need to fix the issue also on POWER.
Comment 24 William Cohen 2015-06-05 17:16:43 UTC 
Testing on ppc64 uncovered an issue that needed an additional patch.  oprofile-0.9.9-13.el7 has been patched.
Comment 26 Michael Petlan 2015-06-05 21:37:40 UTC 
Tested with the rebuild 13.el7. It seems much better on POWERPC now, both LE and BE.
Comment 27 Michael Petlan 2015-06-05 21:50:24 UTC 
Tested on other architectures as well. The release 13.el7 addresses this bug.

VERIFIED.
Comment 29 errata-xmlrpc 2015-11-19 04:12:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2201.html
Note You need to log in before you can comment on or make changes to this bug.