Bug 1897563

+++ This bug was initially created as a clone of Bug #1328503 +++

This bug is not dependent on RH version so cloning for RHEL 8

Description of problem:
hsdis-amd64.so could not be found on RHEL. Due to which The Java Hotspot compiler seem to generate not optimal machine specific code for that CPU family.

# java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly
OpenJDK 64-Bit Server VM warning: PrintAssembly is enabled; turning on DebugNonSafepoints to gain additional output
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 7.2
java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64

How reproducible:
Always

Steps to Reproduce:
1. # java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly


Actual results:
# java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly
OpenJDK 64-Bit Server VM warning: PrintAssembly is enabled; turning on DebugNonSafepoints to gain additional output
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled


Expected results:
Print the assembly code.

# java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly
OpenJDK 64-Bit Server VM warning: PrintAssembly is enabled; turning on DebugNonSafepoints to gain additional output
Loaded disassembler from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64/jre/lib/amd64/server/hsdis-amd64.so
Decoding compiled method 0x00007fefc5107d50:
Code:
[Disassembling for mach='i386:x86-64']
[Entry Point]
  # {method} {0x00007fefd8d94738} 'arraycopy' '(Ljava/lang/Object;ILjava/lang/Object;II)V' in 'java/lang/System'
  # parm0:    rsi:rsi   = 'java/lang/Object'
  # parm1:    rdx       = int
  # parm2:    rcx:rcx   = 'java/lang/Object'
  # parm3:    r8        = int
  # parm4:    r9        = int
  #           [sp+0x60]  (sp of caller)
  0x00007fefc5107ec0: mov    0x8(%rsi),%r10d
  0x00007fefc5107ec4: shl    $0x3,%r10
  0x00007fefc5107ec8: cmp    %r10,%rax
  0x00007fefc5107ecb: je     0x000

Additional info:
We have observed Java application performance slowdown on newest HP Z640 workstation with Intel(R) Xeon(R) CPU E5-1650 v3. So, trying to understand the performance drop of the Java DREM instruction on latest Xeon CPU installed on HP Z640 workstation.

Also tried compiling the library from source, but fails.

cd /data/rpmbuild/BUILD/java-1.8.0-openjdk-1.8.0.77-0.b03.el7.x86_64/openjdk/hotspot/src/share/tools/hsdis
BINUTILS=/home/user/git/binutils-gdb/binutils make all64
... lot of output ...
if [ ! -f build/linux-amd64/bfd/libbfd.a ]; then cd build/linux-amd64; make all-opcodes; fi
make[2]: Entering directory `/data/rpmbuild/BUILD/java-1.8.0-openjdk-1.8.0.77-0.b03.el7.x86_64/openjdk/hotspot/src/share/tools/hsdis/build/linux-amd64'
make[2]: *** No rule to make target `all-opcodes'.  Stop.
make[2]: Leaving directory `/data/rpmbuild/BUILD/java-1.8.0-openjdk-1.8.0.77-0.b03.el7.x86_64/openjdk/hotspot/src/share/tools/hsdis/build/linux-amd64'
make[1]: *** [build/linux-amd64/bfd/libbfd.a] Error 2
make[1]: Leaving directory `/data/rpmbuild/BUILD/java-1.8.0-openjdk-1.8.0.77-0.b03.el7.x86_64/openjdk/hotspot/src/share/tools/hsdis'
make: *** [all64] Error 2

--- Additional comment from Andrey Loskutov on 2016-04-19 14:38:24 UTC ---

I've originally reported this bug via RH case 01614360 and would like to correct the problem statement:

the absence of the hsdis-amd64.so on RHEL does *not* cause the JIT compiler to generate slower or different code!

The problem is that *without* hsdis-amd64.so library we are unable to force JIT compiler / JVM to print generated assembly via "-XX:+PrintAssembly" flag.

Not only the hsdis-amd64.so library is not available as rpm for RHEL, it also *cannot be build* from OpenJDK sources distributed by RH.

So if the library cannot be distributed by RH due some limitations or license issues, it should be at least possible to build it from openjdk/hotspot/src/share/tools/hsdis sources. And this is currently failing with the error:

if [ ! -f build/linux-amd64/bfd/libbfd.a ]; then cd build/linux-amd64; make all-opcodes; fi
make[2]: Entering directory `/data/rpmbuild/BUILD/java-1.8.0-openjdk-1.8.0.77-0.b03.el7.x86_64/openjdk/hotspot/src/share/tools/hsdis/build/linux-amd64'
make[2]: *** No rule to make target `all-opcodes'.  Stop.

Looking at the makefile in linux-amd64, the "all-opcodes" target is obviously missing. So either this makefile or the parent one which tries to trigger it were not properly merged or some of the changes from upstream are missing.

--- Additional comment from Andrew John Hughes on 2016-04-19 20:14:33 UTC ---

The hsdis plugin isn't built as part of OpenJDK and requires a binutils source tree to be compiled. The result is not distributable, as it will combine GPLv2 (HotSpot) and GPLv3 (binutils code).

I'm not seeing the error you're getting locally. How are you trying to build it?

$ make -C src/share/tools/hsdis all64
make: Entering directory '/home/andrew/projects/openjdk/upstream/chailatte-u60/hotspot/src/share/tools/hsdis'
LIBARCH = i386
make LP64=1 all
make[1]: Entering directory '/home/andrew/projects/openjdk/upstream/chailatte-u60/hotspot/src/share/tools/hsdis'
LIBARCH = amd64
[ -d build/linux-amd64 ] || mkdir -p build/linux-amd64
/bin/sh: line 0: cd: build/binutils: No such file or directory
(cd build/linux-amd64; CC=gcc CFLAGS="-m64 -fPIC -g -O0" AR="ar" /home/andrew/projects/openjdk/upstream/chailatte-u60/hotspot/src/share/tools/hsdis/configure --disable-nls )
/bin/sh: /home/andrew/projects/openjdk/upstream/chailatte-u60/hotspot/src/share/tools/hsdis/configure: No such file or directory
Makefile:204: recipe for target 'build/linux-amd64/Makefile' failed

The failure here is because I haven't pointed it at a binutils tree to build.

--- Additional comment from Andrew John Hughes on 2016-04-19 20:17:50 UTC ---

FYI there is information about PrintAssembly and the plugin here: https://wiki.openjdk.java.net/display/HotSpot/PrintAssembly

--- Additional comment from Andrey Loskutov on 2016-04-19 20:39:36 UTC ---

(In reply to Andrew John Hughes from comment #3)
> FYI there is information about PrintAssembly and the plugin here:
> https://wiki.openjdk.java.net/display/HotSpot/PrintAssembly

I know. There were two reasons for me for compilaton from sources:
1. There were few sources in internet describing crashes of the plugin used together with OpenJDK. 
2. Trust. I have no idea who and how compiled that so file and so using this in our R&D lab is not really acceptable. 

So I've cloned binutils, tried to build from the head but failed. So I've checked out some older tag (corresponding one of the "known good" versions mentioned in the readme) and the build of binutils part went through but stopped at missing all-opcodes target in hsdis. Check the main hsdis Makefile, you will see that this all-opcodes is called but for whatever reason there is no such target in the other hsdis makefiles. So I guess they were diverged over time. Unfortunately I have not the source tree with history for that (rpm contained only the snapshot) so I can't say if the problem is in RH OpenJDK specific patches or in the upstream code.

--- Additional comment from Andrew John Hughes on 2016-04-19 22:05:51 UTC ---

(In reply to Andrey Loskutov from comment #4)
> (In reply to Andrew John Hughes from comment #3)
> > FYI there is information about PrintAssembly and the plugin here:
> > https://wiki.openjdk.java.net/display/HotSpot/PrintAssembly
> 
> I know. There were two reasons for me for compilaton from sources:
> 1. There were few sources in internet describing crashes of the plugin used
> together with OpenJDK. 
> 2. Trust. I have no idea who and how compiled that so file and so using this
> in our R&D lab is not really acceptable. 
> 

I wasn't suggesting that page for a binary (ugh) but information about the plugin. That's all.

> So I've cloned binutils, tried to build from the head but failed. So I've
> checked out some older tag (corresponding one of the "known good" versions
> mentioned in the readme) and the build of binutils part went through but
> stopped at missing all-opcodes target in hsdis. Check the main hsdis
> Makefile, you will see that this all-opcodes is called but for whatever
> reason there is no such target in the other hsdis makefiles. So I guess they
> were diverged over time. Unfortunately I have not the source tree with
> history for that (rpm contained only the snapshot) so I can't say if the
> problem is in RH OpenJDK specific patches or in the upstream code.

There aren't really any RH patches for 8, at least not in this area. I'll have a look myself, but it'll have to wait until the current security update is out of the way.

--- Additional comment from Chris Williams on 2016-10-24 17:04:50 UTC ---

This BZ has missed 7.3 and will now be flagged for 7.4. As the subsystem teams meet for 7.4 planning this BZ will be considered for the release. For any questions please contact platform-product-leads

--- Additional comment from Chris Williams on 2017-03-06 16:16:42 UTC ---

This Bugzilla has been added to the Release Priority List for RHEL 7.4. It is not a guarantee it will make 7.4. Instead, Engineering, Product Management, QE and CEE have all agreed that this BZ should be addressed in 7.4 and every effort will be made to ensure this BZ makes the release. Please reach out to Chris Williams (cww) if you have any questions.

--- Additional comment from Deepak Bhole on 2017-03-28 20:10:39 UTC ---

Andrew, what is the status of this?

--- Additional comment from Andrew John Hughes on 2017-03-29 05:16:31 UTC ---

As mentioned in comment #2, we can't include this file.

There may be a potential fix for the makefiles here, but it's not something I've had chance to look at yet. It can be delayed to 7.5.

--- Additional comment from Andrew John Hughes on 2017-10-31 20:24:11 UTC ---

Currently busy on the October 2017 CPU and dev freeze is on Friday. Moving this back to rhel-7.6.0.

--- Additional comment from Lukáš Zachar on 2018-05-25 12:51:34 UTC ---

Change in QA contanct. Sorry for the spam.

--- Additional comment from Deepu K S on 2018-06-15 16:32:41 UTC ---

Any chances of getting this into RHEL 7.6 ?

--- Additional comment from Florian Weimer on 2018-07-27 10:36:40 UTC ---

The hsdis license was updated upstream:

http://hg.openjdk.java.net/jdk/jdk/rev/59b0d8afc831

I don't know if this solves the licensing problem, considering that it's for use with Hotspot, which is GPLv2-only.

--- Additional comment from Andrew John Hughes on 2018-11-06 18:24:08 UTC ---

The license has changed, but while it still requires binutils to be linked against HotSpot (not just the plugin), I tend to concur with aph that there's still an issue:

https://mail.openjdk.java.net/pipermail/build-dev/2018-August/022906.html

(though neither of us are lawyers, so this does not constitute legal advice)

If it does get ported to Capstone, then I guess it becomes more of an issue of getting Capstone packaged in RHEL.

--- Additional comment from Deepu K S on 2019-02-26 15:10:14 UTC ---

Any new updates for this bug? Will this get into 7.7?
We have a customer waiting fir long.

--- Additional comment from Andrew John Hughes on 2019-06-05 02:22:13 UTC ---

It's not 100% clear to me what is being requested.

If it's for us to check that the end user can compile hsdis.so from the sources we ship, we can double check that.

If it's for us to ship a hsdis.so, that's not legally possible.

--- Additional comment from Florian Weimer on 2019-06-05 05:46:13 UTC ---

(In reply to Andrew John Hughes from comment #16)
> If it's for us to ship a hsdis.so, that's not legally possible.

There are alternative implementations of hsdis based on Capstone, avoiding the licensing issue.

--- Additional comment from Andrew John Hughes on 2019-06-12 02:55:26 UTC ---

That doesn't appear to exist yet: https://bugs.openjdk.java.net/browse/JDK-8188073

Plus we'd then also presumably need to ship Capstone, as I said above.

--- Additional comment from Deepu K S on 2019-07-02 07:41:58 UTC ---

(In reply to Andrew John Hughes from comment #16)
> It's not 100% clear to me what is being requested.
> 
> If it's for us to check that the end user can compile hsdis.so from the
> sources we ship, we can double check that.
> 
> If it's for us to ship a hsdis.so, that's not legally possible.

I had checked this with customer, and explained them the licencing problems with shipping hsdis.so .

If https://bugs.openjdk.java.net/browse/JDK-8188073 gets implemented, can we have that in RHEL.

They are fine with any solution bringing this bug forward.

Thanks.

--- Additional comment from Vishal Pakolu on 2020-05-11 08:59:26 UTC ---

Hello,

Do we have any updates on this ? 

--
Regards,
Vishal Pakolu

--- Additional comment from Andrew John Hughes on 2020-05-25 15:16:11 UTC ---

There doesn't seem to have been much movement on the OpenJDK capstone support upstream. I'll ping that bug.

It does look like capstone is available in Fedora & EPEL now. I can't see it being added to RHEL 7 at this late stage.

--- Additional comment from Vishal Pakolu on 2020-06-09 12:03:31 UTC ---

Hello Andrew,

Thanks for the update. 

We will wait for your response on further action-plan for this bug. 

--
Regards,
Vishal Pakolu

--- Additional comment from Andrew John Hughes on 2020-06-09 17:50:05 UTC ---

Thanks Vishal. No response so far, but I'll try and look at the existing work myself when I get chance.

In the meantime, can we move this bug over to RHEL 8? I don't see a feature like this being added to RHEL 7 now it is in its later stages of development.

Thanks.

--- Additional comment from Vishal Pakolu on 2020-06-22 06:38:40 UTC ---

Hello Andrew,

Thanks for the update. I am getting a confirmation from customer to check if moving this over to RHEL 8 will be feasible. 

I will update you as soon as I have a confirmation.

--
Regards,
Vishal Pakolu

--- Additional comment from Vishal Pakolu on 2020-06-26 07:16:58 UTC ---

Hello Andrew,

Customer has confirmed that moving this over to RHEL 8 will be feasible. 

--
Regards,
Vishal Pakolu

--- Additional comment from Chris Williams on 2020-10-23 19:44:12 UTC ---

Red Hat Enterprise Linux 7 shipped it's final minor release on Spetember 29th, 2020. 7.9 was the last minor releases scheduled for RHEL 7.
Unless the remaining Bugzillas meet the inclusion criteria for Maintenance Phase 2 this BZ will be closed during the week of November 9th, 2020. 

From the RHEL life cyle page:
https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
"During Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7,Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available."

If this BZ meets the above criteria please flag if for 7.9.z, provide suitable business and technical justifications, and follow the process for Accelrated Fixes:
https://mojo.redhat.com/docs/DOC-80399  

Feature Requests can be moved to RHEL 8 if the desired functionality is not already present in the product. 

Please reach out to the applicable Product Experience Engineer[0] if you have any questions or concerns.  

[0] https://bugzilla.redhat.com/page.cgi?id=agile_component_mapping.html&product=Red+Hat+Enterprise+Linux+7

--- Additional comment from Chris Williams on 2020-11-11 21:53:08 UTC ---

Red Hat Enterprise Linux 7 shipped it's final minor release on September 29th, 2020. 7.9 was the last minor releases scheduled for RHEL 7.
From intial triage it does not appear the remaining Bugzillas meet the inclusion criteria for Maintenance Phase 2 and will now be closed. 

From the RHEL life cycle page:
https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
"During Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7,Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available."

If this BZ was closed in error and meets the above criteria please re-open it flag for 7.9.z, provide suitable business and technical justifications, and follow the process for Accelerated Fixes:
https://source.redhat.com/groups/public/pnt-cxno/pnt_customer_experience_and_operations_wiki/support_delivery_accelerated_fix_release_handbook  

Feature Requests can re-opened and moved to RHEL 8 if the desired functionality is not already present in the product. 

Please reach out to the applicable Product Experience Engineer[0] if you have any questions or concerns.  

[0] https://bugzilla.redhat.com/page.cgi?id=agile_component_mapping.html&product=Red+Hat+Enterprise+Linux+7