Bug 1567825

for the sake of completeness this is how error looks on SR5 FP11

11:07:48 + java -version
11:07:48 java version "1.8.0_161"
11:07:48 Java(TM) SE Runtime Environment (build 8.0.5.11 - pxi3280sr5fp11-20180326_01(SR5 FP11))
11:07:48 IBM J9 VM (build 2.9, JRE 1.8.0 Linux x86-32 20180309_380776 (JIT enabled, AOT enabled)
11:07:48 OpenJ9   - 49fcaf39
11:07:48 OMR      - 5cbbadf
11:07:48 IBM      - 4453dac)
11:07:48 JCL - 20180319_01 based on Oracle jdk8u161-b12


11:11:22 #0: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so(+0x6f94bb) [0x149b4bb]
11:11:22 #1: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so(+0x6fdbf0) [0x149fbf0]
11:11:22 #2: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so(+0x8b2f5) [0xe2d2f5]
11:11:22 #3: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9prt29.so(+0x1eb8f) [0xbdbb8f]
11:11:22 #4: [0xd6d40c]
11:11:22 Unhandled exception
11:11:22 Type=Segmentation error vmState=0x0005ff06
11:11:22 J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
11:11:22 Handler1=00456DEE Handler2=00BDB99E InaccessibleAddress=0000001C
11:11:22 EDI=94EB3E60 ESI=00000000 EAX=00000000 EBX=0172055C
11:11:22 ECX=000037A0 EDX=00000000
11:11:22 EIP=0111ED82 ES=007B DS=007B ESP=95E35AF0
11:11:22 EFlags=00010206 CS=0073 SS=007B EBP=94C04240
11:11:22 Module=/home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so
11:11:22 Module_base_address=00DA2000
11:11:22 
11:11:22 Method_being_compiled=org/jboss/msc/service/StabilityMonitor.awaitStability(JLjava/util/concurrent/TimeUnit;Ljava/util/Set;Ljava/util/Set;Lorg/jboss/msc/service/StabilityStatistics;)Z
11:11:22 Target=2_90_20180309_380776 (Linux 2.6.32-696.el6.i686)
11:11:22 CPU=x86 (2 logical CPUs) (0xf5510000 RAM)
11:11:22 ----------- Stack Backtrace -----------
11:11:22 (0x00BF9563 [libj9prt29.so+0x3c563])
11:11:22 (0x00BDBFA8 [libj9prt29.so+0x1efa8])
11:11:22 (0x00BF90CE [libj9prt29.so+0x3c0ce])
11:11:22 (0x00BF91C6 [libj9prt29.so+0x3c1c6])
11:11:22 (0x00BF8C93 [libj9prt29.so+0x3bc93])
11:11:22 (0x00BDBFA8 [libj9prt29.so+0x1efa8])
11:11:22 (0x00BF8C07 [libj9prt29.so+0x3bc07])
11:11:22 (0x004560E8 [libj9vm29.so+0x7e0e8])
11:11:22 (0x00BDBFA8 [libj9prt29.so+0x1efa8])
11:11:22 (0x00455DDE [libj9vm29.so+0x7ddde])
11:11:22 (0x00456E41 [libj9vm29.so+0x7ee41])
11:11:22 (0x00BDBB8F [libj9prt29.so+0x1eb8f])
11:11:22 __kernel_rt_sigreturn+0x0 (0x00D6D40C)
11:11:22 ---------------------------------------
11:11:22 JVMDUMP039I Processing dump event "gpf", detail "" at 2018/04/12 11:10:38 - please wait.
11:11:22 JVMDUMP032I JVM requested System dump using '/home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/embed/core.20180412.111038.7420.0001.dmp' in response to an event

------- Comment From chavez.com 2018-04-17 22:36 EDT-------
I have opened a ticket with Java L3. If a core is generated, can you run the jextract utility on it to collect all the binaries needed to analyze it? See http://www-01.ibm.com/support/docview.wss?uid=swg21222464

------- Comment From chavez.com 2018-04-18 08:40 EDT-------
Citing the specifics instructions for what IBM Java will need (jextracted coredump and corresponding javacore):

a) Set ulimits to avoid the truncated system core

ulimit -c unlimited
ulimit -f unlimited

b) Set the following JVM option:

-Xdump:java+system+snap:events=gpf+abort,request=exclusive+prepwalk+preempt,range=1..2

This will generate javacore, system core and snaptrace on gpf.

c) Please reproduce the issue. The above logs will be generated. Please jextract the coredump:

[path to JDK]/jre/bin/jextract [system dump file]

You can refer to the following links for Linux crash mustgather:

Setup: http://www-01.ibm.com/support/docview.wss?uid=swg21222461
Gather: http://www-01.ibm.com/support/docview.wss?uid=swg21222464

We are preparing required files. They have hundreds of MB. 
1. Is it expected?
2. How to technically deliver such amount of data to you?

------- Comment From chavez.com 2018-04-19 10:01 EDT-------
> We are preparing required files. They have hundreds of MB.
> 1. Is it expected?

I believe the jextract tool finds all the necessary binaries along with the core and other dump txt files and produces a single archive so it could be large but since it should be compressed I wouldn't expect it to be hundreds of MB but can't say for sure why it is in your case.

> 2. How to technically deliver such amount of data to you?

I think the easiest way is to use this site:

https://testcase.software.ibm.com/

login with "anonymous" as the id and navigate to /toibm/linux and select the file and upload as binary. Once uploaded, let me know what the filename was and I will transfer it over to the Java folks. Thanks.

I have uploaded core.20180419.112543.1468.0001.dmp.zip

------- Comment From hannsj_uhl.com 2018-04-19 12:26 EDT-------
.

------- Comment From chavez.com 2018-04-19 15:07 EDT-------
(In reply to comment #10)
> I have uploaded core.20180419.112543.1468.0001.dmp.zip

Thanks. I have downloaded the file from the testcase ftp server and am in the process of uploading it to the Java L3's internal ftp server and then will update their support ticket that it is there.

------- Comment From chavez.com 2018-04-25 16:41 EDT-------
IBM Java L3 runtime team has triaged and done a first analysis of the core provided and the ticket has been passed on to the JIT development team for further assistance.

Dear IBM, any update, we are still facing this issue?

any update?

Hello,
RHEL 6 has entered Phase 3. In phase 3 only Critical impact Security Advisories and selected Urgent Priority Bug Fix Advisories will be accepted.
https://access.redhat.com/support/policy/updates/errata

If this fix is required for RHEL 6. Please also provide a justification for this fix.

Thank You
Joe Kachuck

------- Comment From chavez.com 2018-08-31 15:09 EDT-------
From Java L3 today (I replied asking for a timeframe for the release of the update containing the fix):

"Just a quick update to say that the development team has identified the cause of the crash and is working on a fix for the problem."

------- Comment From chavez.com 2018-09-26 16:16 EDT-------
APAR IJ09089 has been created to document this problem and should be visible in the next day or two. The fix has been scheduled for release in Java 8 SR5 FP22

------- Comment From hannsj_uhl.com 2018-10-08 08:22 EDT-------
(In reply to comment #17)
> APAR IJ09089 has been created to document this problem and should be visible
> in the next day or two. The fix has been scheduled for release in Java 8 SR5
> FP22
.
... which is available since 09/27/2018 from
https://developer.ibm.com/javasdk/downloads/sdk8/
with the fixes included as outlined at
https://developer.ibm.com/javasdk/support/fixes/sdk8/#SR5FP22 ...

We've executed a few test runs using the FP22 and didn't see the issue anymore. Looks like it is fixed. Thanks!

fyi ... with IBM Java 8 SR5 FP25 available for RHEL6 from RHN since 11/08/2018
(see https://access.redhat.com/errata/RHSA-2018:3533 ...)
and per the previous comment I am closing this Red Hat bugzilla now ...
... please correct me if I am wrong ...