Bug 1567825 - IBM8 SDK + RHEL6 32bit JVM crashes
Summary: IBM8 SDK + RHEL6 32bit JVM crashes
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: java-1.8.0-ibm
Version: 6.10
Hardware: i686
OS: Linux
high
high
Target Milestone: rc
: 6.10
Assignee: jiri vanek
QA Contact: zzambers
URL:
Whiteboard:
Depends On:
Blocks: 1414846 1425546
TreeView+ depends on / blocked
 
Reported: 2018-04-16 09:18 UTC by mchoma
Modified: 2018-11-26 10:28 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-26 10:28:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 166767 0 None None None 2019-05-03 14:53:16 UTC

Description mchoma 2018-04-16 09:18:40 UTC
During our tests with IBM8 SDK on RHEL6 32bit we see very often "Segmentation error"

#0: /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9jit29.so(+0x2bab5b) [0x1228b5b]
#1: /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9jit29.so(+0x2bf290) [0x122d290]
#2: /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9jit29.so(+0x48fb5) [0xfb6fb5]
#3: /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9prt29.so(+0x1eb6a) [0x218b6a]
#4: [0xf6d40c]
Unhandled exception
Type=Segmentation error vmState=0x0005ff06
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00D809ED Handler2=00218979 InaccessibleAddress=0000001C
EDI=8DB86380 ESI=00000000 EAX=00000000 EBX=018F1644
ECX=00004480 EDX=00000000
EIP=01495362 ES=007B DS=007B ESP=95D98AD0
EFlags=00010216 CS=0073 SS=007B EBP=8D8C3240
10:55:52,455 DEBUG [org.apache.mina.filter.codec.ProtocolCodecFilter] (NioProcessor-1) Processing a MESSAGE_RECEIVED for session 609
Module=/qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9jit29.so
Module_base_address=00F6E000

Method_being_compiled=org/jboss/msc/service/StabilityMonitor.awaitStability(JLjava/util/concurrent/TimeUnit;Ljava/util/Set;Ljava/util/Set;Lorg/jboss/msc/service/StabilityStatistics;)Z
Target=2_90_20180208_378436 (Linux 2.6.32-696.el6.i686)
10:55:52,455 DEBUG [org.apache.mina.core.filterchain.IoFilterEvent] (pool-10-thread-6) Firing a MESSAGE_RECEIVED event for session 609
CPU=x86 (2 logical CPUs) (0xf5510000 RAM)
----------- Stack Backtrace -----------
(0x00235663 [libj9prt29.so+0x3b663])
(0x00218F6E [libj9prt29.so+0x1ef6e])
(0x002351CE [libj9prt29.so+0x3b1ce])
(0x002352C6 [libj9prt29.so+0x3b2c6])
(0x00234D93 [libj9prt29.so+0x3ad93])
(0x00218F6E [libj9prt29.so+0x1ef6e])
(0x00234D07 [libj9prt29.so+0x3ad07])
(0x00D7FCF1 [libj9vm29.so+0x81cf1])
(0x00218F6E [libj9prt29.so+0x1ef6e])
(0x00D7F9EE [libj9vm29.so+0x819ee])
(0x00D80A40 [libj9vm29.so+0x82a40])
(0x00218B6A [libj9prt29.so+0x1eb6a])
__kernel_rt_sigreturn+0x0 (0x00F6D40C)
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2018/03/23 10:55:52 - please wait.
JVMDUMP032I JVM requested System dump using '/home/hudson/hudson_workspace/workspace/eap-7x-ldap-kerberos-testsuite-apacheDS-rhel/853df230/tests-ldap-kerberos/eap7/core.20180323.105552.12943.0001.dmp' in response to an event

Additional info:
Newest IBM SDK 8.0.5.11 has some fixes in release notes that seems to be exactly the solution for this https://developer.ibm.com/javasdk/support/fixes/sdk8/#SR5FP11
particulary http://www-01.ibm.com/support/docview.wss?uid=swg1IJ05407
but iwe hit the issue even with 8.0.5.11

Comment 2 mchoma 2018-04-16 09:30:52 UTC
for the sake of completeness this is how error looks on SR5 FP11

11:07:48 + java -version
11:07:48 java version "1.8.0_161"
11:07:48 Java(TM) SE Runtime Environment (build 8.0.5.11 - pxi3280sr5fp11-20180326_01(SR5 FP11))
11:07:48 IBM J9 VM (build 2.9, JRE 1.8.0 Linux x86-32 20180309_380776 (JIT enabled, AOT enabled)
11:07:48 OpenJ9   - 49fcaf39
11:07:48 OMR      - 5cbbadf
11:07:48 IBM      - 4453dac)
11:07:48 JCL - 20180319_01 based on Oracle jdk8u161-b12


11:11:22 #0: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so(+0x6f94bb) [0x149b4bb]
11:11:22 #1: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so(+0x6fdbf0) [0x149fbf0]
11:11:22 #2: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so(+0x8b2f5) [0xe2d2f5]
11:11:22 #3: /home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9prt29.so(+0x1eb8f) [0xbdbb8f]
11:11:22 #4: [0xd6d40c]
11:11:22 Unhandled exception
11:11:22 Type=Segmentation error vmState=0x0005ff06
11:11:22 J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
11:11:22 Handler1=00456DEE Handler2=00BDB99E InaccessibleAddress=0000001C
11:11:22 EDI=94EB3E60 ESI=00000000 EAX=00000000 EBX=0172055C
11:11:22 ECX=000037A0 EDX=00000000
11:11:22 EIP=0111ED82 ES=007B DS=007B ESP=95E35AF0
11:11:22 EFlags=00010206 CS=0073 SS=007B EBP=94C04240
11:11:22 Module=/home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/ibm-java/jre/lib/i386/default/libj9jit29.so
11:11:22 Module_base_address=00DA2000
11:11:22 
11:11:22 Method_being_compiled=org/jboss/msc/service/StabilityMonitor.awaitStability(JLjava/util/concurrent/TimeUnit;Ljava/util/Set;Ljava/util/Set;Lorg/jboss/msc/service/StabilityStatistics;)Z
11:11:22 Target=2_90_20180309_380776 (Linux 2.6.32-696.el6.i686)
11:11:22 CPU=x86 (2 logical CPUs) (0xf5510000 RAM)
11:11:22 ----------- Stack Backtrace -----------
11:11:22 (0x00BF9563 [libj9prt29.so+0x3c563])
11:11:22 (0x00BDBFA8 [libj9prt29.so+0x1efa8])
11:11:22 (0x00BF90CE [libj9prt29.so+0x3c0ce])
11:11:22 (0x00BF91C6 [libj9prt29.so+0x3c1c6])
11:11:22 (0x00BF8C93 [libj9prt29.so+0x3bc93])
11:11:22 (0x00BDBFA8 [libj9prt29.so+0x1efa8])
11:11:22 (0x00BF8C07 [libj9prt29.so+0x3bc07])
11:11:22 (0x004560E8 [libj9vm29.so+0x7e0e8])
11:11:22 (0x00BDBFA8 [libj9prt29.so+0x1efa8])
11:11:22 (0x00455DDE [libj9vm29.so+0x7ddde])
11:11:22 (0x00456E41 [libj9vm29.so+0x7ee41])
11:11:22 (0x00BDBB8F [libj9prt29.so+0x1eb8f])
11:11:22 __kernel_rt_sigreturn+0x0 (0x00D6D40C)
11:11:22 ---------------------------------------
11:11:22 JVMDUMP039I Processing dump event "gpf", detail "" at 2018/04/12 11:10:38 - please wait.
11:11:22 JVMDUMP032I JVM requested System dump using '/home/hudson/hudson_workspace/workspace/early-testing-cli-embedded-unix-ibm-segmentation-fault/a9da8a54/embed/core.20180412.111038.7420.0001.dmp' in response to an event

Comment 3 IBM Bug Proxy 2018-04-18 02:41:10 UTC
------- Comment From chavez@us.ibm.com 2018-04-17 22:36 EDT-------
I have opened a ticket with Java L3. If a core is generated, can you run the jextract utility on it to collect all the binaries needed to analyze it? See http://www-01.ibm.com/support/docview.wss?uid=swg21222464

Comment 4 IBM Bug Proxy 2018-04-18 12:41:19 UTC
------- Comment From chavez@us.ibm.com 2018-04-18 08:40 EDT-------
Citing the specifics instructions for what IBM Java will need (jextracted coredump and corresponding javacore):

a) Set ulimits to avoid the truncated system core

ulimit -c unlimited
ulimit -f unlimited

b) Set the following JVM option:

-Xdump:java+system+snap:events=gpf+abort,request=exclusive+prepwalk+preempt,range=1..2

This will generate javacore, system core and snaptrace on gpf.

c) Please reproduce the issue. The above logs will be generated. Please jextract the coredump:

[path to JDK]/jre/bin/jextract [system dump file]

You can refer to the following links for Linux crash mustgather:

Setup: http://www-01.ibm.com/support/docview.wss?uid=swg21222461
Gather: http://www-01.ibm.com/support/docview.wss?uid=swg21222464

Comment 5 mchoma 2018-04-19 11:52:51 UTC
We are preparing required files. They have hundreds of MB. 
1. Is it expected?
2. How to technically deliver such amount of data to you?

Comment 6 IBM Bug Proxy 2018-04-19 14:11:45 UTC
------- Comment From chavez@us.ibm.com 2018-04-19 10:01 EDT-------
> We are preparing required files. They have hundreds of MB.
> 1. Is it expected?

I believe the jextract tool finds all the necessary binaries along with the core and other dump txt files and produces a single archive so it could be large but since it should be compressed I wouldn't expect it to be hundreds of MB but can't say for sure why it is in your case.

> 2. How to technically deliver such amount of data to you?

I think the easiest way is to use this site:

https://testcase.software.ibm.com/

login with "anonymous" as the id and navigate to /toibm/linux and select the file and upload as binary. Once uploaded, let me know what the filename was and I will transfer it over to the Java folks. Thanks.

Comment 7 mchoma 2018-04-19 14:57:52 UTC
I have uploaded core.20180419.112543.1468.0001.dmp.zip

Comment 8 IBM Bug Proxy 2018-04-19 16:43:16 UTC
------- Comment From hannsj_uhl@de.ibm.com 2018-04-19 12:26 EDT-------
.

Comment 9 IBM Bug Proxy 2018-04-19 19:11:29 UTC
------- Comment From chavez@us.ibm.com 2018-04-19 15:07 EDT-------
(In reply to comment #10)
> I have uploaded core.20180419.112543.1468.0001.dmp.zip

Thanks. I have downloaded the file from the testcase ftp server and am in the process of uploading it to the Java L3's internal ftp server and then will update their support ticket that it is there.

Comment 10 IBM Bug Proxy 2018-04-25 20:51:18 UTC
------- Comment From chavez@us.ibm.com 2018-04-25 16:41 EDT-------
IBM Java L3 runtime team has triaged and done a first analysis of the core provided and the ticket has been passed on to the JIT development team for further assistance.

Comment 12 mchoma 2018-07-02 08:45:41 UTC
Dear IBM, any update, we are still facing this issue?

Comment 13 mchoma 2018-08-24 08:34:17 UTC
any update?

Comment 15 Joseph Kachuck 2018-08-29 18:20:17 UTC
Hello,
RHEL 6 has entered Phase 3. In phase 3 only Critical impact Security Advisories and selected Urgent Priority Bug Fix Advisories will be accepted.
https://access.redhat.com/support/policy/updates/errata

If this fix is required for RHEL 6. Please also provide a justification for this fix.

Thank You
Joe Kachuck

Comment 16 IBM Bug Proxy 2018-08-31 19:10:24 UTC
------- Comment From chavez@us.ibm.com 2018-08-31 15:09 EDT-------
From Java L3 today (I replied asking for a timeframe for the release of the update containing the fix):

"Just a quick update to say that the development team has identified the cause of the crash and is working on a fix for the problem."

Comment 18 IBM Bug Proxy 2018-09-26 20:20:22 UTC
------- Comment From chavez@us.ibm.com 2018-09-26 16:16 EDT-------
APAR IJ09089 has been created to document this problem and should be visible in the next day or two. The fix has been scheduled for release in Java 8 SR5 FP22

Comment 19 IBM Bug Proxy 2018-10-08 12:55:49 UTC
------- Comment From hannsj_uhl@de.ibm.com 2018-10-08 08:22 EDT-------
(In reply to comment #17)
> APAR IJ09089 has been created to document this problem and should be visible
> in the next day or two. The fix has been scheduled for release in Java 8 SR5
> FP22
.
... which is available since 09/27/2018 from
https://developer.ibm.com/javasdk/downloads/sdk8/
with the fixes included as outlined at
https://developer.ibm.com/javasdk/support/fixes/sdk8/#SR5FP22 ...

Comment 20 Peter Mackay 2018-10-16 08:16:46 UTC
We've executed a few test runs using the FP22 and didn't see the issue anymore. Looks like it is fixed. Thanks!

Comment 21 Hanns-Joachim Uhl 2018-11-26 10:28:40 UTC
fyi ... with IBM Java 8 SR5 FP25 available for RHEL6 from RHN since 11/08/2018
(see https://access.redhat.com/errata/RHSA-2018:3533 ...)
and per the previous comment I am closing this Red Hat bugzilla now ...
... please correct me if I am wrong ...


Note You need to log in before you can comment on or make changes to this bug.