RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1552040 - Sometimes Wildfly 12 process does not exit on RHEL 6 (32/64 bit) and IBM JDK 8
Summary: Sometimes Wildfly 12 process does not exit on RHEL 6 (32/64 bit) and IBM JDK 8
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: java-1.8.0-ibm
Version: 6.9
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 6.10
Assignee: jiri vanek
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks: 1414846 1425546
TreeView+ depends on / blocked
 
Reported: 2018-03-06 11:54 UTC by Miroslav Novak
Modified: 2018-04-05 07:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-05 07:03:54 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
pstack.txt (29.74 KB, text/plain)
2018-03-06 11:54 UTC, Miroslav Novak 		no flags Details
javacore dump (2.47 MB, text/plain)
2018-03-20 16:42 UTC, Miroslav Novak 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 165393 0 None None None 2018-03-06 16:04:44 UTC
Red Hat Issue Tracker WFLY-9943 0 Major Closed Sometimes WF process does not exit on RHEL 6 (32/64 bit) and IBM JDK 8 2018-11-05 14:11:49 UTC

Description Miroslav Novak 2018-03-06 11:54:47 UTC 
Created attachment 1404770 [details]
pstack.txt

Description of problem:
Sometimes happens that WF12 process does not exit when :shutdown CLI operation is called and hangs indefinitely. This happens only RHEL 6 (both 32 and 64 bit arch) with IBM SDK8.

Calling kill -3 does not create any javacore dump. Only output could be gathered from pstack command (attached pstack.txt). It indicates that JVM shutdown thread is hanging on:
Thread 25 (Thread 0xb7792b70 (LWP 25507)):
#0  0x00bd4424 in __kernel_vsyscall ()
#1  0x00b2243c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x004f8b1f in monitor_wait_original () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9thr29.so
#3  0x004f9957 in omrthread_monitor_wait () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9thr29.so
#4  0x0033e217 in protectedDestroyJavaVM () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9vm29.so
#5  0x0042cf6e in omrsig_protect () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9prt29.so
#6  0x0033db93 in DestroyJavaVM () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9vm29.so
#7  0x00815999 in DestroyJavaVM () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libjvm.so
#8  0x00528649 in JavaMain () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/bin/../lib/i386/jli/libjli.so
#9  0x00b1ebc9 in start_thread () from /lib/libpthread.so.0
#10 0x001fc04e in clone () from /lib/libc.so.6

There are other 24 threads in pstack.txt which might be blocking this thread. 

Version-Release number of selected component (if applicable):

- Red Hat Enterprise Linux Server release 6.9 (Santiago):
$ uname -a
Linux rhel6-medium-7888.localdomain 2.6.32-696.el6.x86_64 #1 SMP Tue Feb 21 00:53:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
$ /lib/libc.so.6
lib/   lib64/ 
[hudson@messaging-10 ~]$ /lib64/libc.so.6
GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.8.5 20150623 (Red Hat 4.8.5-16).
Compiled on a Linux 3.10.0 system on 2017-06-16.
Available extensions:
	The C stubs add-on version 2.1.2.
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B
	RT using linux kernel aio
libc ABIs: UNIQUE IFUNC

- IBM SDK 8.0.5.10:
java version "1.8.0_161
Java(TM) SE Runtime Environment (build 8.0.5.10 - pxi3280sr5fp10-20180214_01(SR5 FP10))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux x86-32 20180208_378436 (JIT enabled, AOT enabled)
OpenJ9   - 39bb844
OMR      - c04ccb2
IBM      - 2321a81)
JCL - 20180209_01 based on Oracle jdk8u161-b12


How reproducible:
This issue is hard to reproduce. It happens in less than 1% of cases.

Steps to Reproduce:
1. Download Widfly 12 from http://download.jboss.org/wildfly/12.0.0.Final/wildfly-12.0.0.Final.zip
2. Unzip it and start using standalone-full-ha.xml config:
- unzip wildfly-12.0.0.Final.zip
- cd ./wildfly-12.0.0.Final/bin
- sh standalone.sh -c standalone-full-ha.xml
3. In another console invoke :shutdown command in CLI
- cd ./wildfly-12.0.0.Final/bin
- sh jboss-cli.sh -c
	-- [standalone@localhost:9990 /] :shutdown

Actual results:
Sometimes happens that WF12 process does not exit and hangs indefinitely. (Must be killed by kill -9 ... to exit)

Expected results:
WF12 process exits with 0 exit code.
Comment 2 IBM Bug Proxy 2018-03-06 21:41:03 UTC 
------- Comment From chavez.com 2018-03-06 16:39 EDT-------
Raised problem ticket with Java L3 containing current description.
Comment 3 IBM Bug Proxy 2018-03-07 13:11:22 UTC 
------- Comment From chavez.com 2018-03-07 08:01 EDT-------
Response from Java L3:

I have been trying to reproduce the hang - but not successful so far. The application shuts down properly everytime.

From the native threads - my assumption is that the issue may turn out to be something to do with the OS. Almost all the native threads have the following stacktop:

#0  0x00bd4424 in __kernel_vsyscall ()
#1  0x00b2243c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

Perhaps the OS is not bringing these threads out of the cond wait. The Java VM has already progressed to shut down - and the signal handler services maybe already off - because of which the logs are not generated.

However, coredump from the hang will help us to find more details. Could you please check if you can generate a coredump using gcore? You may follow these steps:

1. Keep the ulimits set to unlimited:

# ulimit -c unlimited
# ulimit -f unlimited

2. Whenever you see the hang during the shutdown, use gcore on the JVM pid:

# gcore <java_pid>

Hopefully this will generate a coredump from the hang.

3. Jextract the coredump:

[path to JDK]/jre/bin/jextract [coredump]

This should create a zip file - containing the coredump and the related libraries. Please upload this for us. Thanks.
Comment 4 IBM Bug Proxy 2018-03-20 16:01:25 UTC 
------- Comment From chavez.com 2018-03-20 11:51 EDT-------
Any update? Java L3 is considering closing the problem ticket on March 23 if nothing is received by then. Thanks.
Comment 5 Miroslav Novak 2018-03-20 16:41:29 UTC 
Sorry guys for late response. I was assigned to another task. I understand that this might issue on RedHat site as well. 

I took the whole day of start/stop WF12 until issue could be hit again. I also figured out that somehow our testing framework catched kill -3 signal so I needed to do it without it. 

I'm attaching java core dump. I did not manage to take a look at it.
Comment 6 Miroslav Novak 2018-03-20 16:42:08 UTC 
Created attachment 1410618 [details]
javacore dump
Comment 7 IBM Bug Proxy 2018-03-20 17:21:17 UTC 
------- Comment From chavez.com 2018-03-20 13:18 EDT-------
Understand but do you have an actual core dump (not just the javacore text file) that you can run jextract on so it can create an archive with the core and all the binaries and libraries necessary to analyze it as previously requested? Let me know. Thanks.
Comment 8 Joseph Kachuck 2018-03-22 15:21:27 UTC 
Hello IBM\Miroslav
Please note at this point in the RHEL 6.10 release. 
Without an exception or blocker request this will not be able to make RHEL 6.10.

If this will be requested for this BZ. Please confirm what a client would see in the field? What it would mean if this was not fixed in RHEL 6.10.

Miroslav, Please supply the logs ASAP if this is required to be fixed in RHEL 6.10.

Thank You
Joe Kachuck
Comment 9 Miroslav Novak 2018-03-26 06:41:45 UTC 
Sorry guys, I've kept it running in cycles for last 4 days but still was not successful to reproduce it again. I'll keep it spinning on more machines to increase chances to hit it.

Thanks Joseph for update. I understand that we'll not meet RHEL 6.10 window if this is not investigated in time.
Comment 10 Miroslav Novak 2018-04-04 07:00:07 UTC 
I've kept the test spinning on multiple machines (IBM JDK 8...10, RHEL 6 32/64bit) the whole week but could not reproduce the issue. There were thousands of runs.

It might be that issue is gone or it was some env issue. I would suggest to close this bz/ticket for now and re-open if this occur again.
Comment 11 IBM Bug Proxy 2018-04-04 17:31:37 UTC 
------- Comment From chavez.com 2018-04-04 13:21 EDT-------
OK. Thanks for the update.
Comment 12 Miroslav Novak 2018-04-05 07:03:54 UTC 
Closing.
Note You need to log in before you can comment on or make changes to this bug.