Bug 1018628 - RHEL7 ipa-server-install abrt on java in tomcat
RHEL7 ipa-server-install abrt on java in tomcat
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pki-core (Show other bugs)
7.0
Unspecified Unspecified
medium Severity unspecified
: rc
: ---
Assigned To: Endi Sukma Dewata
Asha Akkiangady
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-13 19:51 EDT by Scott Poore
Modified: 2014-06-18 03:18 EDT (History)
6 users (show)

See Also:
Fixed In Version: pki-core-10.0.5-2.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 05:49:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Scott Poore 2013-10-13 19:51:02 EDT
Description of problem:
We're seeing abrt crashes during ipa-server-install.  They don't appear so far to be affecting the install but, thought I'd get another opinion.

Version-Release number of selected component (if applicable):

pkg_name:       tomcat
pkg_release:    2.el7
pkg_version:    7.0.40

How reproducible:
Unknown but, has occurred several times so far on test runs.


Steps to Reproduce:
1.  /usr/sbin/ipa-server-install --setup-dns --no-forwarder -p Secret123 -P Secret123 -a Secret123 -r TESTRELM.COM -n testrelm.com --ip-address=$MASTER_IP --hostname=hp-z600-02.testrelm.com -U


Actual results:
1. see abrt crash but server appears installed.

Expected results:
2. no crash


Additional info:

abrt emailed report:

abrt_version:   2.1.7
cmdline:        java -agentpath:/usr/lib64/libabrt-java-connector.so=abrt=on -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
executable:     /usr/share/tomcat/bin/bootstrap.jar
java_executable: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/bin/java-abrt
kernel:         3.10.0-33.el7.x86_64
last_occurrence: 1381706256
pid:            7135
pkg_arch:       noarch
pkg_epoch:      0
pkg_name:       tomcat
pkg_release:    2.el7
pkg_version:    7.0.40
runlevel:       N 3
time:           Sun 13 Oct 2013 07:17:36 PM EDT
uid:            0
username:       root

backtrace:
:Exception in thread "Thread-2" java.util.MissingResourceException: Can't find bundle for base name LogMessages, locale en_US
:        at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1499) [jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/rt.jar!/java/util/ResourceBundle.class]
:        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1322) [jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/rt.jar!/java/util/ResourceBundle.class]
:        at java.util.ResourceBundle.getBundle(ResourceBundle.java:721) [jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/rt.jar!/java/util/ResourceBundle.class]
:        at com.netscape.cmscore.apps.CMSEngine.getLogMessage(CMSEngine.java:1217) [unknown]
:        at com.netscape.cmscore.apps.CMSEngine.getLogMessage(CMSEngine.java:1297) [unknown]
:        at com.netscape.certsrv.apps.CMS.getLogMessage(CMS.java:700) [unknown]
:        at com.netscape.cms.logging.LogFile.shutdown(LogFile.java:866) [unknown]
:        at com.netscape.cms.logging.RollingLogFile.shutdown(RollingLogFile.java:144) [unknown]
:        at com.netscape.cmscore.logging.LogQueue.shutdown(LogQueue.java:67) [unknown]
:        at com.netscape.cmscore.logging.LogSubsystem.shutdown(LogSubsystem.java:174) [unknown]
:        at com.netscape.cmscore.apps.CMSEngine.shutdownSubsystems(CMSEngine.java:1692) [unknown]
:        at com.netscape.cmscore.apps.CMSEngine.shutdown(CMSEngine.java:1639) [unknown]
:        at com.netscape.cmscore.apps.CMSEngine$1.run(CMSEngine.java:278) [unknown]

environ:
KI_INSTANCE_PATH=/var/lib/pki/pki-tomcat
KI_UNSECURE_PORT=8080
KI_GROUP=pkiuser
KI_LOCKFILE=/var/lock/pki/tomcat/pki-tomcat
KI_PIDFILE=/var/run/pki/tomcat/pki-tomcat.pid
:CATALINA_BASE=/var/lib/pki/pki-tomcat
:NAME=pki-tomcat
KI_LOCKDIR=/var/lock/pki/tomcat
KI_PIDDIR=/var/run/pki/tomcat
KI_INSTANCE_INITSCRIPT=/var/lib/pki/pki-tomcat/pki-tomcat
:TOMCAT_GROUP=pkiuser
ATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
:TOMCAT_USER=pkiuser
:_=/usr/lib/jvm/jre/bin/java
WD=/usr/share/tomcat
:LANG=en_US.UTF-8
:TOMCAT_PIDFILE=/var/run/pki/tomcat/pki-tomcat.pid
:SERVICE_NAME=pki-tomcat
:RESTART_SERVER=/var/lib/pki/pki-tomcat/conf/restart_server_after_configuration
KI_USER=pkiuser
:TOMCAT_PROG=pki-tomcat
:SHLVL=3
:CATALINA_PID=/var/run/pki/tomcat/pki-tomcat.pid
KI_WEB_SERVER_TYPE=tomcat
:TOMCAT_LOCKFILE=/var/lock/subsys/pki-tomcat
KI_INSTANCE_ID=pki-tomcat

jvm_environment:
:sun.java.command              : org.apache.catalina.startup.Bootstrap start
:sun.java.launcher             : SUN_STANDARD
:java.home                     : /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre
:java.class.path               : /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
:java.library.path             : /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
:sun.boot.class.path           : /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/netx.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/plugin.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/classes
:sun.boot.library.path         : /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/amd64
:java.ext.dirs                 : /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/lib/ext:/usr/java/packages/lib/ext
:java.endorsed.dirs            :
:cwd                           : /usr/share/tomcat
:java.vm.version               : 24.0-b56
:java.vm.name                  : OpenJDK 64-Bit Server VM
:java.vm.info                  : mixed mode, sharing
:java.vm.vendor                : Oracle Corporation
:java.vm.specification_name    : Java Virtual Machine Specification
:java.vm.specification.vendor  : Oracle Corporation
:java.vm.specification.version : 1.7
Comment 2 Martin Kosek 2013-10-14 03:29:40 EDT
Adding folks from pki-ca to CC to advise what this could mean.
Comment 3 Scott Poore 2013-10-14 14:53:36 EDT
Also, I see this AVC denial during the crash:

type=PATH msg=audit(1381763445.068:607): item=1 name="abrt_checker_7013.log" objtype=CREATE
type=PATH msg=audit(1381763445.068:607): item=0 name="/usr/share/tomcat" inode=68508982 dev=fd:01 mode=040775 ouid=0 ogid=91 rdev=00:00 obj=system_u:object_r:usr_t:s0 objtype=PARENT
type=CWD msg=audit(1381763445.068:607):  cwd="/usr/share/tomcat"
type=SYSCALL msg=audit(1381763445.068:607): arch=c000003e syscall=2 success=no exit=-13 a0=7fd68b7e2200 a1=241 a2=1b6 a3=0 items=2 ppid=1 pid=7651 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="java-abrt" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.40-2.4.2.6.el7.x86_64/jre/bin/java-abrt" subj=system_u:system_r:pki_tomcat_t:s0 key=(null)
type=AVC msg=audit(1381763445.068:607): avc:  denied  { write } for  pid=7651 comm="java-abrt" name="tomcat" dev="dm-1" ino=68508982 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=dir
Comment 4 Nathan Kinder 2013-10-14 19:27:44 EDT
It looks like the crash is happening at shutdown, which is probably why you don't see an actual problem with the server running properly.  We are failing to get the "LogMessages" resource bundle, which should be located in a LogMessages.properties file.  I see this file in the pki source tree, but I'm not sure where it gets installed.  It didn't seem to be a part of pki-base, pki-server, or pki-ca.

The AVC just seems like it's unable to write out the abrt info from the crash.
Comment 5 Martin Kosek 2013-10-15 07:07:52 EDT
Moving to pki-core component based on Comment 4.
Comment 6 Nathan Kinder 2013-10-28 11:27:42 EDT
Upstream ticket:
https://fedorahosted.org/pki/ticket/775
Comment 7 Endi Sukma Dewata 2013-11-01 15:25:25 EDT
The resource bundles are packaged in pki-cmsbundle.jar which is distributed by pki-server. The issue was actually caused by multiple invocations of CMS.shutdown() due to incorrect placement of shutdown hook.

Fixed upstream.
Comment 9 Scott Poore 2013-12-09 16:05:06 EST
Verified.

I have not seen another abrt report email come through since this was fixed.  Latest version checked:

pki-ca-10.0.5-2.el7.noarch

[root@ipaqa64vmg abrt]# abrt-cli list

[root@ipaqa64vmg abrt]# 

[root@ipaqa64vmg ~]# rpm -q pki-ca
pki-ca-10.0.5-2.el7.noarch

[root@ipaqa64vmg ~]# cd /var/tmp/abrt/

[root@ipaqa64vmg abrt]# ls

[root@ipaqa64vmg abrt]# cd /var/run/abrt

[root@ipaqa64vmg abrt]# ls
abrtd.pid  abrt.socket  saved_core_pattern

[root@ipaqa64vmg abrt]# grep abrt /var/log/messages 
Dec  9 15:40:43 ipaqa64vmg yum[12400]: Installed: abrt-libs-2.1.7-5.el7.x86_64
Dec  9 15:40:44 ipaqa64vmg yum[12400]: Installed: abrt-2.1.7-5.el7.x86_64
Dec  9 15:40:44 ipaqa64vmg yum[12400]: Installed: abrt-tui-2.1.7-5.el7.x86_64
Dec  9 15:40:51 ipaqa64vmg yum[12524]: Installed: abrt-addon-ccpp-2.1.7-5.el7.x86_64
Dec  9 15:40:55 ipaqa64vmg abrtd: Searching for unprocessed dump directories
Dec  9 15:40:55 ipaqa64vmg abrtd: Init complete, entering main loop
[root@ipaqa64vmg abrt]# grep crash /var/log/messages 
Dec  9 15:36:42 ipaqa64vmg kernel: [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-54.el7.x86_64 root=/dev/mapper/rhel_ipaqa64vmg-root ro vconsole.font=latarcyrheb-sun16 rd.lvm.lv=rhel_ipaqa64vmg/swap vconsole.keymap=us rd.lvm.lv=rhel_ipaqa64vmg/root crashkernel=auto rhgb quiet LANG=en_US.UTF-8
Dec  9 15:36:42 ipaqa64vmg kernel: [    0.000000] Reserving 129MB of memory at 736MB for crashkernel (System RAM: 2047MB)
Dec  9 15:36:42 ipaqa64vmg kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-54.el7.x86_64 root=/dev/mapper/rhel_ipaqa64vmg-root ro vconsole.font=latarcyrheb-sun16 rd.lvm.lv=rhel_ipaqa64vmg/swap vconsole.keymap=us rd.lvm.lv=rhel_ipaqa64vmg/root crashkernel=auto rhgb quiet LANG=en_US.UTF-8
Dec  9 15:36:42 ipaqa64vmg kernel: [    0.593152] crash memory driver: version 1.1
[root@ipaqa64vmg abrt]# grep panic /var/log/messages 
Dec  9 15:36:42 ipaqa64vmg kernel: [    1.813681] qxl 0000:00:02.0: registered panic notifier
Comment 10 Ludek Smid 2014-06-13 05:49:58 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.