Bug 1464064

Created attachment 1290651 [details]
Error file generated by fatal SIGBUS error in JRE

Description of problem:

After upgrading the Linux kernel from version 3.10.0-514.21.1.el7.x86_64 to version 3.10.0-514.21.2.el7.x86_64 (and rebooting with the new kernel), tomcat was not able to start under JSVC. 

This kernel update (RHSA-2017:1484) was a security update related to the "Stack Guard" security issue.

The problem ONLY occurs when tomcat is configured to start using jsvc, by having the option 
USE_JSVC="true" 
set in the file /etc/tomcat/tomcat.conf .

Java failed to start due to a fatal SIGBUS error. Attached is an error file hs_err_pid15106.log generated when the failure occurred. (Many similar files were generated, one per start attempt).

Rebooting to the prior kernel (3.10.0-514.21.1.el7.x86_64) allowed tomcat to start up again via jsrv.

After some work, I determined that commenting out USE_JSVC="true" in /etc/tomcat/tomcat.conf also allowed tomcat to start when the new kernel (3.10.0-514.21.2.el7.x86_64) is running.


Version-Release number of selected component (if applicable):


How reproducible:
Fully. The problem occurred on 2 similarly configured machines. On the development machine I was able to determine that either using the prior kernel or disabling the USE_JSVC="true" option in /etc/tomcat/tomcat.conf allow tomcat to start. A number of starts/stops confirmed that the USE_JSVC is was the ONLY change needed to trigger the problem under the new kernel (or avoid it).


Steps to Reproduce:
1. Boot with updated linux kernel version 3.10.0-514.21.2.el7.x86_64 (from RHSA-2017:1484).
2. Make sure USE_JSVC="true" is set/enabled in /etc/tomcat/tomcat.conf.
3. Make sure apache-commons-daemon-jsvc is installed, and that /bin/jsvc exists.
4. Try to start tomcat, ex by: systemctl start tomcat.service

Actual results:
Tomcat fails to start fully. Webapps are not deployed.

A single "jsvc.exec" process seems to be running, but the tomcat deployed webaoos are not available.

Attempting to reach the webapp (on Apache via mod_jk) shows:

"Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later."

The tomcat log file: /var/log/tomcat/catalina.out reports:
"A fatal error has been detected by the Java Runtime Environment
...
SIGBUS (0x7) ..."
and refer to detailed error file created as "/tmp/hs_err_pidnnnn.log". (A sample is attached.)

However, "systemctl status tomcat.service" reports the tomcat service as active:

[root@mathnet2 ~]# uname -a 
Linux mathnet2.technion.ac.il 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@mathnet2 ~]# systemctl status tomcat.service 
* tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/tomcat.service.d
           `-depend-httpd.conf, depend-mariadb.conf, post-tomcat-start.conf
   Active: active (running) since Thu 2017-06-22 13:46:41 IDT; 37s ago
  Process: 5676 ExecStartPost=/usr/local/bin/tomcat-start-run-curl.sh (code=exited, status=0/SUCCESS)
 Main PID: 5675 (jsvc)
   CGroup: /system.slice/tomcat.service
           `-5675 jsvc.exec -nodetach -pidfile /var/log/tomcat/jsvc-tomcat.pid -user tomcat -outfile /usr/sh...
...


Expected results:
Tomcat starts properly. Webapps are available.

I always see TWO "jsvc.exec" processes running after the successful startup, and the webapp tomcat serves is available, as noted below.

Additional info:

As mentioned above, commenting out USE_JSVC="true" in /etc/tomcat/tomcat.conf allows tomcat to start. In that case:

[root@mathnet2 ~]# systemctl status tomcat.service 
* tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/tomcat.service.d
           `-depend-httpd.conf, depend-mariadb.conf, post-tomcat-start.conf
   Active: active (running) since Thu 2017-06-22 13:54:40 IDT; 5min ago
  Process: 5807 ExecStartPost=/usr/local/bin/tomcat-start-run-curl.sh (code=exited, status=0/SUCCESS)
 Main PID: 5806 (java)
   CGroup: /system.slice/tomcat.service
           |-5806 /usr/lib/jvm/jre/bin/java -Xms125m -Xmx700m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -ve...
...


When the server is running on the PRIOR kernel, tomcat starts properly, even when USE_JSVC="true" is set. Data below taken from a parallel server with the old kernel and tomcat running under jsrv:

[root@mathnet ~]# uname -a
Linux mathnet.technion.ac.il 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@mathnet ~]# systemctl status tomcat.service 
* tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/tomcat.service.d
           `-depend-httpd.conf, depend-mariadb.conf, limits.conf, post-tomcat-start.conf
   Active: active (running) since Tue 2017-06-20 11:35:21 IDT; 2 days ago
  Process: 2788 ExecStartPost=/usr/local/bin/tomcat-start-run-curl.sh (code=exited, status=0/SUCCESS)
 Main PID: 2787 (jsvc)
   CGroup: /system.slice/tomcat.service
           |-  2787 jsvc.exec -nodetach -pidfile /var/log/tomcat/jsvc-tomcat.pid -user tomcat -outfile /usr/share/tomc...
           |-  2812 jsvc.exec -nodetach -pidfile /var/log/tomcat/jsvc-tomcat.pid -user tomcat -outfile /usr/share/tomc...
...