Bug 467717

+++ This bug was initially created as a clone of Bug #462839 +++

Description of problem:
-----------------------

osa-dispatcher, rhn-database, jabberd, and tomcat5 are started multiple times or stopped multiple times via rc scripts during run-level changes.  This causes the system to send out numerous web tracebacks during a system shutdown, and the web service to constantly return a service temporarily unavailable page after the system has been started again. 

All of these services install their own rc scripts.  The rhn-satellite rc script calls all of their rc scripts directly.  During a run-level change, rc will call each of these scripts and the rhn-satellite script will call them a second time.

The problem with shutdown is that rc calls the rhn-database script to shutdown the database before rhn-satellite is called.  Shutting down the database while rhn-satellite is still trying to use it causes web tracebacks to be generated.

Even if we use chkconfig to turn these services off [ for example rhn-database ] , the tracebacks would be generated as in each rcX.d there is a link to kill the process

[root@dhcp7-119 ~]# chkconfig --list rhn-database
rhn-database   	0:off	1:off	2:off	3:off	4:off	5:off	6:off
[root@dhcp7-119 ~]# ls /etc/rc*.d/ | grep rhn-database
K25rhn-database
K25rhn-database
K25rhn-database
K25rhn-database
K25rhn-database
K25rhn-database
K25rhn-database
[root@dhcp7-119 ~]# 

How reproducible:
-----------------
Always while runlevel is changed

Steps to Reproduce:
-------------------
In Satellite 5.1.1 just issue commands like "init 4", "init 5", "init 3", etc to switch run levels. 

Actual results:
---------------

The traceback would be generated to the email account we have set. Tracebacks have informations like the following : [ Few traceback messages upon syste reboot attached in file traceback-on-runlevel-changes.txt ]

"Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!"

"Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source."

"Caused by: java.sql.SQLException: No more data to read from socket"

"Caused by: java.sql.SQLException: ORA-01089: immediate shutdown in progress - no operations are permitted "

Expected results:
-----------------

There should not be any tracebacks, and runlevel switches should be proper.

Additional info:
----------------

Only workaround is to delete the chkconfig scripts [ or exclude these services from the init script of rhn-satellite ]. That is a chkconfig --del command for each of these components to remove all symlinks from the rc?.d directories, letting rhn-satellite manage their starting and stopping.  This will have to be done every time updates for these packages are installed; the postinstall scripts for these packages execute chkconfig --add for their rc scripts regardless of whether the package is being installed or updated.

	# chkconfig --del tomcat5
	# chkconfig --del rhn-database
	# chkconfig --del jabberd
	# chkconfig --del osa-dispatcher


--- Additional comment ---

Hi,

I checked into the default states of all these init scripts when the
package is installed and had set it to the default settings. 

# chkconfig --list | egrep
"rhn-satellite|rhn-database|tomcat5|osa-dispatcher|jabberd|httpd"
jabberd         0:off   1:off   2:on    3:on    4:on    5:on    6:off
httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
rhn-database    0:off   1:off   2:on    3:on    4:on    5:on    6:off
osa-dispatcher  0:off   1:off   2:on    3:on    4:on    5:on    6:off
tomcat5         0:off   1:off   2:off   3:off   4:off   5:off   6:off
rhn-satellite   0:off   1:off   2:off   3:on    4:on    5:on    6:off
#

After this I tried switching runlevels between 3,4,5 and didn't get a
traceback. But the tracebacks are generated when we switch to runlevel 6.
A total of 7 tracebacks are generated when I switched to runlevel 6. I
doubt whether it is because services which are already stopped are again
attempted to be stopped/ killed.