Red Hat Bugzilla – Bug 475999
Mint not shutdown by initscript
Last modified: 2013-09-12 18:10:14 EDT
After starting cumin, the mint-server gets started as well.
But when I stop it, the mint does not get stopped as it should.
version used: cumin-0.1.2968-1.el5
1. install cumin (incl. the posgresql settings, database-init, add-user)
2. /etc/init.d/cumin start
3. pgrep mint-server && echo mint is running
4. /etc/init.d/cumin stop
5. pgrep mint-server && echo mint is still running
I expect the initscript to shutdown mint-server along with cumin.
My theory, based on some of the instances where I've seen mint-server fail to exit, is that this occurs because of http://bugzilla.redhat.com/show_bug.cgi?id=476038, db deadlocks that can freeze mint threads.
The bug is still valid in cumin-0.1.2986-1.el5 so I doubt it is too
much connected to 476038.
After having looked into sources in svn trunk it is clear
that cumin sends SIGTERM to process mint-server it
previously started via Popen in 'start_mint_processes'
of 'trunk/cumin/python/cumin/tools.py' file.
I am wondering whether the mint-server is still running on
'pop.pid' PID which the above-mentioned function returns
or it gets restarted meanwhile, having another process id...
If I manually kill mint-server with SIGTERM right after
'/etc/init.d/cumin stop', it ends happily, so I do not
suspect it is hanging:
# /etc/init.d/cumin stop; pkill mint-server
I will continue to examine it deeper next week.
In change 3001, I have added some logic to more carefully kill the mint process and check for confirmation.
Still valid on cumin-0.1.3021-1.el5
After stopping cumin via '/etc/init.d/cumin stop',
the mint process starts eating all the CPU.
Jan, just to make sure: did you also reinstall the schema from scratch? Ie, use cumin-database-destroy, then cumin-database-init?
Sure I did.
Again, here is what I have just done (on rev 3030):
#(postgresql set up and running)
yum -y remove cumin
yum -y install cumin
cumin-admin add-user test
# log in, possibly add local broker
# if qpidd is running, log out
pgrep mint-server # running
pgrep mint-server # still running
# though it's not eating all the CPU
# as in previous version
pgrep mint-server # not running anymore
BTW I am testin it on my local RHEL-5.2 running in chroot
(not jailed via vserver or anything like it).
The fact that it's no longer eating all the CPU is a very positive development. I'm now comfortable (a) release noting this for 1.1 and (b) extending the time over which we attempt to kill the mint-server subprocess in the cumin process.
I extended the wait time in change 3036.
Reopening for 1.1.1 so we make sure to flush out any still-hidden issues.
Fixed at revision 3068: added a handler for SIG_TERM that catches the signal from the initscript (it's doing "killproc $servicename -TERM") and shuts down the mint process properly.
Created attachment 329746 [details]
Locally built RPM for testing fix
You can use this temporary RPM to test while the fix does not show up in the candidates repo.
Verified on RHEL5.3 i386. Thanks for fixing it!
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.