Bug 475999 - Mint not shutdown by initscript
Mint not shutdown by initscript
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: cumin (Show other bugs)
All Linux
medium Severity high
: 1.1.1
: ---
Assigned To: Nuno Santos
Jan Sarenik
Depends On:
  Show dependency treegraph
Reported: 2008-12-11 09:29 EST by Jan Sarenik
Modified: 2013-09-12 18:10 EDT (History)
3 users (show)

See Also:
Fixed In Version: cumin-0.1.3073-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-04-21 12:18:58 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Locally built RPM for testing fix (608.99 KB, application/x-rpm)
2009-01-22 17:31 EST, Nuno Santos
no flags Details

  None (edit)
Description Jan Sarenik 2008-12-11 09:29:13 EST
After starting cumin, the mint-server gets started as well.
But when I stop it, the mint does not get stopped as it should.

version used: cumin-0.1.2968-1.el5

100% reproducible:
1. install cumin (incl. the posgresql settings, database-init, add-user)
2. /etc/init.d/cumin start
3. pgrep mint-server && echo mint is running
4. /etc/init.d/cumin stop
5. pgrep mint-server && echo mint is still running

I expect the initscript to shutdown mint-server along with cumin.
Comment 1 Justin Ross 2008-12-11 11:40:56 EST
My theory, based on some of the instances where I've seen mint-server fail to exit, is that this occurs because of http://bugzilla.redhat.com/show_bug.cgi?id=476038, db deadlocks that can freeze mint threads.
Comment 2 Jan Sarenik 2008-12-12 04:33:41 EST
The bug is still valid in cumin-0.1.2986-1.el5 so I doubt it is too
much connected to 476038.
Comment 3 Jan Sarenik 2008-12-12 09:19:22 EST
After having looked into sources in svn trunk it is clear
that cumin sends SIGTERM to process mint-server it
previously started via Popen in 'start_mint_processes'
of 'trunk/cumin/python/cumin/tools.py' file.

I am wondering whether the mint-server is still running on
'pop.pid' PID which the above-mentioned function returns
or it gets restarted meanwhile, having another process id...

If I manually kill mint-server with SIGTERM right after
'/etc/init.d/cumin stop', it ends happily, so I do not
suspect it is hanging:

# /etc/init.d/cumin stop; pkill mint-server

I will continue to examine it deeper next week.
Comment 4 Justin Ross 2008-12-15 13:52:50 EST
In change 3001, I have added some logic to more carefully kill the mint process and check for confirmation.
Comment 5 Jan Sarenik 2008-12-22 04:17:29 EST
Still valid on cumin-0.1.3021-1.el5

After stopping cumin via '/etc/init.d/cumin stop',
the mint process starts eating all the CPU.
Comment 6 Justin Ross 2009-01-07 13:09:16 EST
Jan, just to make sure: did you also reinstall the schema from scratch?  Ie, use cumin-database-destroy, then cumin-database-init?
Comment 7 Jan Sarenik 2009-01-08 05:46:08 EST
Sure I did.

Again, here is what I have just done (on rev 3030):

   #(postgresql set up and running)
   yum -y remove cumin
   yum -y install cumin
   cumin-admin add-user test
   /etc/init.d/cumin start
   firefox http://localhost:45672/
     # log in, possibly add local broker
     # if qpidd is running, log out
   /etc/init.d/cumin stop
   pgrep mint-server # running
   sleep 20
   pgrep mint-server # still running
     # though it's not eating all the CPU
     # as in previous version
   pkill mint-server
   pgrep mint-server # not running anymore

BTW I am testin it on my local RHEL-5.2 running in chroot
(not jailed via vserver or anything like it).
Comment 8 Justin Ross 2009-01-08 10:55:38 EST
The fact that it's no longer eating all the CPU is a very positive development.  I'm now comfortable (a) release noting this for 1.1 and (b) extending the time over which we attempt to kill the mint-server subprocess in the cumin process.

I extended the wait time in change 3036.
Comment 9 Justin Ross 2009-01-14 13:29:42 EST
Reopening for 1.1.1 so we make sure to flush out any still-hidden issues.
Comment 10 Nuno Santos 2009-01-22 15:16:36 EST
Fixed at revision 3068: added a handler for SIG_TERM that catches the signal from the initscript (it's doing "killproc $servicename -TERM") and shuts down the mint process properly.
Comment 11 Nuno Santos 2009-01-22 17:31:48 EST
Created attachment 329746 [details]
Locally built RPM for testing fix

You can use this temporary RPM to test while the fix does not show up in the candidates repo.
Comment 12 Jan Sarenik 2009-01-29 03:41:41 EST
Verified on RHEL5.3 i386. Thanks for fixing it!
Comment 14 errata-xmlrpc 2009-04-21 12:18:58 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.