Cumin package contains following: ----------------------------------------------- preuninstall scriptlet (using /bin/sh): if [ $1 = 0 ]; then /sbin/service cumin stop >/dev/null 2>&1 /sbin/chkconfig --del cumin fi ----------------------------------------------- and despite that the processes stay running. cumin-0.1.4492-1.el5 How reproducible: 100% Steps to Reproduce: 1. Install, configure and run Cumin 2. rpm -e cumin Actual results: The cumin, cumin-data and cumin-web processes stay hanging around after package removal. Expected results: Processes stopped during preuninstall. Additional info: This is a regression.
I cannot reproduce anymore, tried all last builds since 4475 to 4494.
I have noticed it again, will try to describe more when I am sure how to reproduce it.
I think something different may be happening here. /sbin/service cumin start will start additional cumin services, even if one is already running. I tried this on 4410, which goes back to around the time of the 1.3 release. There is a facility to track the pid of the current service, but it is not being used in the init.d script. I am guessing that something in Jan's test recipe is causing /sbin/service cumin start to be called multiple times. I cannot reproduce the case where rpm uninstall simply fails to stop the service. Although if it did, then the pidfile would catch the error on next start anyway. I will try to repair the pidfile feature and see where it leads.
Created attachment 475888 [details] Patch for /etc/init.d/cumin, uses pidfile to track running process Jan, give this a try. I think this should make your multiple cumin's disappear. Let me know if it works.
Oops, change line 43 to this to print out the pid of the current process: echo -e "\n$servicename is already running, pid is `cat $pidfile`"
Created attachment 475913 [details] Revised, patch for /etc/init.d/cumin This is better. No need for pidfile. Cheers to matt for noting standard status() call in /etc/init.d/functions.
Same issue as bug 644302?
(In reply to comment #7) > Same issue as bug 644302? Yes, best evidence suggests this is a duplicate.
*** Bug 644302 has been marked as a duplicate of this bug. ***
I do not think those bugs are duplicate. 1. (bug 644302) the init script does not check if cumin is running, so it can run it again (ending with 'Address already in use') 2. (this bug - at least my original idea) the leftover cumin processes are hanging around when cumin WebUI is accessed while being removed. Needless to say that installing another cumin in this stage results in 'Address already in use' again, but from another perspective - the main problem that the processes of already-uninstalled cumin are still running
Created attachment 476564 [details] reproducer script # sh cumin-clean + killall cumin + rpm -e qpid-cpp-server + rm -rf /var/lib/qpidd /etc/qpidd.conf + yum --quiet -C -y install qpid-cpp-server + service qpidd start Starting Qpid AMQP daemon: [ OK ] + rpm -e cumin postgresql-server + rm -rf /var/lib/pgsql /etc/cumin /etc/sysconfig/cumin + yum --quiet -C -y install cumin + echo yes + cumin-database install + cumin-admin add-user cumin cumin User 'cumin' is added + service cumin start Starting cumin: [ OK ] + ps -e + grep cumin 18679 ? 00:00:00 cumin 18681 ? 00:00:00 cumin-data 18682 ? 00:00:00 cumin-web + leftover + rpm -e cumin postgresql-server Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/cumin/python/mint/update.py", line 52, in run update.process(self) File "/usr/share/cumin/python/mint/update.py", line 197, in process thread.conn.rollback() InterfaceError: connection already closed + yum --quiet -C -y install cumin + service postgresql start Starting postgresql service: [ OK ] + service cumin start Starting cumin: [ OK ] + ps -e + grep cumin 18679 ? 00:00:00 cumin 18681 ? 00:00:01 cumin-data 18682 ? 00:00:02 cumin-web 19042 ? 00:00:00 cumin 19043 ? 00:00:00 cumin-data 19044 ? 00:00:00 cumin + sleep 5 Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/cumin/python/wooly/server.py", line 213, in run self.wsgi_server.start() File "/usr/share/cumin/python/wooly/wsgiserver/__init__.py", line 1660, in start raise socket.error(msg) error: (98, 'Address already in use') + set +x Again? [Y/n]: + leftover + rpm -e cumin postgresql-server + yum --quiet -C -y install cumin + service postgresql start Starting postgresql service: [ OK ] + service cumin start Starting cumin: [ OK ] + ps -e + grep cumin 18679 ? 00:00:00 cumin 18681 ? 00:00:01 cumin-data 18682 ? 00:00:02 cumin-web 19042 ? 00:00:00 cumin 19043 ? 00:00:02 cumin-data 19044 ? 00:00:02 cumin-web 19395 ? 00:00:00 cumin 19396 ? 00:00:00 cumin-data 19398 ? 00:00:00 cumin + sleep 5 Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/cumin/python/mint/update.py", line 52, in run update.process(self) File "/usr/share/cumin/python/mint/update.py", line 197, in process thread.conn.rollback() InterfaceError: connection already closed Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/cumin/python/wooly/server.py", line 213, in run self.wsgi_server.start() File "/usr/share/cumin/python/wooly/wsgiserver/__init__.py", line 1660, in start raise socket.error(msg) error: (98, 'Address already in use') + set +x Again? [Y/n]: n
Reproduced on cumin-0.1.4410-2.el5 (MRG 1.3)
With the same script reproduced also on cumin-0.1.4494-1.el5
For completeness, with cumin-0.1.4494-1.el5 the Python traces are on lines: ----------------------------------------------------------------------------- Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/cumin/python/wooly/server.py", line 209, in run self.wsgi_server.start() File "/usr/share/cumin/python/wooly/wsgiserver/__init__.py", line 1660, in start raise socket.error(msg) error: (98, 'Address already in use') ----------------------------------------------------------------------------- Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/cumin/python/mint/update.py", line 52, in run update.process(self) File "/usr/share/cumin/python/mint/update.py", line 199, in process thread.conn.rollback() InterfaceError: connection already closed ----------------------------------------------------------------------------- Traceback (most recent call last): File "/usr/bin/cumin", line 72, in ? File "/usr/bin/cumin", line 64, in main OSError: [Errno 3] No such process ----------------------------------------------------------------------------- Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "/usr/share/cumin/python/mint/update.py", line 52, in run File "/usr/share/cumin/python/mint/update.py", line 199, in process InterfaceError: connection already closed ----------------------------------------------------------------------------- Traceback (most recent call last): File "/usr/bin/cumin-data", line 78, in ? main() File "/usr/bin/cumin-data", line 31, in main mint.check() File "/usr/share/cumin/python/mint/main.py", line 35, in check self.database.check() File "/usr/share/cumin/python/mint/database.py", line 18, in check self.check_connection() File "/usr/share/cumin/python/mint/database.py", line 24, in check_connection conn = self.get_connection() File "/usr/share/cumin/python/mint/database.py", line 13, in get_connection return psycopg2.connect(self.dsn) psycopg2.OperationalError: could not connect to server: Connection refused Is the server running on host "localhost" and accepting TCP/IP connections on port 5432? ----------------------------------------------------------------------------- Traceback (most recent call last): File "/usr/bin/cumin-data", line 78, in ? main() File "/usr/bin/cumin-data", line 32, in main mint.init() File "/usr/share/cumin/python/mint/main.py", line 50, in init self.update_thread.init() File "/usr/share/cumin/python/mint/update.py", line 30, in init self.conn = self.app.database.get_connection() File "/usr/share/cumin/python/mint/database.py", line 13, in get_connection return psycopg2.connect(self.dsn) psycopg2.OperationalError: FATAL: the database system is shutting down -----------------------------------------------------------------------------
Another note: after all, I found it has nothing to do with multiple cumin instances, neither with an accessed WebUI (e.g. Firefox open with http://localhost:45672). The attached script works with firefox being completely turned off, nothing is accessing localhost:45672 at all.
The severity here is pretty low, for the last thing we found out with Trevor is, that it happens only when yum uninstalls both PostgreSQL and cumin in one transaction (stopping pgsql first, then Cumin hanging, not closing connections). I let the decision to the developers, but it might be NOTABUG, huh?
Fixed in revision 4776. "yum remove postgresql cumin" should now work (read on). Some general checks that the database server is running and that the cumin environment is as expected were being done as blanket tests at the top of the init.d/cumin script. These tests are appropriate for various start commands (start, restart, condrestart) but they are not relevant for the status or stop commands. The result was that cumin could not be shut down with the service script if the postgres server was not running! General problem, but showed up in Jan's install/uninstall testing. These tests have been moved inside the start command. Another problem that may deserve a little attention is how cumin handles the posgtres server going away while it is running. I'm not sure it handles this gracefully (but this is another issue).
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause The /etc/init.d/cumin script checks for a running database server and a full cumin environment before allowing the 'status' or 'stop' service commands to run. These checks are only valid for the 'start' command in its various forms. Consequence The extra checks can erroneously prevent the 'status' and 'stop' service commands from running under certain circumstances. Fix These checks are only applied during startup of the cumin service. Result The 'status' and 'stop' service commands should complete successfully even if the database server is stopped or the cumin environment is not complete.
Verified with cumin-0.1.4865-1.el5
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-1249.html