Red Hat Bugzilla – Bug 613870
Starting or stopping corosync blocks cman from starting or stopping
Last modified: 2010-09-29 06:27:43 EDT
Description of problem: Two variants on the same issue: First: If you start corosync manually and then try to start cman, it fails with the error "Starting cman... corosync died: Error, reason code is 1 [FAILED]". If you then stop corosync and again try to start cman, it starts properly. Second: Perhaps most seriously; If cman is already started and you try to restart or stop corosync, corosync will sit there endlessly "Waiting for corosync services to unload:...". Hitting ctrl+c to stop it *appears* to abort the corosync restart. However, anytime there after, trying to stop or restart cman will fail with "Stopping cman... Timed-out waiting for cluster [FAILED]". Running 'ps aux | grep corosync' shows "root 4262 0.4 1.9 440156 34728 ? SLsl 22:57 0:01 corosync -f". This process can only be killed with '-9'. Once dead though, cman will restart successfully. Version-Release number of selected component (if applicable): - cman-3.0.12-2.fc13.x86_64 - corosync-1.2.3-1.fc13.x86_64 How reproducible: Appears to be 100%. Steps to Reproduce: 1. Start corosync, then start cman 2. Start cman, stop|restart corosync, stop|restart cman 3. Actual results: - cman won't stop/start when corosync is running or restarted. - corosync won't stop/restart when cman is running and then blocks cman from starting/stopping. Expected results: - cman should detect when cman is already running and provide more useful feedback, if not stop corosync itself. - corosync should detect when cman is available and not start with an error telling the user to use cman instead. Additional info: I've got a disposable test cluster. I can run any tests the developers would like me to try.
Thanks for the bug report The common POSIX solution (missing from current corosync) is to have corosync create a file in LOCALSTATEDIR/lock/corosync then use the flock(2) call ie: fd = open (LOCALSTATEDIR"/lock/corosync) retry_flock; res = flock (fd, LOCK_EX|LOCK_NB); if (res == -1) { switch (errno) { case EINTR: goto retry_flock break; case EWOULDBLOCK: print error that corosync is already active and exit break; default print error that flock couldn't be obtained and exit break; } } The flock is GCed on process exit by POSIX allowing a new start of corosync to grab the lock.
Thanks Steven, Will this be added in the next release? If so, I guess this ticket can be closed?
Created attachment 433737 [details] Proposed patch for first part of problem Uses solution described by Steve
Created attachment 435027 [details] Proposed patch for first part - take 2 Better version of patch. It also includes change in initscript to NOT create pid file (corosync itself now does).
Created attachment 435028 [details] Proposed patch for second problem This patch fixes second problem in initscript. If corosync was run by cman, initsript refuses to kill corosync and exit itself.
Created attachment 450432 [details] Cman: Handle corosync exit codes
Created attachment 450433 [details] Cman: Handle "another instance running" error code
Created attachment 450434 [details] Cman: test that corosync is not already running Patch fixes init file so now before cman start is tested, if corosync is running. If so, init script will refuse to start.
Created attachment 450435 [details] Cman: Handle INT and TERM signals correctly Corosync signal handler (SIGINT and SIGTERM) is replaced by cman one, and this was settting quit_threads to 1. Regular cman shutdown sequence (cman_tool leave) tests if quit_threads is set or not. If so, it refuses continue so it was not possible to cleanly leave cluster. Now SIGINT and SIGTERM are ignored, and (un)intentional kill of corosync is no longer problem.
All patches together should give complete solution without any possibility to replicate bugs. Also patches are currently included in cluster STABLE3 tree and/or corosync trunk so closing as upstream.