Description of problem: If the RHQ Server machine crashes, it will not function through rhqctl anymore, as the cassandra.pid still exists (and the rhqctl won't notice it). If you start with rhqctl, it tells you storage node is up and everything is okay (while storage node is actually down): michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ ./rhqctl start 13:52:31,690 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1 RHQ storage node (pid 41238) is running Trying to start the RHQ Server... RHQ Server (pid 3712 ) IS starting Starting RHQ Agent... RHQ Agent (pid 3853 ) IS running michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ ps xa | grep 41238 3931 pts/0 S+ 0:00 grep --colour=auto 41238 michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ And stopping generates an error message, but doesn't remove the pid-file: michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ ./rhqctl stop 13:54:50,448 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1 Stopping RHQ Agent... RHQ Agent (pid=3853) is stopping... RHQ Agent has stopped. Trying to stop the RHQ Server... RHQ Server (pid=3712) is stopping... RHQ Server has stopped. Stopping RHQ storage node... RHQ storage node (pid=41238) is stopping... 13:55:00,003 ERROR [org.rhq.server.control.RHQControl] Failed to stop services [Cause: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)] michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ Version-Release number of selected component (if applicable): 4.9.0-SNAPSHOT (28th of June git checkout) How reproducible: Steps to Reproduce: 1. Start RHQ 2. Shutdown it with force (or remove power from VM, doesn't matter, as long as the pid-file stays) 3. Start again. Actual results: RHQ stays unavailable, although start script says everything is fine. Expected results: rhqctl should notice that the pid-file points to a dead pid, and remove it and restart the storage node. Additional info:
Created attachment 767411 [details] Checks if that pid is really active or not, and deletes the pid file if it isn't
master b3aa6d8a54 Thanks Michael!
when i do kill -9 <process id of cassandra> and then i run ./rhqctl status it still says that RHQ Storage node is running. Not sure if this should be filed as a seperate bug or should this issue be addressed by this bug.
Hi, Fixing that one wasn't in my mind when I created the patch, but here's a fix attached (which I can't right now test unfortunately, but I'll do that later).
Created attachment 781565 [details] Fix the status command with ghost pid files
based on comment #3, this does not look verified.
After applying the newest (attachment 781565 [details]) patch: michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ ./rhqctl status 00:28:53,533 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1 RHQ Storage Node (pid 1462 ) IS running RHQ Server (pid 1651 ) IS running JBossAS Java VM child process (pid 1651 ) IS running RHQ Agent (pid 1861 ) IS running michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ kill -9 1462 michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ ./rhqctl status 00:29:06,301 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1 RHQ Storage Node (no pid file) IS NOT running RHQ Server (pid 1651 ) IS running JBossAS Java VM child process (pid 1651 ) IS running RHQ Agent (pid 1861 ) IS running michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ echo $? 0 michael@grace-mint ~/projects/rhq/dev-container/rhq-server/bin $ So it works fine now with status command also.
2nd patch has been pushed to master as cdf64dfe0d3b3 Thanks again, Michael!
verified. please get screenshot and log attached.
Created attachment 793155 [details] status.png
Created attachment 793161 [details] status.log
Bulk closing of RHQ 4.9 verified items