Bug 1113959

+++ This bug was initially created as a clone of Bug #1113543 +++

Description of problem:
The %post server of gluster.spec says:
killall glusterd &> /dev/null
glusterd --xlator-option *.upgrade=on -N
This doesn't wait for the old glusterd to actually exit, so the new one sees it cannot bind to the interface and quits, and then the original one quits, leaving no glusterd actually running.


Version-Release number of selected component (if applicable):
glusterfs-3.5.1


How reproducible:
Everytime


Steps to Reproduce:
1. Run glusterd
2. Upgrade from 3.5.0 to 3.5.1
3.

Actual results:
No glusterd running anymore


Expected results:
An upgraded glusterd running


Additional info:

--- Additional comment from Niels de Vos on 2014-06-26 09:35:37 EDT ---

The post installation script for the glusterfs-server handles the restarting of glusterd incorrect. This caused an outage when the glusterfs-server package was automatically updated.

After checking the logs together with Patrick, we came to the conclusion that the running glusterd should have received a signal and would be exiting. However, the script does not wait for the running glusterd to exit, and starts a new glusterd process immediately after sending the SIGTERM. In case the 1st glusterd process has not exited yet, the new glusterd process can not listen on port 24007 and exits. The 1st glusterd will exit eventually too, leaving the service unavailable.

Snippet from the .spec:

 735 %post server
 ...
 769 pidof -c -o %PPID -x glusterd &> /dev/null
 770 if [ $? -eq 0 ]; then
 ...
 773     killall glusterd &> /dev/null
 774     glusterd --xlator-option *.upgrade=on -N
 775 else
 776     glusterd --xlator-option *.upgrade=on -N
 777 fi
 ...


I am not sure what the best way is to start glusterd with these specific options once. Maybe these should get listed in /etc/sysconfig/glusterd so that the standard init-script or systemd-job handles it?

--- Additional comment from Kaleb KEITHLEY on 2014-06-26 11:31:35 EDT ---

Which is the primary concern, that the new glusterd was started too soon? That we need a cleaner solution for starting glusterd with the *.upgrade=on option? Or both?

--- Additional comment from Anand Avati on 2014-06-26 17:18:17 EDT ---

REVIEW: http://review.gluster.org/8185 (build/glusterfs.spec.in: %post server doesn't wait for old glusterd) posted (#1) for review on master by Kaleb KEITHLEY (kkeithle)

--- Additional comment from Anand Avati on 2014-06-27 06:36:21 EDT ---

REVIEW: http://review.gluster.org/8185 (build/glusterfs.spec.in: %post server doesn't wait for old glusterd) posted (#2) for review on master by Kaleb KEITHLEY (kkeithle)