Bug 1107649

Issue:
----------------------------------
glusterd_friend_sm ()
{
  quorum_action = _gf_false;

  while (!list_empty (&gd_friend_sm_queue)){
              //blah blah
               quorum_action = _gf_true;
  }
  if (quorum_action) 
                glusterd_spawn_daemons
}
----------------------------------

As long as node 2 is down gd_friend_sm_queue is empty and hence glusterd_spawn_daemons never gets called.

While discussing with KP, I was given to understand that the above code was intentionally written so that each glusterd does not start the glusterfsd processes until it's friends are also up and running and are in sync. Need to come up with a solution which covers the use case given in the bug description. A workaround is to 'gluster volume start <volname> force` on the node which is up.

REVIEW: http://review.gluster.org/8034 (glusterd: spawn daemons/processes when peer count less than 2) posted (#1) for review on master by Ravishankar N (ravishankar)

Have a user configurable timeout.  In fact, that was what I was expecting, but after waiting for a long time I realized that wasn't the way it worked.  I think a timeout value is a good compromise.  Maybe something like 5 minutes as a default?

Closing this as currenly there is no way of determining if the one node that came up after both nodes went down is the pristine one after all.