Description of problem:

I see that gluster-blockd fails to come up on a fresh installed setup of OCP3.11 + OCS 3.11.2 and due to this deployment fails. This issue is not hit everytime but  we hit this intermittently.


Version-Release number of selected component (if applicable):
OCP 3.11 + OCS 3.11.2

How reproducible:
Intermittently (2/3)

Steps to Reproduce:
1. Install OCP 3.11 + OCS 3.11.2 using deploy_cluster.yml
2. Some times gluster-blockd fails to come up on one or two of the pods and due to this gluster install fails.
3.

Actual results:
Issue is hit intermittently where gluster-blockd does not come up on the ndoe.


Expected results:
gluster-blockd should always come up and we should not be hitting any issues

Additional info:

14:42:51 <kasturi>      can you tell me the file path
14:42:56 <kasturi>      let me check in other nodes too
14:43:04 <xiubli>       if tcmu-runner start up just after gluster-blockd service, then the gluster-blockd will faile
14:43:07 <xiubli>       as expected
14:43:27 <xiubli>       # /usr/lib/systemd/system/gluster-blockd.service
14:44:14 <kasturi>      other nodes as well i see the same thing
14:44:14 <kasturi>      sh-4.2# cat /usr/lib/systemd/system/gluster-blockd.service
14:44:14 <kasturi>      [Unit]
14:44:14 <kasturi>      Description=Gluster block storage utility
14:44:14 <kasturi>      Requisite=glusterd.service
14:44:14 <kasturi>      Requires=
14:44:14 <kasturi>      BindsTo=gluster-block-target.service
14:44:14 <kasturi>      After=gluster-block-target.service
14:44:23 <kasturi>      where the pods are in 0/1 state
14:45:18 <kasturi>      sorry, i see the requires empty in other pods too where they are up and running 
14:45:34 <xiubli>       yeah,
14:45:48 <kasturi>      so, is this a bug then
14:46:00 <kasturi>      i do not have a previous setup to compare
14:46:06 <xiubli>       ==================
14:46:07 <xiubli>       [2019-03-06 09:02:39.555722] ERROR: tcmu-runner not running [at gluster-blockd.c+383 :<blockNodeSanityCheck>]
14:46:13 <kasturi>      yes
14:46:18 <kasturi>      but it is actually running
14:46:47 <xiubli>       Active: active (running) since Wed 2019-03-06 09:02:41 UTC; 13min ago
14:47:09 <kasturi>      yes
14:47:12 <xiubli>       you can see the logs time is 02:39
14:47:27 <xiubli>       and the tcmu-runner's active time is 02:41
14:47:42 <xiubli>       means the tcmu-runner start after gluster-blockd
14:47:44 <kasturi>      so do you think tcmu is taking some time
14:47:49 <kasturi>      to start
14:50:04 <xiubli>       yeah
14:50:49 <xiubli>       the gluster-blockd will depend on the tcmu-runenr service, only after tcmu-runner is up, then the gluster-blockd will be successfully up
14:54:18 <kasturi>      okay
14:54:35 <kasturi>      do you think the above is a race then ?
14:57:52 <xiubli>       yeah
14:58:07 <xiubli>       BTW, will this be hit 100% in this pod ?
14:58:41 <kasturi>      did not get when you said 100% in this pod
14:58:51 <kasturi>      did you want to respin the pod and see if this is happening
15:00:13 <xiubli>       yeah, I meant could we reproduce this issue in that pod always when repining ?
15:00:21 <kasturi>      i am not sure
15:00:23 <kasturi>      let me try that
15:00:28 <xiubli>       okay