Bug 1335090

Description of problem:

shared volume doesn't get mounted on one (maybe two) node after rebooting all nodes in cluster, resulting in missing symlink (/var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp42-239.lab.eng.blr.redhat.com/nfs) .

Version-Release number of selected component (if applicable):

glusterfs-3.7.9-4 and nfs-ganesha-2.3.1-6

How reproducible:
Always

Steps to Reproduce:
1. Create a 4 node ganesha cluster.
2. Make sure the shared volume is created and mounted on all the nodes of cluster and the symlink is created as below.

[root@dhcp42-20 ~]# gluster volume status gluster_shared_storage
Status of volume: gluster_shared_storage
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick dhcp42-239.lab.eng.blr.redhat.com:/va
r/lib/glusterd/ss_brick                     49155     0          Y       2293 
Brick dhcp43-175.lab.eng.blr.redhat.com:/va
r/lib/glusterd/ss_brick                     49155     0          Y       2281 
Brick dhcp42-20.lab.eng.blr.redhat.com:/var
/lib/glusterd/ss_brick                      49155     0          Y       2266 
Self-heal Daemon on localhost               N/A       N/A        Y       2257 
Self-heal Daemon on dhcp42-239.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       2287 
Self-heal Daemon on dhcp43-175.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       2253 
Self-heal Daemon on dhcp42-196.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       2258 
 
Task Status of Volume gluster_shared_storage
------------------------------------------------------------------------------
There are no active volume tasks

dhcp42-20.lab.eng.blr.redhat.com:/gluster_shared_storage  27740928 1697152  26043776   7% /run/gluster/shared_storage

dhcp42-239.lab.eng.blr.redhat.com:/gluster_shared_storage  27740928 1697152  26043776   7% /run/gluster/shared_storage

dhcp43-175.lab.eng.blr.redhat.com:/gluster_shared_storage  27740928 1697152  26043776   7% /run/gluster/shared_storage

dhcp42-196.lab.eng.blr.redhat.com:/gluster_shared_storage  27740928 1697152  26043776   7% /run/gluster/shared_storage

[root@dhcp42-20 ~]# ls -ld /var/lib/nfs
lrwxrwxrwx. 1 root root 80 May 11 21:26 /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp42-20.lab.eng.blr.redhat.com/nfs

[root@dhcp42-239 ~]# ls -ld /var/lib/nfs
lrwxrwxrwx. 1 root root 81 May 11 21:26 /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp42-239.lab.eng.blr.redhat.com/nfs

[root@dhcp43-175 ~]# ls -ld /var/lib/nfs
lrwxrwxrwx. 1 root root 81 May 11 21:26 /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp43-175.lab.eng.blr.redhat.com/nfs

[root@dhcp42-196 ~]# ls -ld /var/lib/nfs
lrwxrwxrwx. 1 root root 81 May 11 21:19 /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp42-196.lab.eng.blr.redhat.com/nfs

3. Reboot all the nodes of the cluster.
4. Observe that on 2 of the 4 nodes, shared storage is not mounted. (most of the times it doesnt get mounted on any one node).
5.And the symlink from /var/lib/nfs doesn't get created because of this on these 2 nodes.
6. Both of these nodes have the entries in /etc/fstab and manually mounting the shared storage on these nodes works.


Actual results:

Shared volume doesn't get mounted on few nodes after rebooting all nodes in cluster.

Expected results:

Shared volume should get mounted on all the nodes after reboot

Additional info: