Bug 1317559

Summary: Glusterfs mounts in fstab are not mounted at boot
Product: [Community] GlusterFS Reporter: Mathieu MD <mathieu+redhat>
Component: glusterdAssignee: bugs <bugs>
Status: CLOSED EOL QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3.7.8CC: brian, bugs, hvjunk, ish1301, mliyazud, olim, wfdata
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-08 10:56:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mathieu MD 2016-03-14 14:45:14 UTC
Description of problem:
The Glusterfs mounts listed in /etc/fstab are not mounted at boot.

Version-Release number of selected component (if applicable):
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
glusterd --version
glusterfs 3.7.8 built on Mar 10 2016 20:20:45

How reproducible:
Everytime

Steps to Reproduce:
1. Add a line to mount local gluster volume vol1 in /etc/fstab:
myserver:/vol1 /mnt/test glusterfs defaults,_netdev 0 0
2. Make sure glusterd is enabled and running (systemctl status glusterd)
3. Manually mount /mnt/test: it works
4. Now, reboot the server
5. Make sure glusterd is running
6. Verify that /mnt/test is mounted

Actual results:
Not mounted

Expected results:
Mounted.

Additional info:
The log /var/log/glusterfs/mnt-test.log says:

[2016-03-14 14:28:53.298107] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-vol1-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2016-03-14 14:28:53.298236] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-vol1-client-1: disconnected from vol1-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-03-14 14:28:53.298271] E [MSGID: 108006] [afr-common.c:4015:afr_notify] 0-vol1-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2016-03-14 14:28:53.304312] I [fuse-bridge.c:5139:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-03-14 14:28:53.305185] I [fuse-bridge.c:4060:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-03-14 14:28:53.305425] I [MSGID: 108006] [afr-common.c:4143:afr_local_init] 0-vol1-replicate-0: no subvolumes up
[2016-03-14 14:28:53.307306] W [fuse-bridge.c:758:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected)
[2016-03-14 14:28:53.318197] I [fuse-bridge.c:4986:fuse_thread_proc] 0-fuse: unmounting /mnt/test
The message "I [MSGID: 108006] [afr-common.c:4143:afr_local_init] 0-vol1-replicate-0: no subvolumes up" repeated 2 times between [2016-03-14 14:28:53.305425] and [2016-03-14 14:28:53.316897]
[2016-03-14 14:28:53.321196] W [glusterfsd.c:1236:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7ff87fdbcdc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7ff881427905] -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7ff881427789] ) 0-: received signum (15), shutting down
[2016-03-14 14:28:53.321225] I [fuse-bridge.c:5685:fini] 0-fuse: Unmounting '/mnt/test'.

Notes:
- I also tried to manually create a Systemd unit in /etc/systemd/system/mnt-test.mount with "After=glusterd.service" and "After=glusterd.service sys-module-fuse.device", but it doesn't help to mount this path at boot.
- Manually mounting works everytime; it must be a timing problem during the boot.

Comment 1 Kaushal 2017-03-08 10:56:10 UTC
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life.

Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS.
If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.

Comment 2 Hendrik 2017-05-02 12:43:18 UTC
This is still valid for Gluster 3.10
mount needs to either retry, or the glusterfsd needs to only finish once a "gluster volume list" is functional with the glusterfsd talking to the other in the cluster.
This is also easy to recreate during bootstrapping of the cluster

Comment 3 Ish 2017-09-26 18:57:15 UTC
There should some option to delayed start of GlusterFS mount or auto retry to mount if it fails