Bug 1659824

Summary: Unable to mount gluster fs on glusterfs client: Transport endpoint is not connected
Product: [Community] GlusterFS Reporter: aedu <aedu>
Component: fuseAssignee: bugs <bugs>
Status: CLOSED NOTABUG QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 5CC: amukherj, atumball, bugs, pasik
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-25 13:07:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Logs from all 3 nodes none

Description aedu@wyssmann.com 2018-12-16 19:59:26 UTC 
Created attachment 1514920 [details]
Logs from all 3 nodes

Description of problem:

I have a gluster cluster setup with 3 nodes.

    server1, 192.168.100.1
    server2, 192.168.100.2
    server3, 192.168.100.3

They are connected via an internal network 192.160.100.0/24. However, I want to connect with a client from outside the network using the public ip of one of the servers, which does not work:

sudo mount -t glusterfs x.x.x.x:/datavol /mnt/gluster/

Gives something like this in the log:

[2018-12-15 17:57:29.666819] I [fuse-bridge.c:4153:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.26
[2018-12-15 18:23:47.892343] I [fuse-bridge.c:4259:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.26
[2018-12-15 18:23:47.892375] I [fuse-bridge.c:4870:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-12-15 18:23:47.892475] I [MSGID: 108006] [afr-common.c:5650:afr_local_init] 0-datavol-replicate-0: no subvolumes up
[2018-12-15 18:23:47.892533] E [fuse-bridge.c:4328:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2018-12-15 18:23:47.892651] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2018-12-15 18:23:47.892668] W [fuse-bridge.c:3250:fuse_statfs_resume] 0-glusterfs-fuse: 2: STATFS (00000000-0000-0000-0000-000000000001) resolution fail
[2018-12-15 18:23:47.892773] W [fuse-bridge.c:889:fuse_attr_cbk] 0-glusterfs-fuse: 3: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-12-15 18:23:47.894204] W [fuse-bridge.c:889:fuse_attr_cbk] 0-glusterfs-fuse: 4: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-12-15 18:23:47.894367] W [fuse-bridge.c:889:fuse_attr_cbk] 0-glusterfs-fuse: 5: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-12-15 18:23:47.916333] I [fuse-bridge.c:5134:fuse_thread_proc] 0-fuse: initating unmount of /mnt/gluster
The message "I [MSGID: 108006] [afr-common.c:5650:afr_local_init] 0-datavol-replicate-0: no subvolumes up" repeated 4 times between [2018-12-15 18:23:47.892475] and [2018-12-15 18:23:47.894347]
[2018-12-15 18:23:47.916555] W [glusterfsd.c:1481:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f90f2306494] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x5591a51e87ed] -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x5591a51e8644] ) 0-: received signum (15), shutting down
[2018-12-15 18:23:47.916573] I [fuse-bridge.c:5897:fini] 0-fuse: Unmounting '/mnt/gluster'.
[2018-12-15 18:23:47.916582] I [fuse-bridge.c:5902:fini] 0-fuse: Closing fuse connection to '/mnt/gluster'.


Version-Release number of selected component (if applicable):
GlusterFS 5.2

How reproducible:


Steps to Reproduce:
1. Setup a replicated volume with 3 nodes
2. Run following command from client: 
sudo mount -t glusterfs x.x.x.x:/datavol /mnt/gluster/

Actual results:
mount unsuccessful

Expected results:
mount successful

Additional info:
More details: https://github.com/gluster/glusterfs/issues/615
Comment 1 Amar Tumballi 2019-03-07 08:26:54 UTC 
Sorry about getting back to you late on this.

This is a case of gluster's volgen having IP/Hostname details in 'protocol/client' given during volume creation.

You can check the volfile you received in client side (available on log file), and see what is the 'remote-host' option pointing at. If its a server name, then you can have a overriding DNS resolution through /etc/hosts to use public IP, and that should be enough.

If you are seeing IP address in 'remote-host' option, you may have to copy the volfile, and start glusterfs process with `glusterfs -f$local-volfile $mountpoint`. This should work.
Comment 2 Amar Tumballi 2019-03-07 08:27:33 UTC 
I would say this is not a 'Bug', but missing documentation.
Comment 3 Amar Tumballi 2019-06-25 13:07:24 UTC 
With comment#1 and comment#2