Bug 764603 (GLUSTER-2871)

Glusterfs native client mounts do not work for remote machines as of 3.1.4 due to an xdr error.  We were able to confirm with a 3.1.4 and a 3.2.0 client and server pair (3.1.4 client to 3.1.4 server, and a 3.2.0 client to a 3.2.0 server)

Server side logs are as follows for the specific test case:



[2011-05-02 09:37:13.508855] D [socket.c:193:__socket_rwv] 0-socket.management: EOF from peer 192.168.1.171:1022
[2011-05-02 09:37:13.508886] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (192.168.1.171:1022)
[2011-05-02 09:37:13.508904] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now

There is no firewall on this system, and the ports are unblocked


Client system:


root@metal:/local2/home/landman/glusterfs-3.2.0# glusterfs --debug --volfile-server=jr5-lab /mirror1gfs
[2011-05-02 15:40:20.169571] D [xlator.c:1245:xlator_dynload] 0-xlator: dlsym(reconfigure) on /usr/local/lib/glusterfs/3.2.0/xlator/mount/fuse.so: undefined symbol: reconfigure -- neglecting
[2011-05-02 15:40:20.169624] D [xlator.c:1251:xlator_dynload] 0-xlator: dlsym(validate_options) on /usr/local/lib/glusterfs/3.2.0/xlator/mount/fuse.so: undefined symbol: validate_options -- neglecting
[2011-05-02 15:40:20.169655] D [glusterfsd.c:314:create_fuse_mount] 0-: fuse direct io type 2
[2011-05-02 15:40:20.172294] D [rpc-clnt.c:914:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins
[2011-05-02 15:40:20.172337] D [rpc-transport.c:656:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/local/lib/glusterfs/3.2.0/rpc-transport/socket.so
[2011-05-02 15:40:20.172521] D [rpc-transport.c:97:__volume_option_value_validate] 0-glusterfs: no range check required for 'option remote-port 24007'
[2011-05-02 15:40:20.172562] D [rpc-clnt.c:1295:rpcclnt_cbk_program_register] 0-glusterfs: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2011-05-02 15:40:20.173889] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-192.168.1.154 (port-24007) for hostname: jr5-lab and port: 24007
[2011-05-02 15:40:20.174363] W [rpc-common.c:38:xdr_serialize_generic] (-->glusterfs() [0x408118] (-->glusterfs(glusterfs_volfile_fetch+0x85) [0x408065] (-->glusterfs(mgmt_submit_request+0x174) [0x406b24]))) 0-xdr: XDR encoding failed
[2011-05-02 15:40:20.174387] W [glusterfsd-mgmt.c:428:mgmt_submit_request] 0-: failed to create XDR payload
[2011-05-02 15:40:20.174403] E [glusterfsd-mgmt.c:753:mgmt_rpc_notify] 0-mgmt: failed to fetch volume file (key:(null))
[2011-05-02 15:40:20.174438] W [glusterfsd.c:700:cleanup_and_exit] (-->/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x28) [0x7f5ea1d8aaf8] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xfb) [0x7f5ea1d8fb7b] (-->glusterfs() [0x408228]))) 0-: received signum (0), shutting down
[2011-05-02 15:40:20.174458] D [glusterfsd-mgmt.c:1016:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
[2011-05-02 15:40:20.174474] I [fuse-bridge.c:3688:fini] 0-fuse: Unmounting '/mirror1gfs'.


The same mount works for nfs


root@metal:/local2/home/landman/glusterfs-3.2.0# showmount -e jr5-lab
Export list for jr5-lab:
/jr5mirror1 *
root@metal:/local2/home/landman/glusterfs-3.2.0# mount -t nfs jr5-lab:/jr5mirror1 /mirror1gfs

root@metal:/local2/home/landman/glusterfs-3.2.0# df -h /mirror1gfs/
Filesystem            Size  Used Avail Use% Mounted on
jr5-lab:/jr5mirror1   5.5T  193G  5.3T   4% /mirror1gfs


This appears to be the case on Centos 5.5 and 5.6 clients and servers, Ubuntu 10.04 client.

Oddly enough, the mount works correctly on the same host as the exported directory.


[root@jr5-lab ~]# !1033
mount -t glusterfs  localhost:/jr5mirror1 /mirror1
[root@jr5-lab ~]# df -h /mirror1
Filesystem            Size  Used Avail Use% Mounted on
localhost:/jr5mirror1
                      5.5T  193G  5.3T   4% /mirror1
[root@jr5-lab ~]# mount | grep mirror1
localhost:/jr5mirror1 on /mirror1 type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)

This is the case independent of kernel (verified behavior for 2.6.18-238.9.1.el5, and 2.6.32.{22,39}.scalable on server and client.  

Failure appeared immediately after upgrading to 3.1.4 and beyond.  3.1.2 and below did not have this problem.

Hi Joe,

How are you trying to mount glusterfs? Is it through using mount command or using glusterfs directly on commandline? Can you give the entry of fstab (or mount command line) - if you are using mount - or the exact glusterfs command line which you used to mount fuse clients?

regards,

(In reply to comment #1)
> By reverting to 3.1.3 on client and server, then reforming the volume, I can
> once again mount using gluster native client.

Bug is not reproducible on our local setup. We tried mounting glusterfs using 'mount -t glusterfs ...' and 'glusterfs -s localhost --volfile-id=..." and both were successful.