Bug 859248

Summary: Mount fails when the Gluster Server has an IPv4 and IPv6 address
Product: [Community] GlusterFS Reporter: Pete Zaitcev <zaitcev>
Component: rpcAssignee: bugs <bugs>
Status: CLOSED INSUFFICIENT_DATA QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.1.0CC: bugs, gluster-bugs, iny, joe, jonathansteffan, matthias, ndevos, silas, zaitcev
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-04 13:03:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pete Zaitcev 2012-09-20 21:48:08 UTC 
Description of problem:

[root@kvm-ichi ~]# mount /mnt/g
Mount failed. Please check the log file for more details.
[root@kvm-ichi ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Jun 13 20:14:17 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_kvm--ichi-lv00 /                 ext4    defaults        1 1
UUID=65e38f15-bfb8-4eac-8f3e-bd4bf732c388 /boot ext2    defaults        1 2
/dev/mapper/vg_kvm--ichi-lvswap swap            swap    defaults        0 0
/dev/vdb                /mnt/vdb                xfs     defaults        1 2
/dev/vdc                /mnt/vdc                xfs     defaults        1 2
simbelmyne:/q/zaitcev   /q/zaitcev              nfs     noatime,nolock,defaults 1 2
simbelmyne.zaitcev.lan:/g       /mnt/g    glusterfs     defaults,noauto 0 0
[root@kvm-ichi ~]# 

Version-Release number of selected component (if applicable):

glusterfs-3.3.0-4.fc18.x86_64
kernel-3.6.0-0.rc2.git2.1.fc18.x86_64
util-linux-2.21.2-3.fc18.x86_64

How reproducible:

100% apparently

Steps to Reproduce:
1. add a volume to /etc/fstab
2. mount
  
Actual results:

Mount fails

Expected results:

Mount succeeds

Additional info:

/var/log/glusterfs/mnt-g.log:

[2012-09-20 17:32:14.870897] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0
[2012-09-20 17:32:14.904513] E [socket.c:1715:socket_connect_finish] 0-glusterfs: connection to  failed (Connection refused)
[2012-09-20 17:32:14.904690] E [glusterfsd-mgmt.c:1783:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Transport endpoint is not connected
[2012-09-20 17:32:14.904754] I [glusterfsd-mgmt.c:1786:mgmt_rpc_notify] 0-glusterfsd-mgmt: -1 connect attempts left
[2012-09-20 17:32:14.905100] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libgfrpc.so.0(rpc_transport_notify+0x23) [0x301cc0c313] (-->/lib64/libgfrpc.so.0(rpc_clnt_notify+0x114) [0x301cc10324] (-->/usr/sbin/glusterfs() [0x40c626]))) 0-: received signum (1), shutting down
[2012-09-20 17:32:14.905186] I [fuse-bridge.c:4643:fini] 0-fuse: Unmounting '/mnt/g'.

This used to work on F17. The fstab includes _netdev flag there:

[zaitcev@kvm-nova ~]$ grep gluster /etc/fstab
simbelmyne.zaitcev.lan:/g	/mnt/g     glusterfs	defaults,_netdev 0 0
[zaitcev@kvm-nova ~]$ df /mnt/g
Filesystem                1K-blocks    Used Available Use% Mounted on
simbelmyne.zaitcev.lan:/g 113533568 4386432 103379968   5% /mnt/g
[zaitcev@kvm-nova ~]$ 

Rawhide fails in the same way as F18 (glusterfs-3.3.0-8.fc19.x86_64).

Note that the server serves this volume to all hosts just fine. But just
in case:

[root@simbelmyne zaitcev]# gluster volume info

Volume Name: g
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: simbelmyne.zaitcev.lan:/mnt/brick1
[root@simbelmyne zaitcev]#
Comment 1 Pete Zaitcev 2012-09-20 23:51:20 UTC 
If I disable IPV6 temporarily, the following happens (in the log):

[2012-09-20 19:47:05.134986] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0
[2012-09-20 19:47:05.190509] E [glusterfsd-mgmt.c:1548:mgmt_getspec_cbk] 0-glusterfs: XDR decoding error
[2012-09-20 19:47:05.191276] E [glusterfsd-mgmt.c:1623:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:/g)
[2012-09-20 19:47:05.191723] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libgfrpc.so.0(rpc_clnt_notify+0xad) [0x301cc102bd] (-->/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x301cc0f8a5] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x297) [0x40c937]))) 0-: received signum (0), shutting down
[2012-09-20 19:47:05.191808] I [fuse-bridge.c:4643:fini] 0-fuse: Unmounting '/mnt/g'.

No longer "Connection refused", but XDR decoding error.

For the record, normally it looks like this:

[root@kvm-ichi zaitcev]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:bc:63:7f brd ff:ff:ff:ff:ff:ff
    inet 192.168.129.19/28 brd 192.168.129.31 scope global eth0
    inet6 fd2d:acfb:74cc:3::3/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:febc:637f/64 scope link 
       valid_lft forever preferred_lft forever
[root@kvm-ichi zaitcev]# host simbelmyne.zaitcev.lan
simbelmyne.zaitcev.lan has address 192.168.128.10
simbelmyne.zaitcev.lan has IPv6 address fd2d:acfb:74cc:1::a
[root@kvm-ichi zaitcev]#
Comment 2 Fedora Admin XMLRPC Client 2013-01-18 15:14:45 UTC 
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 3 Niels de Vos 2014-09-25 08:10:40 UTC 
It might be that this has been fixed in current releases (there are some IPv6 improvements). It is much appreciated if you can give us a status update.

Verification of this should be easy enough, configure both IPv4 and IPv6 on a server and client, modify /etc/hosts in different ways on the client to resolve the server over IPv4 only, IPv6 only and IPv4+IPv6.