Bug 764188 - (GLUSTER-2456) IPv6 support for glusterd
IPv6 support for glusterd
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: glusterd (Show other bugs)
3.1.2
All Linux
low Severity low
: ---
: ---
Assigned To: Gaurav
:
: 764583 (view as bug list)
Depends On: GLUSTER-3548
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-23 12:43 EST by Vijay Bellur
Modified: 2015-12-01 11:45 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vijay Bellur 2011-02-23 12:43:33 EST
Peer probe with any IPv6 address fails. Following errors are seen in the logfile:


[2011-02-23 13:16:26.53863] E [name.c:251:af_inet_client_get_remote_sockaddr] management: DNS resolution failed on host 2001::2
[2011-02-23 13:16:26.53932] E [common-utils.c:124:gf_resolve_ip6] resolver: getaddrinfo failed (Address family for hostname not supported)
Comment 1 Anand Avati 2011-07-14 01:31:41 EDT
PATCH: http://patches.gluster.com/patch/7564 in master (Glusterd: IPV6 support for glusterfs.)
Comment 2 Csaba Henk 2011-07-14 09:48:34 EDT
Since patch 7564, gluster cannot connect to glusterd on my system. I get:

# gluster system:: getwd
Connection failed. Please check if gluster daemon is operational.

(with any gluster command, not just for this one).
Comment 3 Anand Avati 2011-07-14 10:07:12 EDT
(In reply to comment #2)
> Since patch 7564, gluster cannot connect to glusterd on my system. I get:
> 
> # gluster system:: getwd
> Connection failed. Please check if gluster daemon is operational.
> 
> (with any gluster command, not just for this one).

Strange, the commit scripts performs tests of creating volume, start,
stop etc from cli to glusterd with every change, including this one -
and it has passed because the commit has gone through.
Comment 4 Csaba Henk 2011-07-14 11:54:29 EDT
(In reply to comment #3)
> (In reply to comment #2)
> > Since patch 7564, gluster cannot connect to glusterd on my system. I get:
> > 
> > # gluster system:: getwd
> > Connection failed. Please check if gluster daemon is operational.
> > 
> > (with any gluster command, not just for this one).
> 
> Strange, the commit scripts performs tests of creating volume, start,
> stop etc from cli to glusterd with every change, including this one -
> and it has passed because the commit has gone through.

OK, turned out the error was on my side. The system was not properly set up for IPv6.
Comment 5 Anand Avati 2011-07-14 11:58:48 EDT
(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #2)
> > > Since patch 7564, gluster cannot connect to glusterd on my system. I get:
> > > 
> > > # gluster system:: getwd
> > > Connection failed. Please check if gluster daemon is operational.
> > > 
> > > (with any gluster command, not just for this one).
> > 
> > Strange, the commit scripts performs tests of creating volume, start,
> > stop etc from cli to glusterd with every change, including this one -
> > and it has passed because the commit has gone through.
> 
> OK, turned out the error was on my side. The system was not properly set up for
> IPv6.

I still think this is a bug. Connections should still work on ipv4 even if system is not set up for ipv6.
Comment 6 Csaba Henk 2011-07-14 12:20:26 EDT
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > (In reply to comment #2)
> > > > Since patch 7564, gluster cannot connect to glusterd on my system. I get:
> > > > 
> > > > # gluster system:: getwd
> > > > Connection failed. Please check if gluster daemon is operational.
> > > > 
> > > > (with any gluster command, not just for this one).
> > > 
> > > Strange, the commit scripts performs tests of creating volume, start,
> > > stop etc from cli to glusterd with every change, including this one -
> > > and it has passed because the commit has gone through.
> > 
> > OK, turned out the error was on my side. The system was not properly set up for
> > IPv6.
> 
> I still think this is a bug. Connections should still work on ipv4 even if
> system is not set up for ipv6.

That's true, but there is a difference between a system which does not support IPv6 and one which has IPv6 enabled but misconfigured. It seems to me that mine was the latter case, please verify and decide if glusterfs can/should do anything about it.

This was my /etc/hosts file:

http://projects.archlinux.org/svntogit/packages.git/plain/filesystem/trunk/hosts?id=b8efbc3c43bf9c8f60edb684e680567427ea0b30

Nb. with this setup I also had an issue with ab -- it got ECONNREFUSED when trying to test a local http server like "ab http://localhost:8001/"

I did the following changes after consulting an Ubuntu hosts file:

# curl -s http://projects.archlinux.org/svntogit/packages.git/plain/filesystem/trunk/hosts?id=b8efbc3c43bf9c8f60edb684e680567427ea0b30 | diff -up - /etc/hosts
--- -   2011-07-14 12:07:08.887668907 -0700
+++ /etc/hosts  2011-07-14 11:49:13.463000071 -0700
@@ -4,6 +4,14 @@
 
 #<ip-address>  <hostname.domain.org>   <hostname>
 127.0.0.1      localhost.localdomain   localhost
-::1            localhost.localdomain   localhost
+#::1           localhost.localdomain   localhost
+
+::1     ip6-localhost ip6-loopback
+fe00::0 ip6-localnet
+ff00::0 ip6-mcastprefix
+ff02::1 ip6-allnodes
+ff02::2 ip6-allrouters
+
+
 
 # End of file

And now, both ab and gluster behave as expected.
Comment 7 Csaba Henk 2011-07-16 16:28:38 EDT
Put it simply, aside hosts file tuning.

Should this not work:

# gluster --remote-host=::1 system:: getwd

? Because it does not, AFAICS.
Comment 8 Anand Avati 2011-07-17 04:35:15 EDT
(In reply to comment #7)
> Put it simply, aside hosts file tuning.
> 
> Should this not work:
> 
> # gluster --remote-host=::1 system:: getwd
> 
> ? Because it does not, AFAICS.

I believe that should have worked.

Gaurav, can you check this? You can compare gluster's behaviour with that of ssh (which supports IPv6) to decide on such questions.

Avati
Comment 9 Philipp Schmid 2011-07-23 11:51:52 EDT
I tried to setup gluster to work over IPv6, but it just wouldn't listen on IPv6.

This is how my glusterd.vol looked like:

volume management
    type mgmt/glusterd
    option working-directory /etc/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.address-family inet/inet6
end-volume


Then I tried to listen only on inet6 and it worked.
To me it looks like if you configure inet/inet6 or inet6/inet, glusterd still only listens on ipv4.
Comment 10 Philipp Schmid 2011-07-23 14:00:12 EDT
I cannot mount a volume over IPv6.

I'm trying to run: 

mount -t glusterfs 2a01:4f8:120:4ffe::13:/test-volume /mnt/

But I get: Mount failed. Please check the log file for more details.

In the log file I see:

I [rpc-clnt.c:1543:rpc_clnt_reconfig] 0-test-volume-client-1: changing port to 24009 (from 0)
I [rpc-clnt.c:1543:rpc_clnt_reconfig] 0-test-volume-client-0: changing port to 24009 (from 0)
E [socket.c:1685:socket_connect_finish] 0-test-volume-client-1: connection to 2a01:4f8:130:3fff:a00:27ff:fe42:f373:24007 failed (Connection refused)
E [socket.c:1685:socket_connect_finish] 0-test-volume-client-0: connection to 2a01:4f8:120:4ffe::13:24007 failed (Connection refused)
E [afr-common.c:3314:afr_notify] 0-test-volume-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
I [fuse-bridge.c:3336:fuse_graph_setup] 0-fuse: switched to graph 0
I [fuse-bridge.c:2924:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16
W [fuse-bridge.c:416:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected)

Now if I run netstat -an I see that 24009 is a IPv4 port instead of a IPv6 port:

root@storage1:/etc/glusterd# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:24009           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp6       0      0 :::24007                :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 2a01:4f8:120:4ffe:24007 2a01:4f8:130:3fff::1022 ESTABLISHED
tcp6       0      0 2a01:4f8:120:4ffe:24007 2a01:4f8:130:3fff::1019 ESTABLISHED
tcp6       0      0 127.0.0.1:24007         127.0.0.1:1023          ESTABLISHED
tcp6       0      0 2a01:4f8:120:4ffe::1022 2a01:4f8:130:3fff:24007 ESTABLISHED
Comment 11 Philipp Schmid 2011-07-23 14:09:39 EDT
I got managed to mount the volume over IPv6 if I add the following:

  option transport.address-family inet6

to

  test-volume.2a01:4f8:120:4ffe::13.storage-gluster-test-volume.vol
  test-volume.2a01:4f8:130:3fff:a00:27ff:fe42:f373.storage-gluster-test-volume.vol

in the folder: etc/glusterd/vols/test-volume

It looks like it should use the same transport.address-family setting as in: 

  /usr/local/etc/glusterfs/glusterd.vol
Comment 12 Marcel Pennewiß 2011-07-24 13:28:47 EDT
(In reply to comment #11)
> It looks like it should use the same transport.address-family setting as in: 
>   /usr/local/etc/glusterfs/glusterd.vol

But this could not be set via cli (gluster volume set ...). Changing volfile manually works also fine here ...
Comment 13 Gaurav 2011-07-24 20:58:51 EDT
glusterd is either listening on ipv4 or ipv6, it will not listen on both...
We are not handling it for inet/inet6 and inet6/inet in the code...have to fix it.
Comment 14 Marcel Pennewiß 2011-07-24 23:23:35 EDT
(In reply to comment #13)
> glusterd is either listening on ipv4 or ipv6, it will not listen on both...
> We are not handling it for inet/inet6 and inet6/inet in the code...have to fix
> it.

After adding option "transport.address-family inet6" to volfiles (on server), glusterfsd listens on tcp6-socket. And IPv4-Clients could also connect (our storage-hosts has only A-Record in DNS for now)...

[2011-07-25 08:15:28.926627] I [server-handshake.c:534:server_setvolume] 0-web-1-www-server: accepted client from 10.42.75.40:1021
[2011-07-25 08:15:28.929903] I [server-handshake.c:534:server_setvolume] 0-web-1-www-server: accepted client from 10.42.75.41:1023

netstat:
tcp6       0      0 :::24009                :::*                    LISTEN      14238/glusterfsd    
tcp6       0      0 :::24010                :::*                    LISTEN      14242/glusterfsd    
tcp6       0      0 10.42.70.11:24009       10.42.75.40:1021        ESTABLISHED 14238/glusterfsd    
tcp6       0      0 10.42.70.11:24009       10.42.75.41:1023        ESTABLISHED 14238/glusterfsd 

But adding any option through gluster-cli, the address-family-config was removed :(
Comment 15 Amar Tumballi 2011-07-24 23:26:54 EDT
(In reply to comment #14)
>
> But adding any option through gluster-cli, the address-family-config was
> removed :(


Thanks for letting us know about this. We are working on fixing the issue. You should have the fix in next release. For now, you need to hand-edit the volume files.

Regards,
Amar
Comment 16 Anand Avati 2011-07-27 01:40:27 EDT
CHANGE: http://review.gluster.com/102 (Change-Id: I0a22b2cc5a0ea1a57633fddabad54aca7b3d4e86) merged in master by Anand Avati (avati@gluster.com)
Comment 17 Gaurav 2011-08-03 08:23:53 EDT
*** Bug 2851 has been marked as a duplicate of this bug. ***
Comment 18 Krishna Srinivas 2011-08-11 03:54:28 EDT
(In reply to comment #16)
> CHANGE: http://review.gluster.com/102 (Change-Id:
> I0a22b2cc5a0ea1a57633fddabad54aca7b3d4e86) merged in master by Anand Avati
> (avati@gluster.com)

This patch causes glusterd not to start on systems without IPv6 support because of the following code:
@@ -412,9 +412,20 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this,
                 ret = -1;
                 goto out;
         }
+        /* IPV6 server can handle both ipv4 and ipv6 clients */
+        for (rp = res; rp != NULL; rp = rp->ai_next) {
+                if (rp->ai_addr == NULL)
+                        continue;
+                if (rp->ai_family == AF_INET6) {
+                        memcpy (addr, rp->ai_addr, rp->ai_addrlen);
+                        *addr_len = rp->ai_addrlen;
+                }
+        }
 
-        memcpy (addr, res->ai_addr, res->ai_addrlen);
-        *addr_len = res->ai_addrlen;
+        if (!(*addr_len)) {
+                memcpy (addr, res->ai_addr, res->ai_addrlen);
+                *addr_len = res->ai_addrlen;
+        }

when listen_hostis NULL getaddrinfo returns addrinfo for both ipv4 and ipv6 because of which addr->sa_family will now be AF_INET6 causing listen() on centos machines without IPv6 support to fail.

Without this patch addr->sa_family would be AF_INET as first entry in the "res" list would be addrinfo for IPv4

If we make listen_host "localhost" instead of NULL, getaddinfo correctly returns just addrinfo for ipv4. Will send a patch but not sure if that is the best way to fix it.
Comment 19 Anand Avati 2011-09-13 00:04:35 EDT
CHANGE: http://review.gluster.com/405 (-----) merged in master by Vijay Bellur (vijay@gluster.com)
Comment 20 Gaurav 2011-09-14 00:25:41 EDT
(In reply to comment #19)
> CHANGE: http://review.gluster.com/405 (-----) merged in master by Vijay Bellur
> (vijay@gluster.com)

This change will not work if family is AF_UNSPEC at the client side, since with AF_UNSPEC client could get either of IPV4 or IPV6 address to connect to the remote server, so remote server should be capable of accepting both IPV4 and IPV6 connections. In case of AI_ADDRCONFIG server will listen on IPV4 if no external IPV6 address is configured and it won't be able to accept IPV6 connection. Instead of this if we always listen on IPV6, then both IPV4 and IPV6 could connect.
Comment 21 Amar Tumballi 2011-09-14 01:28:39 EDT
Gaurav,

Always giving a 'IPv6' address won't work in a setup where IPv6 is disabled. listen() will fail with unsupported address-family error.

The right fix is to consider 'AI_ADDRCONFIG' on client part also while resolving to a host. For this, I have sent a fix wrt bug 765280. If there are no other open items with this bug, you can close the bug once that patch goes in.

Note You need to log in before you can comment on or make changes to this bug.