Peer probe with any IPv6 address fails. Following errors are seen in the logfile: [2011-02-23 13:16:26.53863] E [name.c:251:af_inet_client_get_remote_sockaddr] management: DNS resolution failed on host 2001::2 [2011-02-23 13:16:26.53932] E [common-utils.c:124:gf_resolve_ip6] resolver: getaddrinfo failed (Address family for hostname not supported)
PATCH: http://patches.gluster.com/patch/7564 in master (Glusterd: IPV6 support for glusterfs.)
Since patch 7564, gluster cannot connect to glusterd on my system. I get: # gluster system:: getwd Connection failed. Please check if gluster daemon is operational. (with any gluster command, not just for this one).
(In reply to comment #2) > Since patch 7564, gluster cannot connect to glusterd on my system. I get: > > # gluster system:: getwd > Connection failed. Please check if gluster daemon is operational. > > (with any gluster command, not just for this one). Strange, the commit scripts performs tests of creating volume, start, stop etc from cli to glusterd with every change, including this one - and it has passed because the commit has gone through.
(In reply to comment #3) > (In reply to comment #2) > > Since patch 7564, gluster cannot connect to glusterd on my system. I get: > > > > # gluster system:: getwd > > Connection failed. Please check if gluster daemon is operational. > > > > (with any gluster command, not just for this one). > > Strange, the commit scripts performs tests of creating volume, start, > stop etc from cli to glusterd with every change, including this one - > and it has passed because the commit has gone through. OK, turned out the error was on my side. The system was not properly set up for IPv6.
(In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > Since patch 7564, gluster cannot connect to glusterd on my system. I get: > > > > > > # gluster system:: getwd > > > Connection failed. Please check if gluster daemon is operational. > > > > > > (with any gluster command, not just for this one). > > > > Strange, the commit scripts performs tests of creating volume, start, > > stop etc from cli to glusterd with every change, including this one - > > and it has passed because the commit has gone through. > > OK, turned out the error was on my side. The system was not properly set up for > IPv6. I still think this is a bug. Connections should still work on ipv4 even if system is not set up for ipv6.
(In reply to comment #5) > (In reply to comment #4) > > (In reply to comment #3) > > > (In reply to comment #2) > > > > Since patch 7564, gluster cannot connect to glusterd on my system. I get: > > > > > > > > # gluster system:: getwd > > > > Connection failed. Please check if gluster daemon is operational. > > > > > > > > (with any gluster command, not just for this one). > > > > > > Strange, the commit scripts performs tests of creating volume, start, > > > stop etc from cli to glusterd with every change, including this one - > > > and it has passed because the commit has gone through. > > > > OK, turned out the error was on my side. The system was not properly set up for > > IPv6. > > I still think this is a bug. Connections should still work on ipv4 even if > system is not set up for ipv6. That's true, but there is a difference between a system which does not support IPv6 and one which has IPv6 enabled but misconfigured. It seems to me that mine was the latter case, please verify and decide if glusterfs can/should do anything about it. This was my /etc/hosts file: http://projects.archlinux.org/svntogit/packages.git/plain/filesystem/trunk/hosts?id=b8efbc3c43bf9c8f60edb684e680567427ea0b30 Nb. with this setup I also had an issue with ab -- it got ECONNREFUSED when trying to test a local http server like "ab http://localhost:8001/" I did the following changes after consulting an Ubuntu hosts file: # curl -s http://projects.archlinux.org/svntogit/packages.git/plain/filesystem/trunk/hosts?id=b8efbc3c43bf9c8f60edb684e680567427ea0b30 | diff -up - /etc/hosts --- - 2011-07-14 12:07:08.887668907 -0700 +++ /etc/hosts 2011-07-14 11:49:13.463000071 -0700 @@ -4,6 +4,14 @@ #<ip-address> <hostname.domain.org> <hostname> 127.0.0.1 localhost.localdomain localhost -::1 localhost.localdomain localhost +#::1 localhost.localdomain localhost + +::1 ip6-localhost ip6-loopback +fe00::0 ip6-localnet +ff00::0 ip6-mcastprefix +ff02::1 ip6-allnodes +ff02::2 ip6-allrouters + + # End of file And now, both ab and gluster behave as expected.
Put it simply, aside hosts file tuning. Should this not work: # gluster --remote-host=::1 system:: getwd ? Because it does not, AFAICS.
(In reply to comment #7) > Put it simply, aside hosts file tuning. > > Should this not work: > > # gluster --remote-host=::1 system:: getwd > > ? Because it does not, AFAICS. I believe that should have worked. Gaurav, can you check this? You can compare gluster's behaviour with that of ssh (which supports IPv6) to decide on such questions. Avati
I tried to setup gluster to work over IPv6, but it just wouldn't listen on IPv6. This is how my glusterd.vol looked like: volume management type mgmt/glusterd option working-directory /etc/glusterd option transport-type socket,rdma option transport.socket.keepalive-time 10 option transport.socket.keepalive-interval 2 option transport.address-family inet/inet6 end-volume Then I tried to listen only on inet6 and it worked. To me it looks like if you configure inet/inet6 or inet6/inet, glusterd still only listens on ipv4.
I cannot mount a volume over IPv6. I'm trying to run: mount -t glusterfs 2a01:4f8:120:4ffe::13:/test-volume /mnt/ But I get: Mount failed. Please check the log file for more details. In the log file I see: I [rpc-clnt.c:1543:rpc_clnt_reconfig] 0-test-volume-client-1: changing port to 24009 (from 0) I [rpc-clnt.c:1543:rpc_clnt_reconfig] 0-test-volume-client-0: changing port to 24009 (from 0) E [socket.c:1685:socket_connect_finish] 0-test-volume-client-1: connection to 2a01:4f8:130:3fff:a00:27ff:fe42:f373:24007 failed (Connection refused) E [socket.c:1685:socket_connect_finish] 0-test-volume-client-0: connection to 2a01:4f8:120:4ffe::13:24007 failed (Connection refused) E [afr-common.c:3314:afr_notify] 0-test-volume-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. I [fuse-bridge.c:3336:fuse_graph_setup] 0-fuse: switched to graph 0 I [fuse-bridge.c:2924:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16 W [fuse-bridge.c:416:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected) Now if I run netstat -an I see that 24009 is a IPv4 port instead of a IPv6 port: root@storage1:/etc/glusterd# netstat -an Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:24009 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp6 0 0 :::24007 :::* LISTEN tcp6 0 0 :::22 :::* LISTEN tcp6 0 0 2a01:4f8:120:4ffe:24007 2a01:4f8:130:3fff::1022 ESTABLISHED tcp6 0 0 2a01:4f8:120:4ffe:24007 2a01:4f8:130:3fff::1019 ESTABLISHED tcp6 0 0 127.0.0.1:24007 127.0.0.1:1023 ESTABLISHED tcp6 0 0 2a01:4f8:120:4ffe::1022 2a01:4f8:130:3fff:24007 ESTABLISHED
I got managed to mount the volume over IPv6 if I add the following: option transport.address-family inet6 to test-volume.2a01:4f8:120:4ffe::13.storage-gluster-test-volume.vol test-volume.2a01:4f8:130:3fff:a00:27ff:fe42:f373.storage-gluster-test-volume.vol in the folder: etc/glusterd/vols/test-volume It looks like it should use the same transport.address-family setting as in: /usr/local/etc/glusterfs/glusterd.vol
(In reply to comment #11) > It looks like it should use the same transport.address-family setting as in: > /usr/local/etc/glusterfs/glusterd.vol But this could not be set via cli (gluster volume set ...). Changing volfile manually works also fine here ...
glusterd is either listening on ipv4 or ipv6, it will not listen on both... We are not handling it for inet/inet6 and inet6/inet in the code...have to fix it.
(In reply to comment #13) > glusterd is either listening on ipv4 or ipv6, it will not listen on both... > We are not handling it for inet/inet6 and inet6/inet in the code...have to fix > it. After adding option "transport.address-family inet6" to volfiles (on server), glusterfsd listens on tcp6-socket. And IPv4-Clients could also connect (our storage-hosts has only A-Record in DNS for now)... [2011-07-25 08:15:28.926627] I [server-handshake.c:534:server_setvolume] 0-web-1-www-server: accepted client from 10.42.75.40:1021 [2011-07-25 08:15:28.929903] I [server-handshake.c:534:server_setvolume] 0-web-1-www-server: accepted client from 10.42.75.41:1023 netstat: tcp6 0 0 :::24009 :::* LISTEN 14238/glusterfsd tcp6 0 0 :::24010 :::* LISTEN 14242/glusterfsd tcp6 0 0 10.42.70.11:24009 10.42.75.40:1021 ESTABLISHED 14238/glusterfsd tcp6 0 0 10.42.70.11:24009 10.42.75.41:1023 ESTABLISHED 14238/glusterfsd But adding any option through gluster-cli, the address-family-config was removed :(
(In reply to comment #14) > > But adding any option through gluster-cli, the address-family-config was > removed :( Thanks for letting us know about this. We are working on fixing the issue. You should have the fix in next release. For now, you need to hand-edit the volume files. Regards, Amar
CHANGE: http://review.gluster.com/102 (Change-Id: I0a22b2cc5a0ea1a57633fddabad54aca7b3d4e86) merged in master by Anand Avati (avati)
*** Bug 2851 has been marked as a duplicate of this bug. ***
(In reply to comment #16) > CHANGE: http://review.gluster.com/102 (Change-Id: > I0a22b2cc5a0ea1a57633fddabad54aca7b3d4e86) merged in master by Anand Avati > (avati) This patch causes glusterd not to start on systems without IPv6 support because of the following code: @@ -412,9 +412,20 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this, ret = -1; goto out; } + /* IPV6 server can handle both ipv4 and ipv6 clients */ + for (rp = res; rp != NULL; rp = rp->ai_next) { + if (rp->ai_addr == NULL) + continue; + if (rp->ai_family == AF_INET6) { + memcpy (addr, rp->ai_addr, rp->ai_addrlen); + *addr_len = rp->ai_addrlen; + } + } - memcpy (addr, res->ai_addr, res->ai_addrlen); - *addr_len = res->ai_addrlen; + if (!(*addr_len)) { + memcpy (addr, res->ai_addr, res->ai_addrlen); + *addr_len = res->ai_addrlen; + } when listen_hostis NULL getaddrinfo returns addrinfo for both ipv4 and ipv6 because of which addr->sa_family will now be AF_INET6 causing listen() on centos machines without IPv6 support to fail. Without this patch addr->sa_family would be AF_INET as first entry in the "res" list would be addrinfo for IPv4 If we make listen_host "localhost" instead of NULL, getaddinfo correctly returns just addrinfo for ipv4. Will send a patch but not sure if that is the best way to fix it.
CHANGE: http://review.gluster.com/405 (-----) merged in master by Vijay Bellur (vijay)
(In reply to comment #19) > CHANGE: http://review.gluster.com/405 (-----) merged in master by Vijay Bellur > (vijay) This change will not work if family is AF_UNSPEC at the client side, since with AF_UNSPEC client could get either of IPV4 or IPV6 address to connect to the remote server, so remote server should be capable of accepting both IPV4 and IPV6 connections. In case of AI_ADDRCONFIG server will listen on IPV4 if no external IPV6 address is configured and it won't be able to accept IPV6 connection. Instead of this if we always listen on IPV6, then both IPV4 and IPV6 could connect.
Gaurav, Always giving a 'IPv6' address won't work in a setup where IPv6 is disabled. listen() will fail with unsupported address-family error. The right fix is to consider 'AI_ADDRCONFIG' on client part also while resolving to a host. For this, I have sent a fix wrt bug 765280. If there are no other open items with this bug, you can close the bug once that patch goes in.