Bug 1310445

+++ This bug was initially created as a clone of Bug #1117886 +++

+++ This bug was initially created as a clone of Bug #922801 +++

Description of problem:

  As reported on IRC by alex88, gluster isn't resolving hosts that have
  only IPv6 DNS entries (ie AAAA instead of A).

    [2013-03-18 10:23:42.211253] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
    [2013-03-18 10:23:42.213462] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
    [2013-03-18 10:23:42.213537] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host storage.site.com
    [2013-03-18 10:23:42.213609] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
    [2013-03-18 10:23:42.213649] I [glusterfsd-mgmt.c:1790:mgmt_rpc_notify] 0-glusterfsd-mgmt: -1 connect attempts left
    [2013-03-18 10:23:42.213846] W [glusterfsd.c:831:cleanup_and_exit] (-->/usr/sbin/glusterfs(glusterfs_mgmt_init+0x1ff) [0x7f38a41a41af] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f38a3af0922] (-->/usr/sbin/glusterfs(+0xd486) [0x7f38a41a4486]))) 0-: received signum (1), shutting down
    [2013-03-18 10:23:42.213905] I [fuse-bridge.c:4648:fini] 0-fuse: Unmounting '/mnt/site-development/'.


Version-Release number of selected component (if applicable):

  glusterfs 3.3.1 built on Feb 21 2013 03:24:40
  "got by the ubuntu ppa"


How reproducible:

  Every time.


Steps to Reproduce:

  1. Install gluster.
  2. Attempt to use a host that only has AAAA DNS records.
  3. Fails here.

--- Additional comment from Joe Julian on 2014-04-18 20:15:50 EDT ---

We need IPv6 for 3.6. Can we make this a priority for this cycle?

--- Additional comment from Justin Clift on 2014-04-18 21:59:30 EDT ---

It'll probably depend on the code for "Better Peer Identification" getting in first:

  http://www.gluster.org/community/documentation/index.php/Features/Better_peer_identification

If that gets in with sufficient lead time, it might be possible to add IPv6 support for 3.6 as well.  Probably need to ask Kaushal if he thinks its feasible.

--- Additional comment from Anders Blomdell on 2014-06-11 06:44:36 EDT ---

This is still a problem with 3.5.0, it looks like almost all ipv6 support got ripped out with commit 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff :-(

--- Additional comment from Adam Huffman on 2014-06-12 08:19:18 EDT ---

I see similar problems in a dual-stack setup.

--- Additional comment from Anders Blomdell on 2014-06-12 10:16:26 EDT ---

Reverting 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff (with one small manual merge) made IPv6 work for me. :-)

Let's hope it gets into 3.6.

--- Additional comment from Anders Blomdell on 2014-07-09 11:20:53 EDT ---

Reinstating ipv6 support by reverting 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff
    
Make sure to check for regressions in https://bugzilla.redhat.com/show_bug.cgi?id=764655

--- Additional comment from Anand Avati on 2014-07-10 14:37:12 EDT ---

REVIEW: http://review.gluster.org/8292 (Reinstating ipv6 support by reverting 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff) posted (#2) for review on master by Anders Blomdell (anders.blomdell.se)

--- Additional comment from Anand Avati on 2014-07-10 14:39:17 EDT ---

REVIEW: http://review.gluster.org/8292 (Reinstate ipv6 support) posted (#3) for review on master by Anders Blomdell (anders.blomdell.se)

--- Additional comment from Anand Avati on 2014-07-11 12:02:43 EDT ---

REVIEW: http://review.gluster.org/8292 (Reinstate ipv6 support) posted (#4) for review on master by Anders Blomdell (anders.blomdell.se)

--- Additional comment from Gulf Zhou on 2014-12-12 03:20:36 EST ---

(In reply to Anders Blomdell from comment #1)
> Reinstating ipv6 support by reverting
> 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff
>     
> Make sure to check for regressions in
> https://bugzilla.redhat.com/show_bug.cgi?id=764655

Have this been done on 3.5 branch? Thanks.

--- Additional comment from Anders Blomdell on 2014-12-12 04:05:10 EST ---

(In reply to Gulf Zhou from comment #5)
> Have this been done on 3.5 branch? Thanks.

No, and probably shouldn't. The problem was more complex than I anticipated, the problem occurs in multiple places, and the lookup of hosts and connecting to them is unfortunately decoupled, which means that replacing gethostbyname with getaddrinfo will (always?) return the IPv6 address, which means that hosts without IPv6 connectivity will be hosed (this was the AFAICT the [unexplained/unexplored] reason for removing IPv6 support in the first place.

My current plan is to wait for:

http://supercolony.gluster.org/pipermail/gluster-devel/2014-December/043180.html

to get some momentum before I look into it again, since it looks like it has the potential to provide the right decoupling for network connections.

--- Additional comment from Gulf Zhou on 2014-12-14 20:18:09 EST ---

Thanks, Anders. Read through the thread. The IPv6 support is very important feature for us. How will this be prioritized? Don't have the feeling when it can be solved. Any comments?

--- Additional comment from Gulf Zhou on 2015-03-12 21:20:57 EDT ---

Hi, Anders

   Is there any update for the IPv6 support fix? 

   Thanks.

--- Additional comment from Anders Blomdell on 2015-03-13 08:10:51 EDT ---

Waiting for 4.6 (or 4.7) to get stable enough in my lab before I continue with IPv6, :-/

--- Additional comment from Anders Blomdell on 2015-03-13 08:12:25 EDT ---

I meant: 3.6 or 3.7

--- Additional comment from Nithin on 2015-06-11 14:33:11 EDT ---

Hi Niels & Anders,

Can I contribute to this bug fix ? I've worked on Gluster IPv6 functionality bugs in 3.3.2 in my past organization and was able to successfully bring up gluster on IPv6 link local addresses as well.

Please find my work in progress patch. I was successfully able to create volumes with 3 peers and add bricks. I'll continue testing other basic functionality and see when needs to be modified.

Brief info about the patch:
Here I'm trying to use "transport.address-family" option in /etc/glusterfs/glusterd.vol file and then propagate the same to server and client vol files and their translators.

In this way when user mentions "transport.address-family inet6" in its glusterd.vol file, all glusterd servers open AF_INET6 sockets and then the same information is stored in glusterd_volinfo and used when generating vol config files.
 
-thanks
Nithin

--- Additional comment from Nithin on 2015-06-11 14:34:32 EDT ---



--- Additional comment from Joachim Tingvold on 2015-08-12 13:42:53 EDT ---

Hi,

So, trying to get glusterfs working in an IPv6-only environment. Tried two freshly installed CentOS-boxes, both having only IPv6-adresses and quad-A DNS-records.

Tried latest glusterfs 3.5, 3.6 and 3.7. In all of them, glusterd by-default only listens to IPv4 (tcp 24007), hence initial 'probe' doesn't work.

Am I understanding this bug correctly that _all_ IPv6-support has been (temporarily) removed until this bug is resolved?

--- Additional comment from Nithin on 2015-08-15 11:29:23 EDT ---

(In reply to Joachim Tingvold from comment #13)
> Hi,
> 
> So, trying to get glusterfs working in an IPv6-only environment. Tried two
> freshly installed CentOS-boxes, both having only IPv6-adresses and quad-A
> DNS-records.
> 
> Tried latest glusterfs 3.5, 3.6 and 3.7. In all of them, glusterd by-default
> only listens to IPv4 (tcp 24007), hence initial 'probe' doesn't work.
> 
> Am I understanding this bug correctly that _all_ IPv6-support has been
> (temporarily) removed until this bug is resolved?

Yes, basically this bug is to fix issues that are breaking IPv6 functionality.
Most of the support code is already there but is broken probably due to lack of testing focus on IPv6.

--- Additional comment from Gulf Zhou on 2015-08-18 02:51:09 EDT ---

Hi, Nithin

     Are the patch you proposed merged back to the glusterfs branches? Or is there any plan to merge it? 


     Thanks.

--- Additional comment from Nithin on 2015-08-19 04:06:43 EDT ---

Hi Gulf,

I have the patch that is almost same as attached in one of my above comments.
Basic functionality is working, i'm trying to validate if there are any errors and I'm yet to run regression. Once done, I'll raise a review.

--- Additional comment from Anand Avati on 2015-08-22 07:24:47 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: transport address-family inet6 not working) posted (#1) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Nithin on 2015-08-23 01:48:39 EDT ---



--- Additional comment from Nithin on 2015-08-23 01:49:12 EDT ---



--- Additional comment from Anand Avati on 2015-08-27 14:23:24 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: transport address-family inet6 not working) posted (#2) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Nithin on 2015-08-27 14:27:49 EDT ---



--- Additional comment from Nithin on 2015-08-27 14:28:17 EDT ---



--- Additional comment from Nithin on 2015-08-27 14:28:46 EDT ---



--- Additional comment from Vijay Bellur on 2015-09-09 20:48:57 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: transport address-family inet6 not working) posted (#3) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Vijay Bellur on 2015-09-13 12:08:14 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support) posted (#4) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Nithin on 2015-09-13 12:23:56 EDT ---



--- Additional comment from Nithin on 2015-09-13 12:24:38 EDT ---



--- Additional comment from Vijay Bellur on 2015-09-15 02:24:06 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support) posted (#5) for review on master by Vijay Bellur (vbellur)

--- Additional comment from Kaleb KEITHLEY on 2015-10-22 11:40:20 EDT ---

pre-release version is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.

--- Additional comment from Vijay Bellur on 2015-11-11 01:26:34 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support) posted (#6) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Vijay Bellur on 2015-11-15 12:06:16 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support) posted (#7) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Vijay Bellur on 2015-12-03 19:18:58 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support) posted (#8) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Vijay Bellur on 2015-12-06 04:34:08 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support) posted (#9) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Vijay Bellur on 2016-01-31 11:02:26 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support) posted (#10) for review on master by Nithin Kumar D (nithind1988)

--- Additional comment from Anoop C S on 2016-02-17 03:59:52 EST ---

Re-opening the BZ because we have patches posted which has not yet merged.

--- Additional comment from Vijay Bellur on 2016-02-20 12:16:48 EST ---

COMMIT: http://review.gluster.org/11988 committed in master by Jeff Darcy (jdarcy) 
------
commit 46bd29e0f2a7fc9278068a06d12066d614f365ec
Author: Nithin D <nithind1988>
Date:   Sun Nov 15 22:14:43 2015 +0530

    glusterd: Bug fixes for IPv6 support
    
    Problem:
    Glusterd not working using ipv6 transport. The idea is with proper glusterd.vol configuration,
    1. glusterd needs to listen on default port (240007) as IPv6 TCP listner.
    2. Volume creation/deletion/mounting/add-bricks/delete-bricks/peer-probe
       needs to work using ipv6 addresses.
    3. Bricks needs to listen on ipv6 addresses.
    All the above functionality is needed to say that glusterd supports ipv6 transport and this is broken.
    
    Fix:
    When "option transport.address-family inet6" option is present in glusterd.vol
    file, it is made sure that glusterd creates listeners using ipv6 sockets only and also the same information is saved
    inside brick volume files used by glusterfsd brick process when they are starting.
    
    Tests Run:
    Regression tests using ./run-tests.sh
        IPv4: Ran manually till tests/basic/rpm.t .
        IPv6: (Need to add the above mentioned config and also add an entry for "hostname ::1" in /etc/hosts)
            Started failing at ./tests/basic/glusterd/arbiter-volume-probe.t and ran successfully till here
    
    Unit Tests using Ipv6
        peer probe
        add-bricks
        remove-bricks
        create volume
        replace-bricks
        start volume
        stop volume
        delete volume
    
    Change-Id: Iebc96e6cce748b5924ce5da17b0114600ec70a6e
    BUG: 1117886
    Signed-off-by: Nithin D <nithind1988>
    Reviewed-on: http://review.gluster.org/11988
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Jeff Darcy <jdarcy>