Bug 1119547

Summary:	Improvements to peer identification in glusterd
Product:	[Community] GlusterFS	Reporter:	Kaushal <kaushal>
Component:	glusterd	Assignee:	Kaushal <kaushal>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, gluster-bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
URL:	http://www.gluster.org/community/documentation/index.php/Features/Better_peer_identification
Whiteboard:
Fixed In Version:	glusterfs-3.6.0beta1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-11-11 08:37:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Kaushal 2014-07-15 03:46:43 UTC

The glusterd peer identification mechanism is not really good at the moment. This causes problems when a mixture of hostnames (fqdns and shortnames) and IPs are used. Glusterd also does not correctly identify peers when using multiple networks.

More information regarding the issue and the proposed solution for it can be found at http://www.gluster.org/community/documentation/index.php/Features/Better_peer_identification

There are lots of other bugs which have been filed due to the issues caused by glusterd's peer identification. I'll be marking them as dependent on this bug as I come across them.

Comment 1 Anand Avati 2014-07-15 03:48:10 UTC

REVIEW: http://review.gluster.org/8238 (glusterd: Improvements peer identification) posted (#4) for review on master by Kaushal M (kaushal)

Comment 2 Anand Avati 2014-07-15 09:16:54 UTC

REVIEW: http://review.gluster.org/8238 (glusterd: Improvements to peer identification) posted (#5) for review on master by Kaushal M (kaushal)

Comment 3 Anand Avati 2014-07-15 13:59:41 UTC

COMMIT: http://review.gluster.org/8238 committed in master by Krishnan Parthasarathi (kparthas) 
------
commit fde2b73746e73ea16ba246b8261b552ed2d894dd
Author: Kaushal M <kaushal>
Date:   Tue Jul 15 09:06:58 2014 +0530

    glusterd: Improvements to peer identification
    
    This patch improves the peer identification mechanism in glusterd and
    lays down the framework for further improvements, including better multi
    network support in glusterd.
    
    This patch mainly does two things,
    1. Extend the peerinfo object to store a list of addresses instead of a
    single hostname as it does now. This also includes changes to make the
    peer update behaviour of 'peer probe' to add to the list.
    
    2. Improve glusterd_friend_find_by_hostname() to perform better matching
    of hostnames. glusterd_friend_find_by_hostname() now does and initial
    quick string compare against all the peer addresses known to glusterd,
    after which it tries a more thorough search using address resolution and
    matching the struc sockaddr's.
    
    The above two changes together improve the peer identification situation
    in glusterd a lot.
    
    More information regarding the problem this patch attempts to resolve
    and the approach chosen can be found at
    http://www.gluster.org/community/documentation/index.php/Features/Better_peer_identification
    
    This commit is a squashed commit of the following changes, the
    development branch of which can be viewed at,
    https://github.com/kshlm/glusterfs/tree/better-peer-identification or,
    https://forge.gluster.org/~kshlm/glusterfs-core/kshlms-glusterfs/commits/better-peer-identification
    
    commit 198f86e60fab74faf082eaa02657a4d8f60b92f0
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 15 14:34:06 2014 +0530
    
        Update gluster.8
    
    commit 35d597f3a6b3248373e727f7b7e889c92554d56c
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 15 09:01:01 2014 +0530
    
        Address review comments
    
        https://review.gluster.org/#/c/8238/3
    
    commit 47b5331e17304477322bd2daed5bbed503c34ca1
    Merge: c71b12c 78128af
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 15 08:41:39 2014 +0530
    
        Merge branch 'master' into better-peer-identification
    
    commit c71b12c164330e8d19d1df4734ab34ef9a8caad2
    Merge: 57bc9de 0f5719a
    Author: Kaushal M <kaushal>
    Date:   Thu Jul 10 19:50:19 2014 +0530
    
        Merge branch 'master' into better-peer-identification
    
    commit 57bc9de9e4f49ff2b1620df9906cda50a3527a25
    Author: Kaushal M <kaushal>
    Date:   Thu Jul 10 19:49:08 2014 +0530
    
        More fixes to review comments
    
    commit 5482cc363a687a9e246a0780ec88acd53e218501
    Author: Kaushal M <kaushal>
    Date:   Thu Jul 10 18:36:40 2014 +0530
    
        Code refactoring in peer-utils based on review comments
    
        https://review.gluster.org/#/c/8238/2/xlators/mgmt/glusterd/src/glusterd-peer-utils.c
    
    commit 89b22c34757178f64d5fbaffa31e6302f841c060
    Author: Kaushal M <kaushal>
    Date:   Thu Jul 10 12:30:00 2014 +0530
    
        Hostnames in peer status
    
    commit 63ebf9485cf50d736cf640238a1ab241671fcaf1
    Merge: c8c8fdd f5f9721
    Author: Kaushal M <kaushal>
    Date:   Thu Jul 10 12:06:33 2014 +0530
    
        Merge remote-tracking branch 'origin/master' into better-peer-identification
    
    commit c8c8fdd2104b5b6b8a1af739b1dd952b74e6dd66
    Author: Kaushal M <kaushal>
    Date:   Wed Jul 9 18:35:27 2014 +0530
    
        Hostnames in xml output
    
    commit 732a92a0167ad7b1d70edbc35ebd8307c2766ae1
    Author: Kaushal M <kaushal>
    Date:   Wed Jul 9 15:12:10 2014 +0530
    
        Add hostnames to cli rsp dict during list-friends
    
    commit fcf43e3e317508f0c225024738a988a4af8e9205
    Merge: c0e2624 72d96e2
    Author: Kaushal M <kaushal>
    Date:   Wed Jul 9 12:53:03 2014 +0530
    
        Merge branch 'master' into better-peer-identification
    
    commit c0e262416728a3c536a8347a216e471eb2251535
    Author: Kaushal M <kaushal>
    Date:   Mon Jul 7 16:11:19 2014 +0530
    
        Use list_for_each_entry_safe when cleaning peer hostnames
    
    commit 6132e60224eb592f3657e535a12a3e72c772da42
    Author: Kaushal M <kaushal>
    Date:   Mon Jul 7 15:52:19 2014 +0530
    
        Fix crash in gd_add_friend_to_dict
    
    commit 88ffa9a508fd5aac0b2a76e6e76487ce0cab786a
    Author: Kaushal M <kaushal>
    Date:   Mon Jul 7 13:19:44 2014 +0530
    
        gd_peerinfo_destroy -> glusterd_peerinfo_destroy
    
    commit 4b36930a715b1e13cd1a77d136ef1cf78a06d574
    Author: Kaushal M <kaushal>
    Date:   Mon Jul 7 12:50:12 2014 +0530
    
        More refactoring
    
    commit ee559b081d608c6501c10ae22166f26eeb65690e
    Author: Kaushal M <kaushal>
    Date:   Mon Jul 7 12:14:40 2014 +0530
    
        Major refactoring of code based on review comments at
        https://review.gluster.org/#/c/8238/1/xlators/mgmt/glusterd/src/glusterd-peer-utils.h
    
    commit e96dbc7bbb05fad2a9c424de41a394b8023fe48d
    Merge: 2613d1d 83c09b7
    Author: Kaushal M <kaushal>
    Date:   Mon Jul 7 09:47:05 2014 +0530
    
        Merge remote-tracking branch 'origin/master' into better-peer-identification
    
    commit 2613d1daebff0c56812de821c06ed4c16bb9d447
    Merge: b242cf6 9a50211
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 15:28:57 2014 +0530
    
        Merge remote-tracking branch 'origin/master' into better-peer-identification
    
    commit b242cf66d95dd3dd5e3975aa430baa6bd74b8a29
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 15:08:18 2014 +0530
    
        Fix a silly mistake, if (ctx->req) => if (ctx->req == NULL)
    
    commit c835ed26433830ceed57289143f596cf60421558
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 14:58:23 2014 +0530
    
        Fix reverse probe.
    
    commit 9ede17f9329b854b02e8ad159f173244789fd08c
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 13:31:32 2014 +0530
    
        Fix friend import for existing peers
    
    commit 891bf74c7350064dfb008d1b7294bcec28d680fd
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 13:08:36 2014 +0530
    
        Set first hostname in peerinfo->hostnames to peerinfo->hostname
    
    commit 9421d6a217381a7427a7d84f369280883ca4297a
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 12:21:40 2014 +0530
    
        Fix gf_asprintf return val check in glusterd_store_peer_write
    
    commit defac978c1d94011ce8195e311839b9ffce057e7
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 11:16:13 2014 +0530
    
        Fix store_retrieve_peers to correctly cleanup.
    
    commit 00a799f5de1121b0cb7421da8285f9407063e1bd
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 10:52:11 2014 +0530
    
        Update address list in glusterd_probe_cbk only when needed.
    
    commit 7a628e8a9c562d85709c69cfa13fb1774c521b75
    Merge: d191985 dc46d5e
    Author: Kaushal M <kaushal>
    Date:   Fri Jul 4 09:24:12 2014 +0530
    
        Merge remote-tracking branch 'origin/master' into better-peer-identification
    
    commit d1919858e6639d2b54d716a61f662d9752ec5ff1
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 1 18:59:49 2014 +0530
    
        gf_compare_addrinfo -> gf_compare_sockaddr
    
    commit 31d8ef730d408f8d9ba8f504fa648f7dcd59da87
    Merge: 93bbede 86ee233
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 1 18:16:13 2014 +0530
    
        Merge remote-tracking branch 'origin/master' into better-peer-identification
    
    commit 93bbedeac5181e29f59b2acd08f638146812ec41
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 1 18:15:16 2014 +0530
    
        Improve glusterd_friend_find_by_hostname
    
        glusterd_friend_find_by_hostname will now do an initial quick search for
        the peerinfo performing string comparisions on the given host string. It
        follows it with a more thorough match, by resolving the addresses and
        comparing addrinfos instead of strings.
    
    commit 2542cdbc45aa9cfcaf1f174686158d5565cdd07b
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 1 17:21:10 2014 +0530
    
        New utility gf_compare_addrinfo
    
    commit 338676e8389a44bd91136eebd110197429c2566c
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 1 14:55:56 2014 +0530
    
        Use gd_peer_has_address instead of strcmp
    
    commit 28d45be51f594328741c44455bd80ac9d64ca501
    Merge: 728266e 991dd5e
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 1 14:54:40 2014 +0530
    
        Merge branch 'master' into better-peer-identification
    
    commit 728266eb16d5f5a4bf36266044425ae164337f99
    Merge: 7d9b87b 2417de9
    Author: Kaushal M <kaushal>
    Date:   Tue Jul 1 09:55:13 2014 +0530
    
        Merge remote-tracking branch 'origin/master' into better-peer-identification
    
    commit 7d9b87b84955ec17daeaf88a3e7462914039430f
    Merge: b890625 e02275c
    Author: Kaushal M <kshlmster>
    Date:   Tue Jul 1 08:41:40 2014 +0530
    
        Merge pull request #4 from vpshastry/better-peer-identification
    
        Better peer identification
    
    commit e02275c52fb83c72ad082c098fd3e432c2b9c526
    Merge: 75ee90d b890625
    Author: Varun Shastry <vshastry>
    Date:   Mon Jun 30 16:44:29 2014 +0530
    
        Merge branch 'better-peer-identification' of https://github.com/kshlm/glusterfs into better-peer-identification-kaushal-github
    
    commit 75ee90d2f272e49b94d24c9ca4571e89a83055ff
    Author: Varun Shastry <vshastry>
    Date:   Mon Jun 30 15:36:10 2014 +0530
    
        glusterd: add to the list if the probed uuid pre-exists
    
        Signed-off-by: Varun Shastry <vshastry>
    
    commit b890625d8164c660695daef3285c67979eef723e
    Merge: 04c5d60 187a7a9
    Author: Kaushal M <kaushal>
    Date:   Mon Jun 30 11:44:13 2014 +0530
    
        Merge remote-tracking branch 'origin/master' into better-peer-identification
    
    commit 04c5d60cb938c8d94b214689580b40abb1b0ffcd
    Merge: 3a5bfa1 e01edb6
    Author: Kaushal M <kshlmster>
    Date:   Sat Jun 28 19:23:33 2014 +0530
    
        Merge pull request #3 from vpshastry/better-peer-identification
    
        glusterd: search through the list of hostnames in the peerinfo
    
    commit 0c64f3346a977f9165ac55a84a1e03c40a7573a7
    Merge: e01edb6 3a5bfa1
    Author: Varun Shastry <vshastry>
    Date:   Sat Jun 28 10:43:29 2014 +0530
    
        Merge branch 'better-peer-identification' of https://github.com/kshlm/glusterfs into better-peer-identification-kaushal-github
    
    commit e01edb63153a1008db70b8fa76ae5b535e099326
    Author: Varun Shastry <vshastry>
    Date:   Fri Jun 27 12:29:36 2014 +0530
    
        glusterd: search through the list of hostnames in the peerinfo
    
        Signed-off-by: Varun Shastry <vshastry>
    
    commit 3a5bfa15855e660db2bfde644727371dd2d618cc
    Merge: cda6d31 371ea35
    Author: Kaushal M <kshlmster>
    Date:   Fri Jun 27 11:31:17 2014 +0530
    
        Merge pull request #1 from vpshastry/better-peer-identification
    
        glusterd: Add hostname to list instead of replaceing upon update
    
    commit 371ea354f198b4182382d5403c5960c0b2add6b6
    Author: Varun Shastry <vshastry>
    Date:   Fri Jun 27 11:24:54 2014 +0530
    
        glusterd: Add hostname to list instead of replaceing upon update
    
        Signed-off-by: Varun Shastry <vshastry>
    
    commit cda6d3152886623ecbf46baf0048ebe0119b30b6
    Author: Kaushal M <kaushal>
    Date:   Thu Jun 26 19:52:52 2014 +0530
    
        Import address lists
    
    commit 6649b54aa0440130c08e827e0a1d1bbfb840eca9
    Author: Kaushal M <kaushal>
    Date:   Thu Jun 26 19:15:37 2014 +0530
    
        Implement export address list
    
    commit 55990034eead92bc9b936240029e460a4bf152d5
    Author: Kaushal M <kaushal>
    Date:   Thu Jun 26 18:11:59 2014 +0530
    
        Use first address in list to when setting up the peer RPC.
    
    commit a35fde8d19b9988eb04c652fb3a5e4f84d90ad00
    Author: Kaushal M <kaushal>
    Date:   Thu Jun 26 18:03:04 2014 +0530
    
        Properly free addresses on glusterd_peer_destroy
    
    commit 1988081db09ac9205f3dc7268cef8be267f3ce8b
    Author: Kaushal M <kaushal>
    Date:   Thu Jun 26 17:52:35 2014 +0530
    
        Restore peerinfo with address list implemented.
    
    commit 66f524d5749a12f4910dd6b06c9d91f37e1d831e
    Author: Kaushal M <kaushal>
    Date:   Mon Jun 23 13:02:23 2014 +0530
    
        Move out all peer related utilities from glusterd-utils to glusterd-peer-utils
    
    commit 14a2a326a4dff11b55490dca2a14f39320931340
    Author: Kaushal M <kaushal>
    Date:   Tue May 27 12:16:41 2014 +0530
    
        Compilation fix
    
    commit c59cd351d0a102d0d5f3ea9001fd33c4edcb262f
    Author: Kaushal M <kaushal>
    Date:   Mon May 5 12:51:11 2014 +0530
    
        Add store support for hostname list
    
    commit b70325f0beb884ad12645ef40185f0bf6cedd741
    Author: Kaushal M <kaushal>
    Date:   Fri May 2 15:58:07 2014 +0530
    
        Add a hostnames list to glusterd_peerinfo_t
    
        glusterd_peerinfo_new will now init this list and add the given hostname
        as the lists first member.
    
    Signed-off-by: Kaushal M <kaushal>
    Signed-off-by: Varun Shastry <vshastry>
    Change-Id: Ief3c5d6d6f16571ee2fab0a45e638b9d6506a06e
    BUG: 1119547
    Reviewed-on: http://review.gluster.org/8238
    Reviewed-by: Atin Mukherjee <amukherj>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Krishnan Parthasarathi <kparthas>
    Tested-by: Krishnan Parthasarathi <kparthas>

Comment 4 Niels de Vos 2014-09-22 12:44:50 UTC

A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 5 Niels de Vos 2014-11-11 08:37:03 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users