Red Hat Bugzilla – Bug 1292749
Friend update floods can render the cluster incapable of handling other commands
Last modified: 2016-06-16 09:51:17 EDT
A flood of glusterd friend updates happen whenever a glusterd restarts and re-establishes all it's connections.
In a large cluster (100s) nodes, this would go on for several minutes. During this period the cluster isn't able to respond to commands. Simple local commands, like `gluster volume list` will take relatively very long time to complete.
When a large number of nodes come back up simultaneously, say due to a network problem, this flood can last for a long time, longer than expected.
REVIEW: http://review.gluster.org/12999 (glusterd: reduce friend update flood) posted (#1) for review on master by Kaushal M (firstname.lastname@example.org)
REVIEW: http://review.gluster.org/12999 (glusterd: reduce friend update flood) posted (#2) for review on master by Kaushal M (email@example.com)
COMMIT: http://review.gluster.org/12999 committed in master by Atin Mukherjee (firstname.lastname@example.org)
Author: Kaushal M <email@example.com>
Date: Thu Dec 17 11:13:36 2015 +0530
glusterd: reduce friend update flood
When in a befriended state, glusterd would broadcast friend updates to
all other peers whenver a ACC or LOCAL_ACC event occurred.
When a downed glusterd came back up and established connections again,
this lead to a flood of friend updates to happen on the order of N^2 (N
is the number of peers in the cluster)
In larger clusters this was problematic, and could lead to very long
times for the cluster to settle down when a peer came back up. Multiple
peers coming back up at the same time would compound the problem.
Broadcasting of friend updates doesn't have much use in places other
that during a peer probe. Instead of broadcasting friend updates on
connection re-establishment, updates can just be exchanged between the
peers involved in the connection.
This patch changes the glusterd friend state-machine to send updates
only to the required peer for ACC or LOCAL_ACC events when in befriended
state. The number of updates sent now is in the order of N.
For a 10 node cluster, the number of updates reduced by 5 times. When
creating the 10 node cluster, the updates reduced from ~500 to ~150.
When a glusterd restarted, the number of exchanges reduced from ~160 to
Signed-off-by: Kaushal M <firstname.lastname@example.org>
Reviewed-by: Atin Mukherjee <email@example.com>
Tested-by: NetBSD Build System <firstname.lastname@example.org>
Tested-by: Gluster Build System <email@example.com>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.
glusterfs-3.8.0 has been announced on the Gluster mailinglists , packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist  and the update infrastructure for your distribution.