Bug 1322805
Summary: | [scale] Brick process does not start after node reboot | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Atin Mukherjee <amukherj> | |
Component: | glusterd | Assignee: | Atin Mukherjee <amukherj> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
Severity: | unspecified | Docs Contact: | ||
Priority: | unspecified | |||
Version: | mainline | CC: | bugs, nerawat, prasanna.kalever, sasundar, smohan, storage-qa-internal | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.8rc2 | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | ||
Clone Of: | 1322306 | |||
: | 1323564 1333711 (view as bug list) | Environment: | ||
Last Closed: | 2016-06-16 14:02:34 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1322306 | |||
Bug Blocks: | 1323564, 1333711 |
Description
Atin Mukherjee
2016-03-31 11:13:41 UTC
REVIEW: http://review.gluster.org/13865 (glusterd: Do not persist brickinfo ports) posted (#2) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/13865 (glusterd: Allocate fresh port on brick (re)start) posted (#3) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/13865 (glusterd: Allocate fresh port on brick (re)start) posted (#4) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/13865 (glusterd: Allocate fresh port on brick (re)start) posted (#5) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/13865 (glusterd: Allocate fresh port on brick (re)start) posted (#6) for review on master by Atin Mukherjee (amukherj) COMMIT: http://review.gluster.org/13865 committed in master by Jeff Darcy (jdarcy) ------ commit 34899d71f21fd2b4c523b68ffb2d7c655c776641 Author: Atin Mukherjee <amukherj> Date: Thu Mar 31 11:01:53 2016 +0530 glusterd: Allocate fresh port on brick (re)start There is no point of using the same port through the entire volume life cycle for a particular bricks process since there is no guarantee that the same port would be free and no other application wouldn't consume it in between the glusterd/volume restart. We hit a race where on glusterd restart the daemon services start followed by brick processes and the time brick process tries to bind with the port which was allocated by glusterd before a restart is been already consumed by some other client like NFS/SHD/... Note : This is a short term solution as here we reduce the race window but don't eliminate it completely. As a long term solution the port allocation has to be done by glusterfsd and the same should be communicated back to glusterd for book keeping Change-Id: Ibbd1e7ca87e51a7cd9cf216b1fe58ef7783aef24 BUG: 1322805 Signed-off-by: Atin Mukherjee <amukherj> Reviewed-on: http://review.gluster.org/13865 Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy> REVIEW: http://review.gluster.org/13989 (Revert "glusterd: Allocate fresh port on brick (re)start") posted (#1) for review on master by Gaurav Kumar Garg (ggarg) REVIEW: http://review.gluster.org/13990 (glusterd: use new brick port if existing brick port occupied by other process) posted (#1) for review on master by Gaurav Kumar Garg (ggarg) REVIEW: http://review.gluster.org/13989 (Revert "glusterd: Allocate fresh port on brick (re)start") posted (#2) for review on master by Gaurav Kumar Garg (ggarg) REVIEW: http://review.gluster.org/13990 (glusterd: use new brick port if existing brick port is occupied by other process) posted (#2) for review on master by Gaurav Kumar Garg (ggarg) REVIEW: http://review.gluster.org/13998 (rpc: define client port range) posted (#1) for review on master by Prasanna Kumar Kalever (pkalever) COMMIT: http://review.gluster.org/13989 committed in master by Atin Mukherjee (amukherj) ------ commit 401d591de168fdb648663f01c4c4f8ed60777558 Author: Gaurav Kumar Garg <garg.gaurav52> Date: Wed Apr 13 14:38:11 2016 +0530 Revert "glusterd: Allocate fresh port on brick (re)start" This reverts commit 34899d7 Commit 34899d7 introduced a change, where restarting a volume or rebooting a node result into fresh allocation of brick port. In production environment generally administrator makes firewall configuration for a range of ports for a volume. With commit 34899d7, on rebooting of node or restarting a volume might result into volume start fail because firewall might block fresh allocated port of a brick and also it will be difficult in testing because of fresh allocation of port. Change-Id: I7a90f69e8c267a013dc906b5228ca76e819d84ad BUG: 1322805 Signed-off-by: Gaurav Kumar Garg <ggarg> Reviewed-on: http://review.gluster.org/13989 Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> REVIEW: http://review.gluster.org/13998 (rpc: define client port range) posted (#2) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/13990 (glusterd: Allocate new brick port if port is already in use) posted (#3) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/13990 (glusterd: Allocate new brick port if port is already in use) posted (#4) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/13990 (glusterd: Allocate new port for the brick if the port is already in use) posted (#5) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/13998 (rpc: define client port range) posted (#3) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/14043 (glusterd: add defence mechanism to avoid brick port clashes) posted (#1) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/14043 (glusterd: add defence mechanism to avoid brick port clashes) posted (#2) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/14080 (glusterd: try to connect on GF_PMAP_PORT_FOREIGN aswell) posted (#1) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/13998 (rpc: define client port range) posted (#4) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/13998 (rpc: define client port range) posted (#5) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/14080 (glusterd: try to connect on GF_PMAP_PORT_FOREIGN aswell) posted (#2) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/14043 (glusterd: add defence mechanism to avoid brick port clashes) posted (#3) for review on master by Prasanna Kumar Kalever (pkalever) COMMIT: http://review.gluster.org/14080 committed in master by Atin Mukherjee (amukherj) ------ commit 3af9b53d13a88b93d026d599c9f86f8bb1845b6c Author: Prasanna Kumar Kalever <prasanna.kalever> Date: Tue Apr 26 18:40:04 2016 +0530 glusterd: try to connect on GF_PMAP_PORT_FOREIGN aswell This patch fix couple of things mentioned below: 1. previously we use to try to connect on only GF_PMAP_PORT_FREE in the pmap_registry_alloc(), it could happen that some foreign process would have freed the port by this time ?, hence it is worth giving a try on GF_PMAP_PORT_FOREIGN ports as well instead of wasting them all. 2. fix pmap_registry_remove() to mark the port asGF_PMAP_PORT_FREE 3. added useful comments on gf_pmap_port_type enum members Change-Id: Id2aa7ad55e76ae3fdece21bed15792525ae33fe1 BUG: 1322805 Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever> Reviewed-on: http://review.gluster.org/14080 Tested-by: Prasanna Kumar Kalever <pkalever> Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> REVIEW: http://review.gluster.org/14043 (glusterd: add defence mechanism to avoid brick port clashes) posted (#4) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/14043 (glusterd: add defence mechanism to avoid brick port clashes) posted (#5) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/14043 (glusterd: add defence mechanism to avoid brick port clashes) posted (#6) for review on master by Prasanna Kumar Kalever (pkalever) REVIEW: http://review.gluster.org/13998 (rpc: define client port range) posted (#6) for review on master by Prasanna Kumar Kalever (pkalever) COMMIT: http://review.gluster.org/13998 committed in master by Raghavendra G (rgowdapp) ------ commit 50ca12326bde53895b7227b12ef4d74a69e2a0f9 Author: Prasanna Kumar Kalever <prasanna.kalever> Date: Thu Apr 14 19:02:19 2016 +0530 rpc: define client port range Problem: when bind-insecure is 'off', all the clients bind to secure ports, if incase all the secure ports exhaust the client will no more bind to secure ports and tries gets a random port which is obviously insecure. we have seen the client obtaining a port number in the range 49152-65535 which are actually reserved as part of glusterd's pmap_registry for bricks, hence this will lead to port clashes between client and brick processes. Solution: If we can define different port ranges for clients incase where secure ports exhaust, we can avoid the maximum port clashes with in gluster processes. Still we are prone to have clashes with other non-gluster processes, but the chances being very low in the rhgs Env, but that's a different story on its own, which will be handled in upcoming patches. Change-Id: Ib5ce05991aa1290ccb17f6f04ffd65caf411feaf BUG: 1322805 Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever> Reviewed-on: http://review.gluster.org/13998 Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> Reviewed-by: Raghavendra G <rgowdapp> REVIEW: http://review.gluster.org/14043 (glusterd: add defence mechanism to avoid brick port clashes) posted (#7) for review on master by Prasanna Kumar Kalever (pkalever) COMMIT: http://review.gluster.org/14043 committed in master by Raghavendra G (rgowdapp) ------ commit 89759de7e47d99eb1fca2763931b4f33ac765173 Author: Prasanna Kumar Kalever <prasanna.kalever> Date: Wed Apr 27 19:12:19 2016 +0530 glusterd: add defence mechanism to avoid brick port clashes Intro: Currently glusterd maintain the portmap registry which contains ports that are free to use between 49152 - 65535, this registry is initialized once, and updated accordingly as an then when glusterd sees they are been used. Glusterd first checks for a port within the portmap registry and gets a FREE port marked in it, then checks if that port is currently free using a connect() function then passes it to brick process which have to bind on it. Problem: We see that there is a time gap between glusterd checking the port with connect() and brick process actually binding on it. In this time gap it could be so possible that any process would have occupied this port because of which brick will fail to bind and exit. Case 1: To avoid the gluster client process occupying the port supplied by glusterd : we have separated the client port map range with brick port map range more @ http://review.gluster.org/#/c/13998/ Case 2: (Handled by this patch) To avoid the other foreign process occupying the port supplied by glusterd : To handle above situation this patch implements a mechanism to return EADDRINUSE error code to glusterd, upon which a new port is allocated and try to restart the brick process with the newly allocated port. Note: Incase of glusterd restarts i.e. runner_run_nowait() there is no way to handle Case 2, becuase runner_run_nowait() will not wait to get the return/exit code of the executed command (brick process). Hence as of now in such case, we cannot know with what error the brick has failed to connect. This patch also fix the runner_end() to perform some cleanup w.r.t return values. Change-Id: Iec52e7f5d87ce938d173f8ef16aa77fd573f2c5e BUG: 1322805 Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever> Reviewed-on: http://review.gluster.org/14043 Tested-by: Prasanna Kumar Kalever <pkalever> Reviewed-by: Atin Mukherjee <amukherj> Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> The patches which address this bug: http://review.gluster.org/14043 http://review.gluster.org/13998 http://review.gluster.org/14080 were already modified in the code repository. This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user |