Bug 1637379
| Summary: | Portmap entries showing stale brick entries when bricks are down | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Upasana <ubansal> | |
| Component: | glusterd | Assignee: | Mohit Agrawal <moagrawa> | |
| Status: | CLOSED ERRATA | QA Contact: | Upasana <ubansal> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | rhgs-3.4 | CC: | amukherj, moagrawa, nchilaka, rhs-bugs, sankarshan, srakonde, storage-qa-internal, ubansal, vbellur | |
| Target Milestone: | --- | Keywords: | Reopened, ZStream | |
| Target Release: | RHGS 3.4.z Batch Update 3 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.12.2-33 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1646892 (view as bug list) | Environment: | ||
| Last Closed: | 2019-02-04 07:41:25 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1646892 | |||
|
Description
Upasana
2018-10-09 07:34:11 UTC
"With brickmux enabled if we kill the brick process on a node and take a glusterd statedump the portmap entries show stale brick entries" ---> Can you please explain what do you mean by stale brick entries in the portmap here? A snip from the description: <snip> glusterd.pmap_port=49153 glusterd.pmap[49153].type=0 glusterd.pmap[49153].brickname=(null) glusterd.pmap_port=49154 glusterd.pmap[49154].type=4 glusterd.pmap[49154].brickname=/gluster/brick1/v21 /gluster/brick2/vol_-9-2 [root@dhcp35-36 gluster]# The brick path /gluster/brick2/vol_-9-2 exists in both the port </snip> Are you assuming /gluster/brick2/vol_-9-2 is assigned to port 49153 as well? If so, why? Remember brickname of the same portmap entry is set to NULL. This isn't a bug. If you have any justification of why you think otherwise please add a comment. Related to brick not coming up, I see a separate bug https://bugzilla.redhat.com/show_bug.cgi?id=1637445 is filed which will be updated in sometime. (In reply to Atin Mukherjee from comment #5) > "With brickmux enabled if we kill the brick process on a node and take a > glusterd statedump the portmap entries show stale brick entries" ---> Can > you please explain what do you mean by stale brick entries in the portmap > here? > > A snip from the description: > > <snip> > glusterd.pmap_port=49153 > glusterd.pmap[49153].type=0 > glusterd.pmap[49153].brickname=(null) > glusterd.pmap_port=49154 > glusterd.pmap[49154].type=4 > glusterd.pmap[49154].brickname=/gluster/brick1/v21 /gluster/brick2/vol_-9-2 > [root@dhcp35-36 gluster]# > > The brick path /gluster/brick2/vol_-9-2 exists in both the port > </snip> glusterd.pmap_port=49152 glusterd.pmap[49152].type=4 glusterd.pmap[49152].brickname=/gluster/brick1/v2 /gluster/brick2/vol3 /gluster/brick2/vol4 /gluster/brick2/vol5 /gluster/brick2/vol6 /gluster/brick1/vol_-1-1 /gluster/brick2/vol_-1-2 /gluster/brick3/vol_-1-3 /gluster/brick1/vol_-10-1 /gluster/brick2/vol_-10-2 /gluster/brick3/vol_-10-3 /gluster/brick1/vol_-2-1 /gluster/brick2/vol_-2-2 /gluster/brick3/vol_-2-3 /gluster/brick1/vol_-3-1 /gluster/brick2/vol_-3-2 /gluster/brick3/vol_-3-3 /gluster/brick1/vol_-4-1 /gluster/brick2/vol_-4-2 /gluster/brick3/vol_-4-3 /gluster/brick1/vol_-5-1 /gluster/brick2/vol_-5-2 /gluster/brick3/vol_-5-3 /gluster/brick1/vol_-6-1 /gluster/brick2/vol_-6-2 /gluster/brick3/vol_-6-3 /gluster/brick1/vol_-7-1 /gluster/brick2/vol_-7-2 /gluster/brick3/vol_-7-3 /gluster/brick1/vol_-8-1 /gluster/brick2/vol_-8-2 /gluster/brick3/vol_-8-3 /gluster/brick1/vol_-9-1 /gluster/brick2/vol_-9-2 glusterd.pmap_port=49153 glusterd.pmap[49153].type=0 glusterd.pmap[49153].brickname=(null) glusterd.pmap_port=49154 glusterd.pmap[49154].type=4 glusterd.pmap[49154].brickname=/gluster/brick1/v21 /gluster/brick2/vol_-9-2 [root@dhcp35-36 gluster]# As per this the port 49152 still has all the bricks assigned to it. whereas those bricks have been killed. Can you please confirm if this is still not a bug > Are you assuming /gluster/brick2/vol_-9-2 is assigned to port 49153 as well? > If so, why? Remember brickname of the same portmap entry is set to NULL. > > This isn't a bug. If you have any justification of why you think otherwise > please add a comment. > > Related to brick not coming up, I see a separate bug > https://bugzilla.redhat.com/show_bug.cgi?id=1637445 is filed which will be > updated in sometime. Thanks for the confirmation. I kind of oversighted that information. I'm reopening the BZ. But please note, that kill -9 is not a right way to gracefully bring down a process as this doesn't initiate the PMAP_SIGNOUT event. However, let me check where the problem is. Analysis so far: What's happening here, in __glusterd_brick_rpc_notify when we initiate pmap_registry_remove if a brick is started and the process is killed, we check for the existance of the pidfile and if the service is running. Ideally the service should be killed. However we're still seeing the process and hence pmap_registry_remove () is not invoked and hence we're left with the stale entries. This doesn't happen every time. I'll dig this further. So here's the experiment I did. In gf_is_service_running () if I add couple of log entries to see if the process is running or not, the race goes away. gf_is_service_running () which calls gf_is_pid_running () checks for /proc/<pidfile>/cmdline to see if such entry exists. This gets me to a hypothesis that by the time glusterd starts processing the brick disconnect event in __glusterd_brick_rpc_notify (), the pid entry from the proc is not yet wiped out which means kernel is taking an additional time to take care of post kill formalities of the process and this is where glusterd assumes that process is still running and happily ignore initiating pmap_registry_remove (). We'd need to see how we can get this working. I just came up with a patch which I'd like to try testing, however unfortunately I am unable to reproduce this anymore. No luck with upstream master, rhgs-3.4.0 & rhgs-3.4.1 branches.
I need help in a setup where this can be reproducible, and then I can give a private build with the following change to test this out:
atin@dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ gd
diff --git a/xlators/mgmt/glusterd/src/glusterd-handler.c b/xlators/mgmt/glusterd/src/glusterd-handler.c
index aa8892784..4748ea337 100644
--- a/xlators/mgmt/glusterd/src/glusterd-handler.c
+++ b/xlators/mgmt/glusterd/src/glusterd-handler.c
@@ -6209,7 +6209,9 @@ __glusterd_brick_rpc_notify(struct rpc_clnt *rpc, void *mydata,
* the same if the process is not running
*/
GLUSTERD_GET_BRICK_PIDFILE(pidfile, volinfo, brickinfo, conf);
- if (!gf_is_service_running(pidfile, &pid)) {
+ if (!gf_is_service_running(pidfile, &pid) ||
+ (is_brick_mx_enabled() &&
+ !search_brick_path_from_proc(pid, brickinfo->path))) {
ret = pmap_registry_remove(
THIS, brickinfo->port, brickinfo->path,
GF_PMAP_PORT_BRICKSERVER, NULL, _gf_true);
diff --git a/xlators/mgmt/glusterd/src/glusterd-utils.h b/xlators/mgmt/glusterd/src/glusterd-utils.h
index 4bdc048dd..5c6fa95ae 100644
--- a/xlators/mgmt/glusterd/src/glusterd-utils.h
+++ b/xlators/mgmt/glusterd/src/glusterd-utils.h
@@ -873,4 +873,7 @@ glusterd_get_volinfo_from_brick(char *brick, glusterd_volinfo_t **volinfo);
gf_boolean_t
glusterd_is_profile_on(glusterd_volinfo_t *volinfo);
+char *
+search_brick_path_from_proc(pid_t brick_pid, char *brickpath);
+
#endif
Upstream patch : https://review.gluster.org/#/c/21568/ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263 |