Bug 1461655 - glusterd crashes when statedump is taken
glusterd crashes when statedump is taken
Product: GlusterFS
Classification: Community
Component: glusterd (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Atin Mukherjee
: Triaged
Depends On:
Blocks: 1461649
  Show dependency treegraph
Reported: 2017-06-15 02:03 EDT by Atin Mukherjee
Modified: 2017-10-26 10:35 EDT (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.12.0
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1461649
Last Closed: 2017-09-05 13:34:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Atin Mukherjee 2017-06-15 02:03:27 EDT
+++ This bug was initially created as a clone of Bug #1461649 +++

Description of problem:

* start glusterd
* take statedump
* glusterd crashes

Thread 1 (Thread 0x7f535fd71700 (LWP 1557)):
#0  glusterd_dump_priv (this=<optimized out>) at glusterd-statedump.c:243
#1  0x00007f5367aaf655 in gf_proc_dump_xlator_info (top=<optimized out>) at statedump.c:502
#2  0x00007f5367aafb92 in gf_proc_dump_info (signum=signum@entry=10, ctx=0xd1e010) at statedump.c:837
#3  0x00000000004089b9 in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2083
#4  0x0000003866e07d14 in start_thread (arg=0x7f535fd71700) at pthread_create.c:309
#5  0x0000003866af168d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) l
248	                                                "pmap[%d].type", port);
249	                        gf_proc_dump_write (key, "%d", pmap->ports[port].type);
250	                        gf_proc_dump_build_key (key, "glusterd",
251	                                                "pmap[%d].brickname", port);
252	                        gf_proc_dump_write (key, "%s",
253	                                            pmap->ports[port].brickname);
255	                }
256	                /* Dump client details */
257	                glusterd_dump_client_details (priv);
(gdb) p pmap
$2 = (struct pmap_registry *) 0x0
(gdb) l glusterd-statedump.c:243
238	                /* Dump peer details */
239	                GLUSTERD_DUMP_PEERS (&priv->peers, uuid_list, _gf_false);
241	                /* Dump pmap data structure from base port to last alloc */
242	                pmap = priv->pmap;
243	                for (port = pmap->base_port; port <= pmap->last_alloc;
244	                     port++) {
245	                        gf_proc_dump_build_key (key, "glusterd", "pmap_port");
246	                        gf_proc_dump_write (key, "%d", port);
247	                        gf_proc_dump_build_key (key, "glusterd",
(gdb) p pmap
$3 = (struct pmap_registry *) 0x0

I think since there are no bricks, pmap is NULL resulting in crash. This _MAY_ not happen when there are volumes and bricks associated with glusterd

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-15 01:27:10 EDT ---

This bug is automatically being proposed for the current release of Red Hat Gluster Storage 3 under active development, by setting the release flag 'rhgs‑3.3.0' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Atin Mukherjee on 2017-06-15 01:46:53 EDT ---

You are right. This is because the pmap object is created when glusterd allocates the the port for the first time and that will happen only when the first volume is started.
Comment 1 Worker Ant 2017-06-15 02:04:12 EDT
REVIEW: https://review.gluster.org/17549 (glusterd: fix crash on statedump when no volumes are started) posted (#1) for review on master by Atin Mukherjee (amukherj@redhat.com)
Comment 2 Worker Ant 2017-06-15 06:49:51 EDT
COMMIT: https://review.gluster.org/17549 committed in master by Atin Mukherjee (amukherj@redhat.com) 
commit cd67c34e64e2c85a7d0abbce43a1db269a12be19
Author: Atin Mukherjee <amukherj@redhat.com>
Date:   Thu Jun 15 11:31:39 2017 +0530

    glusterd: fix crash on statedump when no volumes are started
    pmap object is created when glusterd allocates a port for the very first
    time, however before that if someone tries to take statedump glusterd
    will crash.
    Solution : Add a NULL check before accessing pmap reference.
    Change-Id: I206b02e07a4717e68af2c6bf05fac55119353de8
    BUG: 1461655
    Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
    Reviewed-on: https://review.gluster.org/17549
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Samikshan Bairagya <samikshan@gmail.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Comment 3 Shyamsundar 2017-09-05 13:34:18 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.0, please open a new bug report.

glusterfs-3.12.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-September/000082.html
[2] https://www.gluster.org/pipermail/gluster-users/

Note You need to log in before you can comment on or make changes to this bug.