Bug 1364329

Summary: Glusterd crashes upon receiving SIGUSR1
Product: [Community] GlusterFS Reporter: Atin Mukherjee <amukherj>
Component: glusterdAssignee: Atin Mukherjee <amukherj>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.8.2CC: amukherj, bugs, p.malishev
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.8.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1359190
: 1364331 (view as bug list) Environment:
Last Closed: 2016-08-12 09:48:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1359131, 1359190    
Bug Blocks: 1364331    

Description Atin Mukherjee 2016-08-05 05:50:39 UTC 
+++ This bug was initially created as a clone of Bug #1359190 +++

+++ This bug was initially created as a clone of Bug #1359131 +++

Description of problem:
I try to generate glusterdump file with SIGUSR1 to find out maximum op-version for the glusterd on this host.
Glusterd crashes upon receiving SIGUSR1


Version-Release number of selected component (if applicable):
# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"

# dpkg -l | grep gluster
ii  glusterfs-client                      3.8.1-ubuntu1~trusty1               amd64        clustered file-system (client package)
ii  glusterfs-common                      3.8.1-ubuntu1~trusty1               amd64        GlusterFS common libraries and translator modules
ii  glusterfs-server                      3.8.1-ubuntu1~trusty1               amd64        clustered file-system (server package)


How reproducible:
Glusterd crashes each time I send SIGUSR1


Steps to Reproduce:
1. rm -f /var/run/gluster/glusterdump.*
2. service glusterfs-server start
3. kill -SIGUSR1 $(pidof glusterd)
4. ps -ef | grep glusterd ; ls /var/run/gluster | grep glusterdump

Actual results:
No glusterd in the ps output.
No glusterdump file created.

glusterd crashes with the following lines in the log:
[2016-07-22 11:22:50.923915] I [MSGID: 106493] [glusterd-rpc-ops.c:691:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c6654b8f-3ca8-424f-8d0f-7c0b7d8108cc
pending frames:
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2016-07-22 11:23:09
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.1
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f21f4b87b02]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f21f4b9104d]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f21f3f81cb0]
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1d03)[0x7f21f3f96943]
/lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0x95)[0x7f21f4054165]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section_fd+0x69)[0x7f21f4bb18a9]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section+0xa5)[0x7f21f4bb1ab5]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4ce3e)[0x7f21f4bb1e3e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_xlator_info+0x58)[0x7f21f4bb2658]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_info+0x500)[0x7f21f4bb2d00]
/usr/sbin/glusterd(glusterfs_sigwaiter+0xce)[0x55ffe99d2bae]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f21f4318184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f21f404537d]
---------



Expected results:
Glusterd creates glusterdump file in /var/run/gluster


Additional info:

--- Additional comment from Atin Mukherjee on 2016-07-22 08:47:08 EDT ---

Does this happen everytime? I've seen it earlier as well but couldn't reproduce.

--- Additional comment from Atin Mukherjee on 2016-07-22 08:48:06 EDT ---

coredump file?

--- Additional comment from Vijay Bellur on 2016-07-22 09:10:07 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#1) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-22 09:16:55 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#2) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-24 23:56:40 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#3) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-26 14:10:38 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#4) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-29 00:57:17 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#5) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-08-01 07:23:11 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#6) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-08-04 11:44:13 EDT ---

COMMIT: http://review.gluster.org/14987 committed in master by Jeff Darcy (jdarcy) 
------
commit 049c91565dddb622b8902ccfeb36c0d414c609e1
Author: Atin Mukherjee <amukherj>
Date:   Tue May 31 16:44:48 2016 +0530

    libglusterfs: fix glusterd statedump crash
    
    commit 3c04a91 removed setting typeStr to NULL if num_allocs is set to 0, this
    has caused this regression. Code has been put back like earlier and to avoid
    statedump printing all the NULL values check is modified to see skip the records
    if num_allocs is 0 instead of total_allocs
    
    Change-Id: Ib8bcc2fba908e88cf52b641c3f6bcba74f5e667c
    BUG: 1359190
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: http://review.gluster.org/14987
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: N Balachandran <nbalacha>
    Reviewed-by: Prashanth Pai <ppai>
    Reviewed-by: Jeff Darcy <jdarcy>
Comment 1 Vijay Bellur 2016-08-05 05:52:07 UTC 
REVIEW: http://review.gluster.org/15091 (libglusterfs: fix glusterd statedump crash) posted (#1) for review on release-3.8 by Atin Mukherjee (amukherj)
Comment 2 Atin Mukherjee 2016-08-05 05:53:59 UTC 
*** Bug 1359131 has been marked as a duplicate of this bug. ***
Comment 3 Vijay Bellur 2016-08-07 11:53:51 UTC 
COMMIT: http://review.gluster.org/15091 committed in release-3.8 by Niels de Vos (ndevos) 
------
commit ffc67d3fc64a016e9900bdb8b5fed5866850a1d8
Author: Atin Mukherjee <amukherj>
Date:   Tue May 31 16:44:48 2016 +0530

    libglusterfs: fix glusterd statedump crash
    
    Backport of http://review.gluster.org/14987
    
    commit 3c04a91 removed setting typeStr to NULL if num_allocs is set to 0, this
    has caused this regression. Code has been put back like earlier and to avoid
    statedump printing all the NULL values check is modified to see skip the records
    if num_allocs is 0 instead of total_allocs
    
    >Reviewed-on: http://review.gluster.org/14987
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >Smoke: Gluster Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.org>
    >Reviewed-by: N Balachandran <nbalacha>
    >Reviewed-by: Prashanth Pai <ppai>
    >Reviewed-by: Jeff Darcy <jdarcy>
    
    Change-Id: Ib8bcc2fba908e88cf52b641c3f6bcba74f5e667c
    BUG: 1364329
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: http://review.gluster.org/15091
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>
    Reviewed-by: Niels de Vos <ndevos>
Comment 4 Niels de Vos 2016-08-12 09:48:35 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.2, please open a new bug report.

glusterfs-3.8.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/announce/2016-August/000058.html
[2] https://www.gluster.org/pipermail/gluster-users/