Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1364331

Summary:	Glusterd crashes upon receiving SIGUSR1
Product:	[Community] GlusterFS	Reporter:	Atin Mukherjee <amukherj>
Component:	glusterd	Assignee:	bugs <bugs>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.7.14	CC:	amukherj, bugs, p.malishev
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1364329	Environment:
Last Closed:	2016-08-05 06:05:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1359131, 1359190, 1364329
Bug Blocks:

Description Atin Mukherjee 2016-08-05 05:55:03 UTC

+++ This bug was initially created as a clone of Bug #1364329 +++

+++ This bug was initially created as a clone of Bug #1359190 +++

+++ This bug was initially created as a clone of Bug #1359131 +++

Description of problem:
I try to generate glusterdump file with SIGUSR1 to find out maximum op-version for the glusterd on this host.
Glusterd crashes upon receiving SIGUSR1


Version-Release number of selected component (if applicable):
# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"

# dpkg -l | grep gluster
ii  glusterfs-client                      3.8.1-ubuntu1~trusty1               amd64        clustered file-system (client package)
ii  glusterfs-common                      3.8.1-ubuntu1~trusty1               amd64        GlusterFS common libraries and translator modules
ii  glusterfs-server                      3.8.1-ubuntu1~trusty1               amd64        clustered file-system (server package)


How reproducible:
Glusterd crashes each time I send SIGUSR1


Steps to Reproduce:
1. rm -f /var/run/gluster/glusterdump.*
2. service glusterfs-server start
3. kill -SIGUSR1 $(pidof glusterd)
4. ps -ef | grep glusterd ; ls /var/run/gluster | grep glusterdump

Actual results:
No glusterd in the ps output.
No glusterdump file created.

glusterd crashes with the following lines in the log:
[2016-07-22 11:22:50.923915] I [MSGID: 106493] [glusterd-rpc-ops.c:691:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c6654b8f-3ca8-424f-8d0f-7c0b7d8108cc
pending frames:
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2016-07-22 11:23:09
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.1
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f21f4b87b02]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f21f4b9104d]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f21f3f81cb0]
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1d03)[0x7f21f3f96943]
/lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0x95)[0x7f21f4054165]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section_fd+0x69)[0x7f21f4bb18a9]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section+0xa5)[0x7f21f4bb1ab5]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4ce3e)[0x7f21f4bb1e3e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_xlator_info+0x58)[0x7f21f4bb2658]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_info+0x500)[0x7f21f4bb2d00]
/usr/sbin/glusterd(glusterfs_sigwaiter+0xce)[0x55ffe99d2bae]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f21f4318184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f21f404537d]
---------



Expected results:
Glusterd creates glusterdump file in /var/run/gluster


Additional info:

--- Additional comment from Atin Mukherjee on 2016-07-22 08:47:08 EDT ---

Does this happen everytime? I've seen it earlier as well but couldn't reproduce.

--- Additional comment from Atin Mukherjee on 2016-07-22 08:48:06 EDT ---

coredump file?

--- Additional comment from Vijay Bellur on 2016-07-22 09:10:07 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#1) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-22 09:16:55 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#2) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-24 23:56:40 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#3) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-26 14:10:38 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#4) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-07-29 00:57:17 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#5) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-08-01 07:23:11 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump crash) posted (#6) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Vijay Bellur on 2016-08-04 11:44:13 EDT ---

COMMIT: http://review.gluster.org/14987 committed in master by Jeff Darcy (jdarcy) 
------
commit 049c91565dddb622b8902ccfeb36c0d414c609e1
Author: Atin Mukherjee <amukherj>
Date:   Tue May 31 16:44:48 2016 +0530

    libglusterfs: fix glusterd statedump crash
    
    commit 3c04a91 removed setting typeStr to NULL if num_allocs is set to 0, this
    has caused this regression. Code has been put back like earlier and to avoid
    statedump printing all the NULL values check is modified to see skip the records
    if num_allocs is 0 instead of total_allocs
    
    Change-Id: Ib8bcc2fba908e88cf52b641c3f6bcba74f5e667c
    BUG: 1359190
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: http://review.gluster.org/14987
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: N Balachandran <nbalacha>
    Reviewed-by: Prashanth Pai <ppai>
    Reviewed-by: Jeff Darcy <jdarcy>

--- Additional comment from Vijay Bellur on 2016-08-05 01:52:07 EDT ---

REVIEW: http://review.gluster.org/15091 (libglusterfs: fix glusterd statedump crash) posted (#1) for review on release-3.8 by Atin Mukherjee (amukherj)

--- Additional comment from Atin Mukherjee on 2016-08-05 01:53:59 EDT ---

Comment 1 Atin Mukherjee 2016-08-05 06:05:54 UTC

Since commit 3c04a91 is not in 3.7 release branch, this bug is not there. Closing it.