Bug 1040844 - glusterd process crashed
Summary: glusterd process crashed
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Atin Mukherjee
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-12 08:40 UTC by Raghavendra Bhat
Modified: 2014-11-11 08:25 UTC (History)
4 users (show)

Fixed In Version: glusterfs-3.6.0beta1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-11-11 08:25:40 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Raghavendra Bhat 2013-12-12 08:40:37 UTC
Description of problem:

glusterd process crashed with the following backtrace

[2013-12-12 08:35:19.490199] E [glusterd-store.c:1273:glusterd_store_global_info] 0-management: Unable to get store handle
pending frames:
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2013-12-12 08:35:19configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3git
glusterd(glusterfsd_print_trace+0x22)[0x4087d7]
/lib64/libc.so.6[0x3fc54359b0]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_store_global_info+0x399)[0x7f40c1e6f553]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_uuid_generate_save+0xa6)[0x7f40c1e1e86e]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_uuid_init+0xae)[0x7f40c1e1e782]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(+0xb4682)[0x7f40c1eb0682]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(gd_sync_task_begin+0xdc)[0x7f40c1eb3884]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_op_begin_synctask+0x8b)[0x7f40c1eb3b83]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(__glusterd_handle_create_volume+0x970)[0x7f40c1ea0ce9]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x4e)[0x7f40c1e3025b]
/usr/local/lib/glusterfs/3git/xlator/mgmt/glusterd.so(glusterd_handle_create_volume+0x22)[0x7f40c1ea0df9]
/usr/local/lib/libglusterfs.so.0(synctask_wrap+0x37)[0x7f40c54f95ca]
/lib64/libc.so.6[0x3fc5446370]
---------

Core was generated by `glusterd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f40c1e6f553 in glusterd_store_global_info (this=0x2518900) at ../../../../../xlators/mgmt/glusterd/src/glusterd-store.c:1314
1314	        if (ret && (handle->fd > 0))
Missing separate debuginfos, use: debuginfo-install device-mapper-event-libs-1.02.74-6.fc17.x86_64 device-mapper-libs-1.02.74-6.fc17.x86_64 glibc-2.15-59.fc17.x86_64 keyutils-libs-1.5.5-2.fc17.x86_64 krb5-libs-1.10.2-12.fc17.x86_64 libcom_err-1.42.3-3.fc17.x86_64 libgcc-4.7.2-2.fc17.x86_64 libselinux-2.1.10-3.fc17.x86_64 libsepol-2.1.7-3.fc17.x86_64 libudev-182-3.fc17.x86_64 libxml2-2.7.8-9.fc17.x86_64 lvm2-libs-2.02.95-6.fc17.x86_64 openssl-1.0.0k-1.fc17.x86_64 python-libs-2.7.3-7.2.fc17.x86_64 zlib-1.2.5-7.fc17.x86_64
(gdb) bt
#0  0x00007f40c1e6f553 in glusterd_store_global_info (this=0x2518900) at ../../../../../xlators/mgmt/glusterd/src/glusterd-store.c:1314
#1  0x00007f40c1e1e86e in glusterd_uuid_generate_save () at ../../../../../xlators/mgmt/glusterd/src/glusterd.c:170
#2  0x00007f40c1e1e782 in glusterd_uuid_init () at ../../../../../xlators/mgmt/glusterd/src/glusterd.c:142
#3  0x00007f40c1eb0682 in __glusterd_uuid () at ../../../../../xlators/mgmt/glusterd/src/glusterd.h:471
#4  0x00007f40c1eb3884 in gd_sync_task_begin (op_ctx=0x7f40c3ad91f0, req=0x2525bec) at ../../../../../xlators/mgmt/glusterd/src/glusterd-syncop.c:1216
'#5  0x00007f40c1eb3b83 in glusterd_op_begin_synctask (req=0x2525bec, op=GD_OP_CREATE_VOLUME, dict=0x7f40c3ad91f0)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-syncop.c:1291
#6  0x00007f40c1ea0ce9 in __glusterd_handle_create_volume (req=0x2525bec) at ../../../../../xlators/mgmt/glusterd/src/glusterd-volume-ops.c:174
#7  0x00007f40c1e3025b in glusterd_big_locked_handler (req=0x2525bec, actor_fn=0x7f40c1ea0379 <__glusterd_handle_create_volume>)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-handler.c:77
#8  0x00007f40c1ea0df9 in glusterd_handle_create_volume (req=0x2525bec) at ../../../../../xlators/mgmt/glusterd/src/glusterd-volume-ops.c:198
#9  0x00007f40c54f95ca in synctask_wrap (old_task=0x2535240) at ../../../libglusterfs/src/syncop.c:293
#10 0x0000003fc5446370 in ?? () from /lib64/libc.so.6
#11 0x0000000000000000 in ?? ()
(gdb)  f 0
#0  0x00007f40c1e6f553 in glusterd_store_global_info (this=0x2518900) at ../../../../../xlators/mgmt/glusterd/src/glusterd-store.c:1314
1314	        if (ret && (handle->fd > 0))
(gdb) p handle
$1 = (gf_store_handle_t *) 0x0
(gdb) p ret
$2 = -1
(gdb) p conf->handle
$3 = (gf_store_handle_t *) 0x0
(gdb) 


If getting handle fails, then handle pointer should not be accessed before returning without checking it for NULL.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Atin Mukherjee 2013-12-30 13:21:42 UTC
Although this test case is not reproducible (Can reproduce if the handle is explicitly set to NULL from gdb) , but from implementation perspective this is a valid bug if a negative test case is considered where store handle is NULL. 
As per the basic coding guidelines we should have checked the pointer (handle) before dereferencing it which caused the segmentation fault.

Comment 2 Anand Avati 2013-12-30 13:42:31 UTC
REVIEW: http://review.gluster.org/6619 (Bug-1040844-glusterd process crashes due to NULL handle) posted (#1) for review on master by Atin Mukherjee (amukherj)

Comment 3 Anand Avati 2014-01-03 08:06:30 UTC
REVIEW: http://review.gluster.org/6636 (GF_ASSERT macro does gurantee the NULL pointer dereferencing check, so     replaced it with GF_VALIDATE_OR_GOTO whereever there is a danger of     program getting crashed due to NULL pointer dereference.) posted (#1) for review on master by Atin Mukherjee (amukherj)

Comment 4 Anand Avati 2014-01-03 08:09:11 UTC
REVIEW: http://review.gluster.org/6636 (GF_ASSERT macro does not gurantee the NULL pointer dereferencing check, so replaced it with GF_VALIDATE_OR_GOTO whereever there is a danger of program getting crashed due to NULL pointer dereference.) posted (#2) for review on master by Atin Mukherjee (amukherj)

Comment 5 Anand Avati 2014-01-03 08:14:09 UTC
REVIEW: http://review.gluster.org/6619 (Description of problem:) posted (#2) for review on master by Atin Mukherjee (amukherj)

Comment 6 Anand Avati 2014-01-03 10:18:48 UTC
REVIEW: http://review.gluster.org/6619 (Glusterd : glusterd process generates core due to NULL store handle) posted (#3) for review on master by Atin Mukherjee (amukherj)

Comment 7 Anand Avati 2014-01-03 10:45:26 UTC
REVIEW: http://review.gluster.org/6619 (Glusterd : glusterd process generates core due to NULL store handle) posted (#4) for review on master by Atin Mukherjee (amukherj)

Comment 8 Anand Avati 2014-01-04 06:24:35 UTC
COMMIT: http://review.gluster.org/6619 committed in master by Vijay Bellur (vbellur) 
------
commit 79cbf27b9b98d1feebcc2f1db5fc1c976d2c24cf
Author: Atin Mukherjee <amukherj>
Date:   Fri Jan 3 13:36:40 2014 +0530

    Glusterd : glusterd process generates core due to NULL store handle
    
    Problem : glusterd crashed as backtrace revealed that store handle was
    set to NULL.
    
    Solution : In glusterd_store_global_info() function out block the
    handle is dereferenced with out any NULL check which caused this
    segmentation fault. A NULL check is introduced to avoid this. While
    testing this fix, another issue was noticed where GF_ASSERT macro again
    does not gurantee the NULL dereference check and hence this macro call
    has been replaced by GF_VALIDATE_OR_GOTO macro call in places where
    there is a danger of macro getting crashed due to NULL dereference
    check.
    
    Change-Id: Ic301aa45ce4bbdc2da751d2386439df7bb24c016
    BUG: 1040844
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: http://review.gluster.org/6619
    Reviewed-by: Niels de Vos <ndevos>
    Reviewed-by: Krishnan Parthasarathi <kparthas>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 9 Niels de Vos 2014-09-22 12:33:43 UTC
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 10 Niels de Vos 2014-11-11 08:25:40 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users


Note You need to log in before you can comment on or make changes to this bug.