Bug 765098 (GLUSTER-3366)

Summary: [glusterfs-3.1.6qa7]: glusterd crashed on gluster volume set <volname> help
Product: [Community] GlusterFS Reporter: Raghavendra Bhat <rabhat>
Component: glusterdAssignee: Vijay Bellur <vbellur>
Severity: high Docs Contact:
Priority: medium    
Version: pre-releaseCC: gluster-bugs, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Raghavendra Bhat 2011-08-08 15:28:28 EDT
glusterd crashed on gluster volume set <volname> help. The crashed glusterd was running stat-prefetch on and off in a loop. volume set help command was issued from other glusterd.

This is the backtrace.

Core was generated by `glusterd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002aaaaab041ce in glusterd_get_volopt_content (xml_out=_gf_false)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1603
1603            list_for_each_entry_safe (vol_opt, tmp, &vol_opt_handle.list, list) {
(gdb) bt
#0  0x00002aaaaab041ce in glusterd_get_volopt_content (xml_out=_gf_false)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1603
#1  0x00002aaaaaae6c5a in glusterd_volset_help (dict=0x18afb030) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:3917
#2  0x00002aaaaaae6e3c in glusterd_op_set_volume (dict=0x18afb030) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:3954
#3  0x00002aaaaaaeb76c in glusterd_op_commit_perform (op=GD_OP_SET_VOLUME, dict=0x18afb030, op_errstr=0x7fff13c20180, 
    rsp_dict=0x18b090a0) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5642
#4  0x00002aaaaaaeb2b2 in glusterd_op_ac_commit_op (event=0x18b0db00, ctx=0x18b0a9e0)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5495
#5  0x00002aaaaaaebbf4 in glusterd_op_sm () at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5916
#6  0x00002aaaaaad0b2f in glusterd_handle_commit_op (req=0x2aaaaad25044)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-handler.c:577
#7  0x00002b9f96885c3f in rpcsvc_handle_rpc_call (svc=0x18af40a0, trans=0x18af7d00, msg=0x18afdea0)
    at ../../../../rpc/rpc-lib/src/rpcsvc.c:998
#8  0x00002b9f96885fcf in rpcsvc_notify (trans=0x18af7d00, mydata=0x18af40a0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x18afdea0)
    at ../../../../rpc/rpc-lib/src/rpcsvc.c:1094
#9  0x00002b9f9688bfc9 in rpc_transport_notify (this=0x18af7d00, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x18afdea0)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:1043
#10 0x00002aaaaadd44bf in socket_event_poll_in (this=0x18af7d00) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1623
#11 0x00002aaaaadd4830 in socket_event_handler (fd=5, idx=1, data=0x18af7d00, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1737
#12 0x00002b9f96647770 in event_dispatch_epoll_handler (event_pool=0x18aecba0, events=0x18af70c0, i=0)
    at ../../../libglusterfs/src/event.c:812
#13 0x00002b9f9664795f in event_dispatch_epoll (event_pool=0x18aecba0) at ../../../libglusterfs/src/event.c:876
#14 0x00002b9f96647cbb in event_dispatch (event_pool=0x18aecba0) at ../../../libglusterfs/src/event.c:984
#15 0x0000000000405fc7 in main (argc=1, argv=0x7fff13c20b58) at ../../../glusterfsd/src/glusterfsd.c:1455
(gdb) f 0
#0  0x00002aaaaab041ce in glusterd_get_volopt_content (xml_out=_gf_false)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1603
1603            list_for_each_entry_safe (vol_opt, tmp, &vol_opt_handle.list, list) {
(gdb) p vol_opt
$1 = (volume_opt_list_t *) 0x18000030
(gdb) p *vol_opt
Cannot access memory at address 0x18000030
(gdb) p vol_opt_handle.list
$2 = {next = 0x18000030, prev = 0x18b08fd0}
(gdb) l glusterd_get_volopt_content
1496    }
1499    int
1500    glusterd_get_volopt_content (gf_boolean_t xml_out)
1501    {
1503            char                    *xlator_type = NULL;
1504            void                    *dl_handle = NULL;
1505            volume_opt_list_t        vol_opt_handle;
1506            volume_opt_list_t       *vol_opt = NULL;
1507            volume_opt_list_t       *tmp = NULL;
1508            char                    *key = NULL;
1509            struct volopt_map_entry *vme = NULL;
1510            int                      ret = -1;
1511            char                    *def_val = NULL;
1512            char                    *descr = NULL;
1513            char                     output_string[8192] = {0, };
1514            char                    *output = NULL;
1515            char                     tmp_str[1024] = {0, };
Comment 1 Anand Avati 2011-08-08 23:49:42 EDT
CHANGE: http://review.gluster.com/197 (Change-Id: I63883aa17a76a39f6d5f862ad7270f56a46e2598) merged in release-3.1 by Vijay Bellur (vijay@gluster.com)
Comment 2 Raghavendra Bhat 2011-08-09 00:43:55 EDT
It seems just gluster volume set help command is suffecient to crash the glusterd. Here the glusterd where the help command is issued does not crash. But other glusterd's present in the trusted storage pool will crash.
Comment 3 Raghavendra Bhat 2011-08-22 00:44:14 EDT
On master gluster volume set help is crashing glusterd. Reopening it.
Comment 4 Raghavendra Bhat 2011-08-22 02:00:39 EDT
The above comment regarding the crash in the master is not this bug, but a different one.
Comment 5 Anand Avati 2011-09-11 21:47:57 EDT
CHANGE: http://review.gluster.com/380 (Change-Id: I503364c855d52605e301f4d3c205af6c9fc0e1df) merged in master by Vijay Bellur (vijay@gluster.com)