glusterd crashed on gluster volume set <volname> help. The crashed glusterd was running stat-prefetch on and off in a loop. volume set help command was issued from other glusterd. This is the backtrace. Core was generated by `glusterd'. Program terminated with signal 11, Segmentation fault. #0 0x00002aaaaab041ce in glusterd_get_volopt_content (xml_out=_gf_false) at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1603 1603 list_for_each_entry_safe (vol_opt, tmp, &vol_opt_handle.list, list) { (gdb) bt #0 0x00002aaaaab041ce in glusterd_get_volopt_content (xml_out=_gf_false) at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1603 #1 0x00002aaaaaae6c5a in glusterd_volset_help (dict=0x18afb030) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:3917 #2 0x00002aaaaaae6e3c in glusterd_op_set_volume (dict=0x18afb030) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:3954 #3 0x00002aaaaaaeb76c in glusterd_op_commit_perform (op=GD_OP_SET_VOLUME, dict=0x18afb030, op_errstr=0x7fff13c20180, rsp_dict=0x18b090a0) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5642 #4 0x00002aaaaaaeb2b2 in glusterd_op_ac_commit_op (event=0x18b0db00, ctx=0x18b0a9e0) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5495 #5 0x00002aaaaaaebbf4 in glusterd_op_sm () at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5916 #6 0x00002aaaaaad0b2f in glusterd_handle_commit_op (req=0x2aaaaad25044) at ../../../../../xlators/mgmt/glusterd/src/glusterd-handler.c:577 #7 0x00002b9f96885c3f in rpcsvc_handle_rpc_call (svc=0x18af40a0, trans=0x18af7d00, msg=0x18afdea0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:998 #8 0x00002b9f96885fcf in rpcsvc_notify (trans=0x18af7d00, mydata=0x18af40a0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x18afdea0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1094 #9 0x00002b9f9688bfc9 in rpc_transport_notify (this=0x18af7d00, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x18afdea0) at ../../../../rpc/rpc-lib/src/rpc-transport.c:1043 #10 0x00002aaaaadd44bf in socket_event_poll_in (this=0x18af7d00) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1623 #11 0x00002aaaaadd4830 in socket_event_handler (fd=5, idx=1, data=0x18af7d00, poll_in=1, poll_out=0, poll_err=0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1737 #12 0x00002b9f96647770 in event_dispatch_epoll_handler (event_pool=0x18aecba0, events=0x18af70c0, i=0) at ../../../libglusterfs/src/event.c:812 #13 0x00002b9f9664795f in event_dispatch_epoll (event_pool=0x18aecba0) at ../../../libglusterfs/src/event.c:876 #14 0x00002b9f96647cbb in event_dispatch (event_pool=0x18aecba0) at ../../../libglusterfs/src/event.c:984 #15 0x0000000000405fc7 in main (argc=1, argv=0x7fff13c20b58) at ../../../glusterfsd/src/glusterfsd.c:1455 (gdb) f 0 #0 0x00002aaaaab041ce in glusterd_get_volopt_content (xml_out=_gf_false) at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1603 1603 list_for_each_entry_safe (vol_opt, tmp, &vol_opt_handle.list, list) { (gdb) p vol_opt $1 = (volume_opt_list_t *) 0x18000030 (gdb) p *vol_opt Cannot access memory at address 0x18000030 (gdb) p vol_opt_handle.list $2 = {next = 0x18000030, prev = 0x18b08fd0} (gdb) l glusterd_get_volopt_content 1496 } 1497 1498 1499 int 1500 glusterd_get_volopt_content (gf_boolean_t xml_out) 1501 { 1502 1503 char *xlator_type = NULL; 1504 void *dl_handle = NULL; 1505 volume_opt_list_t vol_opt_handle; (gdb) 1506 volume_opt_list_t *vol_opt = NULL; 1507 volume_opt_list_t *tmp = NULL; 1508 char *key = NULL; 1509 struct volopt_map_entry *vme = NULL; 1510 int ret = -1; 1511 char *def_val = NULL; 1512 char *descr = NULL; 1513 char output_string[8192] = {0, }; 1514 char *output = NULL; 1515 char tmp_str[1024] = {0, }; (gdb)
CHANGE: http://review.gluster.com/197 (Change-Id: I63883aa17a76a39f6d5f862ad7270f56a46e2598) merged in release-3.1 by Vijay Bellur (vijay)
It seems just gluster volume set help command is suffecient to crash the glusterd. Here the glusterd where the help command is issued does not crash. But other glusterd's present in the trusted storage pool will crash.
On master gluster volume set help is crashing glusterd. Reopening it.
The above comment regarding the crash in the master is not this bug, but a different one.
CHANGE: http://review.gluster.com/380 (Change-Id: I503364c855d52605e301f4d3c205af6c9fc0e1df) merged in master by Vijay Bellur (vijay)