This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 815038 - [glusterfs-3.2.6qa1]: glusterd crashed when parallel operations were executed
[glusterfs-3.2.6qa1]: glusterd crashed when parallel operations were executed
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
1.0
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: krishnan parthasarathi
:
Depends On: 783913
Blocks: 811632
  Show dependency treegraph
 
Reported: 2012-04-22 03:20 EDT by Scott Haines
Modified: 2015-11-03 18:04 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 783913
Environment:
Last Closed: 2012-05-01 07:18:05 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Scott Haines 2012-04-22 03:20:26 EDT
+++ This bug was initially created as a clone of Bug #783913 +++

Description of problem:
On a glusterfs peer, volume set commands were being executed in a loop. On the same machine, executed a command to remove the quota limit set on a directory, and then again tried to set a limit on the same directory. glusterd crashed with the below backtrace.

Core was generated by `glusterd'.
Program terminated with signal 6, Aborted.
#0  0x00000034c2232905 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6_1.3.x86_64 libgcc-4.4.5-6.el6.x86_64
(gdb) bt
#0  0x00000034c2232905 in raise () from /lib64/libc.so.6
#1  0x00000034c22340e5 in abort () from /lib64/libc.so.6
#2  0x00000034c222b9be in __assert_fail_base () from /lib64/libc.so.6
#3  0x00000034c222ba80 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fc2e1d04042 in glusterd_op_send_cli_response (op=GD_OP_SET_VOLUME, op_ret=16, op_errno=0, req=0x7fc2e1bfe560, op_ctx=0x0, 
    op_errstr=0x7fc2e1d1ed17 "operation failed") at ../../../../../xlators/mgmt/glusterd/src/glusterd-rpc-ops.c:61
#5  0x00007fc2e1cd513c in glusterd_handle_set_volume (req=0x7fc2e1bfe560) at ../../../../../xlators/mgmt/glusterd/src/glusterd-handler.c:1975
#6  0x00007fc2e3982890 in rpcsvc_handle_rpc_call (svc=0x6ada90, trans=0x6b2c40, msg=0x6b5dc0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:480
#7  0x00007fc2e3982c33 in rpcsvc_notify (trans=0x6b2c40, mydata=0x6ada90, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x6b5dc0)
    at ../../../../rpc/rpc-lib/src/rpcsvc.c:576
#8  0x00007fc2e39886ac in rpc_transport_notify (this=0x6b2c40, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x6b5dc0)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:919
#9  0x00007fc2e19f2af9 in socket_event_poll_in (this=0x6b2c40) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1647
#10 0x00007fc2e19f307d in socket_event_handler (fd=15, idx=9, data=0x6b2c40, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1762
#11 0x00007fc2e3be57e8 in event_dispatch_epoll_handler (event_pool=0x6a63f0, events=0x6b0cb0, i=0) at ../../../libglusterfs/src/event.c:794
#12 0x00007fc2e3be5a0b in event_dispatch_epoll (event_pool=0x6a63f0) at ../../../libglusterfs/src/event.c:856
#13 0x00007fc2e3be5d96 in event_dispatch (event_pool=0x6a63f0) at ../../../libglusterfs/src/event.c:956
#14 0x000000000040700c in main (argc=1, argv=0x7fff0dabbf18) at ../../../glusterfsd/src/glusterfsd.c:1509
(gdb) f 4
#4  0x00007fc2e1d04042 in glusterd_op_send_cli_response (op=GD_OP_SET_VOLUME, op_ret=16, op_errno=0, req=0x7fc2e1bfe560, op_ctx=0x0, 
    op_errstr=0x7fc2e1d1ed17 "operation failed") at ../../../../../xlators/mgmt/glusterd/src/glusterd-rpc-ops.c:61
61              GF_ASSERT (op_ctx);
(gdb) p op_ctx
$1 = (void *) 0x0
(gdb) l
56              void            *cli_rsp = NULL;
57              dict_t          *ctx = NULL;
58              char            *free_ptr = NULL;
59              glusterd_conf_t *conf = NULL;
60
61              GF_ASSERT (op_ctx);
62              GF_ASSERT (THIS);
63
64              conf = THIS->private;
65
(gdb) up
#5  0x00007fc2e1cd513c in glusterd_handle_set_volume (req=0x7fc2e1bfe560) at ../../../../../xlators/mgmt/glusterd/src/glusterd-handler.c:1975
1975                    ret = glusterd_op_send_cli_response (cli_op, ret, 0, req,
(gdb) l
1970            glusterd_op_sm ();
1971
1972            if (ret) {
1973                    if (dict)
1974                            dict_unref (dict);
1975                    ret = glusterd_op_send_cli_response (cli_op, ret, 0, req,
1976                                                         NULL, "operation failed");
1977                    if (!lock_fail)
1978                            (void) glusterd_opinfo_unlock ();
1979            }
(gdb) l glusterd_handle_set_volume
1885            return ret;
1886    }
1887
1888    int
1889    glusterd_handle_set_volume (rpcsvc_request_t *req)
1890    {
1891            int32_t                         ret = -1;
1892            gf1_cli_set_vol_req             cli_req = {0,};
1893            dict_t                          *dict = NULL;
1894            int                             lock_fail = 0;
(gdb) 
1895            glusterd_op_t                   cli_op = GD_OP_SET_VOLUME;
1896            char                            *key = NULL;
1897            char                            *value = NULL;
1898            char                            *volname = NULL;
1899
1900            GF_ASSERT (req);
1901
1902            ret = glusterd_op_set_cli_op (cli_op);
1903            if (ret) {
1904                    gf_log ("", GF_LOG_ERROR, "Unable to set cli op: %d",
(gdb) l
1905                            ret);
1906                    lock_fail = 1;
1907                    goto out;
1908            }
1909
1910            ret = -1;
1911            if (!gf_xdr_to_cli_set_vol_req (req->msg[0], &cli_req)) {
1912                    //failed to decode msg;
1913                    req->rpc_err = GARBAGE_ARGS;
1914                    goto out;
(gdb)  p ret
$1 = 16
(gdb) p cli_op
$2 = GD_OP_SET_VOLUME
(gdb) p lock_fail
$3 = 1
(gdb) 





Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

[2012-01-23 03:31:18.791635] I [glusterd-utils.c:2297:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
[2012-01-23 03:31:18.791816] I [glusterd-utils.c:2302:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
[2012-01-23 03:31:18.792040] I [glusterd-utils.c:2307:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
[2012-01-23 03:31:18.810502] I [glusterd-op-sm.c:6854:glusterd_op_ac_send_commit_op] 0-glusterd: Sent op req to 2 peers
[2012-01-23 03:31:18.810850] E [glusterd-handler.c:1905:glusterd_handle_set_volume] 0-: Unable to set cli op: 16
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 2012-01-23 03:31:18
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.2.6qa1
/lib64/libc.so.6[0x34c2232980]
/lib64/libc.so.6(gsignal+0x35)[0x34c2232905]
/lib64/libc.so.6(abort+0x175)[0x34c22340e5]
/lib64/libc.so.6[0x34c222b9be]
/lib64/libc.so.6(__assert_perror_fail+0x0)[0x34c222ba80]
/usr/local/lib/glusterfs/3.2.6qa1/xlator/mgmt/glusterd.so(glusterd_op_send_cli_response+0x8b)[0x7fc2e1d04042]
/usr/local/lib/glusterfs/3.2.6qa1/xlator/mgmt/glusterd.so(glusterd_handle_set_volume+0x4a0)[0x7fc2e1cd513c]
/usr/local/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x344)[0x7fc2e3982890]
/usr/local/lib/libgfrpc.so.0(rpcsvc_notify+0x181)[0x7fc2e3982c33]
/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x130)[0x7fc2e39886ac]
/usr/local/lib/glusterfs/3.2.6qa1/rpc-transport/socket.so(socket_event_poll_in+0x54)[0x7fc2e19f2af9]
/usr/local/lib/glusterfs/3.2.6qa1/rpc-transport/socket.so(socket_event_handler+0x21d)[0x7fc2e19f307d]
/usr/local/lib/libglusterfs.so.0(+0x4f7e8)[0x7fc2e3be57e8]
/usr/local/lib/libglusterfs.so.0(+0x4fa0b)[0x7fc2e3be5a0b]
/usr/local/lib/libglusterfs.so.0(event_dispatch+0x88)[0x7fc2e3be5d96]
glusterd(main+0x1b7)[0x40700c]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x34c221ecdd]
glusterd[0x403709]

--- Additional comment from aavati@redhat.com on 2012-01-24 11:37:09 EST ---

CHANGE: http://review.gluster.com/2678 (mgmt/glusterd: do not assert if op_ctx is NULL) merged in release-3.2 by Anand Avati (avati@gluster.com)

--- Additional comment from rabhat@redhat.com on 2012-02-13 00:56:03 EST ---

This is fixed now. Parallel gluster cli operations on the same machine is not crashing it. Tested with glusterfs-3.2.6qa2.
Comment 3 errata-xmlrpc 2012-05-01 07:18:05 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0538.html

Note You need to log in before you can comment on or make changes to this bug.