Bug 764507 (GLUSTER-2775)

Summary: crash in reconfigure
Product: [Community] GlusterFS Reporter: Amar Tumballi <amarts>
Component: quotaAssignee: Junaid <junaid>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: mainlineCC: gluster-bugs, vagarwal, vraman
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Raghavendra G 2011-04-15 14:25:38 UTC
Do you've a log file of client?

regards,
Raghavendra.

Comment 1 Amar Tumballi 2011-04-15 16:28:27 UTC
#0  0x00007f92d6868a75 in *__GI_raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f92d686c5c0 in *__GI_abort () at abort.c:92
#2  0x00007f92d6861941 in *__GI___assert_fail (assertion=0x7f92d199c7a7 "0", 
    file=<value optimized out>, line=2737, function=0x7f92d199c98f "reconfigure") at assert.c:81
#3  0x00007f92d199c218 in reconfigure (this=0x7f92c4016b60, options=0x7f92c4027930)
    at ../../../../../xlators/features/quota/src/quota.c:2737
#4  0x00007f92d742fbce in xlator_reconfigure_rec (old_xl=0x7f92c4016b60, new_xl=0x7f92c4080290)
    at ../../../libglusterfs/src/xlator.c:1469
#5  0x00007f92d742faf9 in xlator_reconfigure_rec (old_xl=0x7f92c4050ce0, new_xl=0x7f92c4080c10)
    at ../../../libglusterfs/src/xlator.c:1455
#6  0x00007f92d742faf9 in xlator_reconfigure_rec (old_xl=0x7f92c4081a20, new_xl=0x7f92c4069260)
    at ../../../libglusterfs/src/xlator.c:1455
#7  0x00007f92d742faf9 in xlator_reconfigure_rec (old_xl=0x7f92c4073ef0, new_xl=0x7f92c4069be0)
    at ../../../libglusterfs/src/xlator.c:1455
#8  0x00007f92d742faf9 in xlator_reconfigure_rec (old_xl=0x7f92c403df00, new_xl=0x7f92c406a560)
    at ../../../libglusterfs/src/xlator.c:1455
#9  0x00007f92d742faf9 in xlator_reconfigure_rec (old_xl=0x7f92c4014e20, new_xl=0x7f92c406aee0)
    at ../../../libglusterfs/src/xlator.c:1455
#10 0x00007f92d742faf9 in xlator_reconfigure_rec (old_xl=0x7f92c40157a0, new_xl=0x7f92c406b860)
    at ../../../libglusterfs/src/xlator.c:1455
#11 0x00007f92d7430296 in xlator_tree_reconfigure (old_xl=0x7f92c40157a0, new_xl=0x7f92c406b860)
    at ../../../libglusterfs/src/xlator.c:1610
#12 0x00007f92d746a886 in glusterfs_graph_reconfigure (oldgraph=0x7f92c403fe40, 
    newgraph=0x7f92c4027780) at ../../../libglusterfs/src/graph.c:549
#13 0x0000000000408c03 in glusterfs_volfile_reconfigure (newvolfile_fp=0x7f92c4073a10)
    at ../../../glusterfsd/src/glusterfsd-mgmt.c:587

#14 0x0000000000408ecb in mgmt_getspec_cbk (req=0x7f92d4e8f044, iov=0x7f92d4e8f084, count=1, 
    myframe=0x7f92d5b9e068) at ../../../glusterfsd/src/glusterfsd-mgmt.c:658
#15 0x00007f92d720713d in rpc_clnt_handle_reply (clnt=0x13cd720, pollin=0x7f92c4069070)
    at ../../../../rpc/rpc-lib/src/rpc-clnt.c:736
#16 0x00007f92d720749c in rpc_clnt_notify (trans=0x13cd940, mydata=0x13cd750, 
    event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f92c4069070)
    at ../../../../rpc/rpc-lib/src/rpc-clnt.c:849
#17 0x00007f92d7203ab4 in rpc_transport_notify (this=0x13cd940, event=RPC_TRANSPORT_MSG_RECEIVED, 
    data=0x7f92c4069070) at ../../../../rpc/rpc-lib/src/rpc-transport.c:902
#18 0x00007f92d4c83dce in socket_event_poll_in (this=0x13cd940)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1647
#19 0x00007f92d4c84352 in socket_event_handler (fd=6, idx=2, data=0x13cd940, poll_in=1, 
    poll_out=0, poll_err=0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1762
#20 0x00007f92d745ce9a in event_dispatch_epoll_handler (event_pool=0x13cab90, events=0x13cf6b0, 
    i=0) at ../../../libglusterfs/src/event.c:794
#21 0x00007f92d745d0bd in event_dispatch_epoll (event_pool=0x13cab90)
    at ../../../libglusterfs/src/event.c:856
#22 0x00007f92d745d448 in event_dispatch (event_pool=0x13cab90)
    at ../../../libglusterfs/src/event.c:956
#23 0x00000000004075c1 in main (argc=5, argv=0x7fffa9bf9128)
    at ../../../glusterfsd/src/glusterfsd.c:1476

(gdb) f 3
#3  0x00007f92d199c218 in reconfigure (this=0x7f92c4016b60, options=0x7f92c4027930)
    at ../../../../../xlators/features/quota/src/quota.c:2737
2737	                GF_ASSERT (0);
(gdb) l
2732	        }
2733	
2734	        ret = quota_parse_options (priv, this, options);
2735	
2736	        if (ret == -1)
2737	                GF_ASSERT (0);
2738	
2739	        return ret;
2740	}
2741	
(gdb) 

This happened because of "DEBUG" build. Overall, not sure if we need ASSERT here.

Comment 2 Amar Tumballi 2011-04-16 01:10:57 UTC
[2011-04-15 21:39:14.344212] I [quota.c:2638:quota_parse_options] 1-vol-quota: /dir:10485760
[2011-04-15 21:39:14.344239] I [quota.c:2638:quota_parse_options] 1-vol-quota: /:3145728
[2011-04-15 21:39:14.344255] I [quota.c:2662:quota_parse_options] 1-vol-quota: quota timeout value = 0
[2011-04-15 21:39:14.344268] I [quota.c:2672:quota_parse_options] 1-vol-quota: /dir:10485760
[2011-04-15 21:39:14.344292] I [quota.c:2672:quota_parse_options] 1-vol-quota: /:3145728
[2011-04-15 21:39:14.344352] W [dict.c:1153:data_to_str] (-->/usr/local/lib/libglusterfs.so.0(+0x1faf9) [0x7f92d742faf9] (-->/usr/local/lib/libglusterfs.so.0(+0x1fbce) [0x7f92d742fbce] (-->/usr/local/lib/glusterfs/3.1.0git/xlator/performance/io-cache.so(reconfigure+0x58b) [0x7f92d3a0814e]))) 1-dict: data is NULL
[2011-04-15 21:39:14.344400] W [dict.c:1153:data_to_str] (-->/usr/local/lib/libglusterfs.so.0(+0x1faf9) [0x7f92d742faf9] (-->/usr/local/lib/libglusterfs.so.0(+0x1fbce) [0x7f92d742fbce] (-->/usr/local/lib/glusterfs/3.1.0git/xlator/performance/io-cache.so(reconfigure+0x695) [0x7f92d3a08258]))) 1-dict: data is NULL
[2011-04-15 21:39:14.344446] W [dict.c:1153:data_to_str] (-->/usr/local/lib/libglusterfs.so.0(+0x1faf9) [0x7f92d742faf9] (-->/usr/local/lib/libglusterfs.so.0(+0x1fbce) [0x7f92d742fbce] (-->/usr/local/lib/glusterfs/3.1.0git/xlator/performance/io-cache.so(reconfigure+0x839) [0x7f92d3a083fc]))) 1-dict: data is NULL
[2011-04-15 21:39:14.344490] W [dict.c:1153:data_to_str] (-->/usr/local/lib/libglusterfs.so.0(+0x1faf9) [0x7f92d742faf9] (-->/usr/local/lib/libglusterfs.so.0(+0x1fbce) [0x7f92d742fbce] (-->/usr/local/lib/glusterfs/3.1.0git/xlator/performance/io-cache.so(reconfigure+0x865) [0x7f92d3a08428]))) 1-dict: data is NULL
[2011-04-15 21:39:30.981245] I [glusterfsd-mgmt.c:62:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2011-04-15 21:39:32.19520] W [common-utils.c:1271:gf_string2bytesize] (-->/usr/local/lib/libglusterfs.so.0(+0x1fbce) [0x7f92d742fbce] (-->/usr/local/lib/glusterfs/3.1.0git/xlator/features/quota.so(reconfigure+0xaa) [0x7f92d199c1f0] (-->/usr/local/lib/glusterfs/3.1.0git/xlator/features/quota.so(quota_parse_options+0xb4) [0x7f92d199bbd0]))) 1-: arguement invalid
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 2011-04-15 21:39:32
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.0git

Comment 3 Junaid 2011-06-10 05:36:21 UTC
Saurabh, can you confirm whether this bug still exists. It works on my machine.

Comment 4 Amar Tumballi 2011-06-10 05:41:43 UTC
(In reply to comment #3)
> Saurabh, can you confirm whether this bug still exists. It works on my machine.

-----
       ret = quota_parse_options (priv, this, options);

       if (ret == -1)
                GF_ASSERT (0);
-----

This code still exists in code, hence the abort issue should also exist. You can find the issue by building glusterfs with 'CFLAGS="-DDEBUG' (make sure you have done a make clean before). And setting some failure option to quota.

Comment 5 Anand Avati 2011-06-20 04:21:03 UTC
PATCH: http://patches.gluster.com/patch/7553 in master (features/quota: Print a log when reconfigure fails.)