Bug 809771 - [glusterfs-3.3.0qa33] - gluster CLI crashed trying access a free'd memory
Summary: [glusterfs-3.3.0qa33] - gluster CLI crashed trying access a free'd memory
Keywords:
Status: CLOSED DUPLICATE of bug 812735
Alias: None
Product: GlusterFS
Classification: Community
Component: cli
Version: pre-release
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
Assignee: Raghavendra Bhat
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-04-04 11:04 UTC by M S Vishwanath Bhat
Modified: 2016-06-01 01:55 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-04-17 10:34:26 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description M S Vishwanath Bhat 2012-04-04 11:04:35 UTC
Description of problem:
Gluster CLI crashed trying to access the memory which is already free'd up. The code was compiled with -lgcov --coverage flags and crash happened while running the rebalance status command. 

Version-Release number of selected component (if applicable):
glusterfs-3.3.30qa33

How reproducible:
Once in as many tries.

Steps to Reproduce:
1. Create and start a gluster volume.
2. Run top_profile sanity in it which in turn will start rebalance and run rebalance status
  
Actual results:
Gluster CLI crashed with the following back trace.

Program terminated with signal 11, Segmentation fault.
#0  0x000000379ea0c110 in pthread_spin_lock () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.el6_2.9.x86_64 libgcc-4.4.6-3.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64 readline-6.0-3.el6.x86_64
(gdb) bt
#0  0x000000379ea0c110 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00007f6b7328e760 in mem_put (ptr=0x7f6b64000980) at mem-pool.c:530
#2  0x000000000040d70a in STACK_DESTROY (stack=0x6d552c) at ../../libglusterfs/src/stack.h:188
#3  0x0000000000411151 in cli_cmd_volume_defrag_cbk (state=0x7ffffb572420, word=0x6ff7f0, words=0x7ffffb572600, wordcount=4) at cli-cmd-volume.c:818
#4  0x000000000040b852 in cli_cmd_process (state=0x7ffffb572420, argc=4, argv=0x7ffffb572600) at cli-cmd.c:150
#5  0x000000000040af05 in cli_batch (d=0x7ffffb572420) at input.c:44
#6  0x000000379ea077f1 in start_thread () from /lib64/libpthread.so.0
#7  0x000000379e6e5ccd in clone () from /lib64/libc.so.6
(gdb) f 0
#0  0x000000379ea0c110 in pthread_spin_lock () from /lib64/libpthread.so.0
(gdb) f 1
#1  0x00007f6b7328e760 in mem_put (ptr=0x7f6b64000980) at mem-pool.c:530
530             LOCK (&pool->lock);
(gdb) p pool
$1 = (struct mem_pool *) 0xcafebabe
(gdb) 


Expected results:
rebalance status should not crash.

Additional info:

Entries from the cli.log 


[2012-04-04 06:32:35.512415] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:35.567260] I [input.c:46:cli_batch] 0-: Exiting with: 0
[2012-04-04 06:32:35.574202] W [rpc-transport.c:18[2012-04-04 06:32:43.219256] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:43.273457] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:43.273676] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7f49fced7939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7f49fced683a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7f49fced5f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:43.273525 (xid=0x1x)
[2012-04-04 06:32:43.286435] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:43.340563] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:43.340777] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7fe60bc05939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7fe60bc0483a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7fe60bc03f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:43.340631 (xid=0x1x)
[2012-04-04 06:32:43.360692] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:43.414558] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:43.414775] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7fe9df2ab939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7fe9df2aa83a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7fe9df2a9f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:43.414625 (xid=0x1x)
[2012-04-04 06:32:43.421734] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:43.479309] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:43.479549] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7f37eed16939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7f37eed1583a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7f37eed14f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:43.479378 (xid=0x1x)
[2012-04-04 06:32:43.487432] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:43.543146] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:43.543401] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7f2473484939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7f247348383a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7f2473482f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:43.543245 (xid=0x1x)
[2012-04-04 06:32:43.550550] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:43.604622] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:43.604845] E [rpc-clnt.c:382:saved_fram[2012-04-04 06:32:44.127021] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:44.181093] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:44.181339] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7eff7664e939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7eff7664d83a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7eff7664cf09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:44.181169 (xid=0x1x)
[2012-04-04 06:32:44.194521] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:44.309875] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:44.310285] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7f1e45ac5939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7f1e45ac483a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7f1e45ac3f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:44.309982 (xid=0x1x)
[2012-04-04 06:32:44.337885] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:44.437475] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:44.437834] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7f62055e2939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7f62055e183a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7f62055e0f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:44.437582 (xid=0x1x)
[2012-04-04 06:32:44.460935] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:44.563323] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:44.563684] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7f3ba30ef939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7f3ba30ee83a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7f3ba30edf09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:44.563432 (xid=0x1x)
[2012-04-04 06:32:44.580651] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:44.679018] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:44.679423] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7f17a8eb0939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7f17a8eaf83a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7f17a8eaef09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:44.679126 (xid=0x1x)
[2012-04-04 06:32:44.708768] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:44.763242] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:44.763462] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7ff3fad57939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7ff3fad5683a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7ff3fad55f09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:44.763310 (xid=0x1x)
[2012-04-04 06:32:44.775680] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2012-04-04 06:32:44.829422] W [socket.c:1521:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (Transport endpoint is not connected), peer (::1:24007)
[2012-04-04 06:32:44.829636] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x186) [0x7fd3991d0939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x22e) [0x7fd3991cf83a] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7fd3991cef09]))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(11)) called at 2012-04-04 06:32:44.829488 (xid=0x1x)
[2012-04-04 06:32:44.842414] W [rpc-transport.c:183:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"


I couldn't upload the cli.log since it's ~20MB. I have archived all the log files.

Comment 1 Amar Tumballi 2012-04-17 10:34:26 UTC
duping to another bug filed later because the patch was sent against that bug id.

*** This bug has been marked as a duplicate of bug 812735 ***


Note You need to log in before you can comment on or make changes to this bug.