Bug 816163

Summary: [glusterfs-3.3.0qa38] - nfs server crashed because free is called on 'gf_calloc'ed memory
Product: [Community] GlusterFS Reporter: M S Vishwanath Bhat <vbhat>
Component: unclassifiedAssignee: Kaushal <kaushal>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: pre-releaseCC: gluster-bugs, kaushal, mzywusko
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.4.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-24 17:53:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 817967    

Description M S Vishwanath Bhat 2012-04-25 11:47:05 UTC
Description of problem:
gluster nfs server crashed because of free'ing of a gf_calloc'ed memory.

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa38

How reproducible:
1/1

Steps to Reproduce:
1. Create and start a 2 node distribute volume.
2. Run remove brick start on one of the bricks.
3. Run remove brick status in a while loop continuously. 
  
Actual results:
gluster nfs server crashed with following bt.

(gdb) bt
#0  0x00000032a6232885 in raise () from /lib64/libc.so.6
#1  0x00000032a6234065 in abort () from /lib64/libc.so.6
#2  0x00000032a626f977 in __libc_message () from /lib64/libc.so.6
#3  0x00000032a6275296 in malloc_printerr () from /lib64/libc.so.6
#4  0x000000000041187c in glusterfs_handle_nfs_profile (req=0x1e2ed6c) at glusterfsd-mgmt.c:1228
#5  0x0000000000411bcb in glusterfs_handle_rpc_msg (req=0x1e2ed6c) at glusterfsd-mgmt.c:1264
#6  0x00007f16fac0f7a0 in rpcsvc_handle_rpc_call (svc=0x1e2eb70, trans=0x1ebcba0, msg=0x1e43b80) at rpcsvc.c:520
#7  0x00007f16fac0fd66 in rpcsvc_notify (trans=0x1ebcba0, mydata=0x1e2eb70, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1e43b80) at rpcsvc.c:616
#8  0x00007f16fac197c2 in rpc_transport_notify (this=0x1ebcba0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1e43b80) at rpc-transport.c:498
#9  0x00007f16f76a805b in socket_event_poll_in (this=0x1ebcba0) at socket.c:1686
#10 0x00007f16f76a8ab1 in socket_event_handler (fd=14, idx=7, data=0x1ebcba0, poll_in=1, poll_out=0, poll_err=0) at socket.c:1801
#11 0x00007f16faea5f75 in event_dispatch_epoll_handler (event_pool=0x1e29c50, events=0x1e42d40, i=0) at event.c:794
#12 0x00007f16faea632d in event_dispatch_epoll (event_pool=0x1e29c50) at event.c:856
#13 0x00007f16faea698c in event_dispatch (event_pool=0x1e29c50) at event.c:956
#14 0x000000000040b19f in main (argc=11, argv=0x7fffac896a28) at glusterfsd.c:1651
(gdb) fr 4
#4  0x000000000041187c in glusterfs_handle_nfs_profile (req=0x1e2ed6c) at glusterfsd-mgmt.c:1228
1228                    free (rsp.output.output_val);
(gdb) x /12 ((char*)rsp.output.output_val - 12)
0x1ebea44:      -889275714      0       0       268435456
0x1ebea54:      251658240       201326592       926035248       2019650861
0x1ebea64:      1702125932      7955310 875573554       808464430
(gdb) x /12x ((char*)rsp.output.output_val - 12)
0x1ebea44:      0xcafebabe      0x00000000      0x00000000      0x10000000
0x1ebea54:      0x0f000000      0x0c000000      0x37322d30      0x78616d2d
0x1ebea64:      0x6574616c      0x0079636e      0x34303132      0x3030302e


Expected results:
glusterfs server should not crash.

Additional info:

I have archived all the logs and core file.

Comment 1 Anand Avati 2012-04-27 03:24:30 UTC
CHANGE: http://review.gluster.com/3231 (glusterfsd: Change a free() to GF_FREE()) merged in master by Vijay Bellur (vijay)

Comment 2 M S Vishwanath Bhat 2012-05-11 08:14:30 UTC
with glusterfs-3.3.0qa40, I'm not seeing this crash anymore. Moving it to verified.