Bug 1004417

Summary: Smbd crashed while doing "gluster volume replace-brick" operation on the gluster volume
Product: Red Hat Gluster Storage Reporter: Lalatendu Mohanty <lmohanty>
Component: sambaAssignee: Raghavendra Talur <rtalur>
Status: CLOSED DUPLICATE QA Contact: Lalatendu Mohanty <lmohanty>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.1CC: sbhaloth, vagarwal
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-05 16:58:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Lalatendu Mohanty 2013-09-04 15:11:31 UTC 
Description of problem:

Smbd crashed while doing "gluster volume replace-brick" operation on the gluster volume

From /var/log/glusterfs/.cmd_log_history
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>.

2013-09-04 14:47:19.758747]  : volume replace-brick testvol3 10.16.159.15:/rhs/brick2/testvol3-b1 10.16.159.15:/rhs/brick3/testvol3-b1 status : SUCCESS
[2013-09-04 14:48:44.823808]  : volume replace-brick testvol3 10.16.159.15:/rhs/brick2/testvol3-b1 10.16.159.15:/rhs/brick3/testvol3-b1 status : SUCCESS
[2013-09-04 14:48:47.431961]  : volume replace-brick testvol3 10.16.159.15:/rhs/brick2/testvol3-b1 10.16.159.15:/rhs/brick3/testvol3-b1 status : SUCCESS
[2013-09-04 14:49:08.494874]  : volume replace-brick testvol3 10.16.159.15:/rhs/brick2/testvol3-b1 10.16.159.15:/rhs/brick3/testvol3-b1 commit : SUCCESS


Back Trace of crash from /var/log/messages
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Sep  4 10:49:08 dhcp159-136 smbd[19077]:   ===============================================================
Sep  4 10:49:08 dhcp159-136 smbd[19077]: [2013/09/04 10:49:08.390237,  0] lib/fault.c:48(fault_report)
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   INTERNAL ERROR: Signal 11 in pid 19077 (3.6.9-160.3.el6rhs)
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Sep  4 10:49:08 dhcp159-136 smbd[19077]: [2013/09/04 10:49:08.390336,  0] lib/fault.c:50(fault_report)
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Sep  4 10:49:08 dhcp159-136 smbd[19077]: [2013/09/04 10:49:08.390447,  0] lib/fault.c:51(fault_report)
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   ===============================================================
Sep  4 10:49:08 dhcp159-136 smbd[19077]: [2013/09/04 10:49:08.390514,  0] lib/util.c:1117(smb_panic)
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   PANIC (pid 19077): internal error
Sep  4 10:49:08 dhcp159-136 smbd[19077]: [2013/09/04 10:49:08.392130,  0] lib/util.c:1221(log_stack_trace)
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   BACKTRACE: 20 stack frames:
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #0 smbd(log_stack_trace+0x1a) [0x7fbc60d394fa]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #1 smbd(smb_panic+0x2b) [0x7fbc60d395cb]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #2 smbd(+0x41a054) [0x7fbc60d2a054]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #3 /lib64/libc.so.6(+0x3ff1832960) [0x7fbc5cbe2960]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #4 /lib64/libc.so.6(+0x3ff1881381) [0x7fbc5cc31381]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #5 /lib64/libc.so.6(xdr_string+0x37) [0x7fbc5ccc7a97]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #6 /usr/lib64/libgfxdr.so.0(xdr_gf_getspec_req+0x41) [0x7fbc5dda86b1]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #7 /lib64/libc.so.6(xdr_sizeof+0xa1) [0x7fbc5ccc9711]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #8 /usr/lib64/libgfapi.so.0(mgmt_submit_request+0x140) [0x7fbc5e450a80]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #9 /usr/lib64/libgfapi.so.0(glfs_volfile_fetch+0x113) [0x7fbc5e450c43]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #10 /usr/lib64/libgfapi.so.0(mgmt_cbk_spec+0x10) [0x7fbc5e450e50]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #11 /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_cbk+0x132) [0x7fbc5dfc2c12]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #12 /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1b8) [0x7fbc5dfc3fa8]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #13 /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28) [0x7fbc5dfbf838]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #14 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0x8be6) [0x7fbc4f22bbe6]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #15 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0xa4fd) [0x7fbc4f22d4fd]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #16 /usr/lib64/libglusterfs.so.0(+0x3ff245e8c7) [0x7fbc5e2288c7]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #17 /usr/lib64/libgfapi.so.0(+0x5834) [0x7fbc5e44f834]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #18 /lib64/libpthread.so.0(+0x3ff2007851) [0x7fbc5b360851]
Sep  4 10:49:08 dhcp159-136 smbd[19077]:    #19 /lib64/libc.so.6(clone+0x6d) [0x7fbc5cc9894d]
Sep  4 10:49:08 dhcp159-136 smbd[19077]: [2013/09/04 10:49:08.393044,  0] lib/fault.c:372(dump_core)
Sep  4 10:49:08 dhcp159-136 smbd[19077]:   dumping core in /var/log/core

Version-Release number of selected component (if applicable):


How reproducible:

Intermittent

Steps to Reproduce:
1. Create a volume, start it, mount it on Windows client
2. Create files on the mount point. Set some ACLs for the files
3. Add bricks and do a rebalance. Wait for the rebalance to finish
4. Perform replace brick operation

Actual results:

Smbd crashed

Expected results:

Additional info:

 
Volume Name: testvol3
Type: Distributed-Replicate
Volume ID: fd25d234-c003-4f46-81fc-bfbf82638da5
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: 10.16.159.136:/rhs/brick1/testvol3-b1
Brick2: 10.16.159.16:/rhs/brick1/testvol3-b1
Brick3: 10.16.159.238:/rhs/brick1/testvol3-b1
Brick4: 10.16.159.15:/rhs/brick1/testvol3-b1
Brick5: 10.16.159.238:/rhs/brick2/testvol3-b1
Brick6: 10.16.159.15:/rhs/brick3/testvol3-b1
Options Reconfigured:
performance.stat-prefetch: off
server.allow-insecure: on
Comment 2 Lalatendu Mohanty 2013-09-04 15:19:45 UTC 
This issue looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1003665. But I am not sure about it.
Comment 3 Vivek Agarwal 2013-09-05 16:58:06 UTC 

*** This bug has been marked as a duplicate of bug 1003665 ***