Bug 1003584

Summary: SMB:smbd crashes while doing volume operations
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: surabhi <sbhaloth>
Component: sambaAssignee: Raghavendra Talur <rtalur>
Status: CLOSED ERRATA QA Contact: surabhi <sbhaloth>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 2.1CC: lmohanty, sdharane, surs, vagarwal
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1004091 1004519 (view as bug list) Environment:
Last Closed: 2013-09-23 22:32:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1004091, 1004519    

Description surabhi 2013-09-02 12:41:12 UTC
Description of problem:
While doing operations below , the smbd crashes and generates a core file.

Version-Release number of selected component (if applicable):
glusterfs-server-3.4.0.30rhs-2.el6rhs.x86_64
samba-glusterfs-3.6.9-160.3.el6rhs.x86_64

How reproducible:
Not always

Steps to Reproduce:
1.Mount a volume via cifs on linux client
2.Start I/o's
3.Do a add-brick operation on the same volume
4 start rebalance
5.Do another add brick operation 

Actual results:
Smbd generated cores.

Sep  2 07:41:55 RHSVM1 smbd[3113]: [2013/09/02 07:41:55.243185,  0] lib/util.c:1117(smb_panic)
Sep  2 07:41:55 RHSVM1 smbd[3113]:   PANIC (pid 3113): internal error
Sep  2 07:41:55 RHSVM1 smbd[3113]: [2013/09/02 07:41:55.264781,  0] lib/util.c:1221(log_stack_trace)
Sep  2 07:41:55 RHSVM1 smbd[3113]:   BACKTRACE: 17 stack frames:
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #0 smbd(log_stack_trace+0x1a) [0x7f2135eb54fa]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #1 smbd(smb_panic+0x2b) [0x7f2135eb55cb]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #2 smbd(+0x41a054) [0x7f2135ea6054]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #3 /lib64/libc.so.6(+0x3178032960) [0x7f2131d5e960]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #4 /usr/lib64/libglusterfs.so.0(+0x317905f522) [0x7f21333a5522]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #5 /usr/lib64/libglusterfs.so.0(yyparse+0x47e) [0x7f21333a5b0e]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #6 /usr/lib64/libglusterfs.so.0(glusterfs_graph_construct+0x38a) [0x7f21333a66ca]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #7 /usr/lib64/libgfapi.so.0(mgmt_getspec_cbk+0x4e0) [0x7f21335cd440]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #8 /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f213313ef35]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #9 /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x147) [0x7f213313ff37]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #10 /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28) [0x7f213313b838]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #11 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0x8be6) [0x7f21248eebe6]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #12 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0xa4fd) [0x7f21248f04fd]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #13 /usr/lib64/libglusterfs.so.0(+0x317905e8c7) [0x7f21333a48c7]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #14 /usr/lib64/libgfapi.so.0(+0x5834) [0x7f21335cb834]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #15 /lib64/libpthread.so.0(+0x3178807851) [0x7f21304dc851]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #16 /lib64/libc.so.6(clone+0x6d) [0x7f2131e1494d]
Sep  2 07:41:55 RHSVM1 smbd[3113]: [2013/09/02 07:41:55.266403,  0] lib/fault.c:372(dump_core)


Expected results:
Smbd should not crash.

Additional info:
Since debug-info was not available will try to reproduce the issue.

Comment 2 Lalatendu Mohanty 2013-09-02 14:53:50 UTC
I too have faced issue while doing remove-brick operations. But this is an intermittent issue.

ep  2 10:33:02 dhcp159-136 smbd[348]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.696364,  0] lib/fault.c:51(fault_report)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   ===============================================================
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.696476,  0] lib/util.c:1117(smb_panic)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   PANIC (pid 348): internal error
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.739896,  0] lib/util.c:1221(log_stack_trace)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   BACKTRACE: 18 stack frames:
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #0 smbd(log_stack_trace+0x1a) [0x7fd4493d64fa]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #1 smbd(smb_panic+0x2b) [0x7fd4493d65cb]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #2 smbd(+0x41a054) [0x7fd4493c7054]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #3 /lib64/libc.so.6(+0x3ff1832960) [0x7fd44527f960]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #4 /lib64/libpthread.so.0(pthread_mutex_lock+0) [0x7fd4439ff220]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #5 /usr/lib64/libglusterfs.so.0(iobuf_get2+0x42) [0x7fd4468a5b32]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #6 /usr/lib64/libgfapi.so.0(mgmt_submit_request+0x14f) [0x7fd446aeda8f]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #7 /usr/lib64/libgfapi.so.0(glfs_volfile_fetch+0x113) [0x7fd446aedc43]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #8 /usr/lib64/libgfapi.so.0(mgmt_cbk_spec+0x10) [0x7fd446aede50]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #9 /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_cbk+0x132) [0x7fd44665fc12]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #10 /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1b8) [0x7fd446660fa8]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #11 /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28) [0x7fd44665c838]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #12 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0x8be6) [0x7fd43801ebe6]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #13 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0xa4fd) [0x7fd4380204fd]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #14 /usr/lib64/libglusterfs.so.0(+0x3ff245e8c7) [0x7fd4468c58c7]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #15 /usr/lib64/libgfapi.so.0(+0x5834) [0x7fd446aec834]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #16 /lib64/libpthread.so.0(+0x3ff2007851) [0x7fd4439fd851]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #17 /lib64/libc.so.6(clone+0x6d) [0x7fd44533594d]
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.740650,  0] lib/fault.c:372(dump_core)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   dumping core in /var/log/core
Sep  2 10:33:02 dhcp159-136 smbd[348]:

Comment 3 Lalatendu Mohanty 2013-09-02 15:15:55 UTC
After looking more into the back-trace and comparing my command history, it seems this samba crash is different then what Surabhi has found, hence logging a new bug.

Comment 5 surabhi 2013-09-06 11:25:37 UTC
Verified the issue with latest version:

glusterfs-server-3.4.0.31rhs-1.el6rhs.x86_64


Following steps doesn't generate core.

1Mount a volume via cifs on linux client
2.Start I/o's
3.Do a add-brick operation on the same volume
4 start rebalance
5.Do another add brick operation

Comment 6 Scott Haines 2013-09-23 22:32:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html