1003584 – SMB:smbd crashes while doing volume operations

Bug 1003584 - SMB:smbd crashes while doing volume operations

Summary: SMB:smbd crashes while doing volume operations

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat
Component:	samba
Sub Component:
Version:	2.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Raghavendra Talur
QA Contact:	surabhi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1004091 1004519
TreeView+	depends on / blocked

Reported:	2013-09-02 12:41 UTC by surabhi
Modified:	2013-09-23 22:32 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1004091 1004519 (view as bug list)
Environment:
Last Closed:	2013-09-23 22:32:17 UTC
Target Upstream Version:
Dependent Products:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description surabhi 2013-09-02 12:41:12 UTC

Description of problem:
While doing operations below , the smbd crashes and generates a core file.

Version-Release number of selected component (if applicable):
glusterfs-server-3.4.0.30rhs-2.el6rhs.x86_64
samba-glusterfs-3.6.9-160.3.el6rhs.x86_64

How reproducible:
Not always

Steps to Reproduce:
1.Mount a volume via cifs on linux client
2.Start I/o's
3.Do a add-brick operation on the same volume
4 start rebalance
5.Do another add brick operation 

Actual results:
Smbd generated cores.

Sep  2 07:41:55 RHSVM1 smbd[3113]: [2013/09/02 07:41:55.243185,  0] lib/util.c:1117(smb_panic)
Sep  2 07:41:55 RHSVM1 smbd[3113]:   PANIC (pid 3113): internal error
Sep  2 07:41:55 RHSVM1 smbd[3113]: [2013/09/02 07:41:55.264781,  0] lib/util.c:1221(log_stack_trace)
Sep  2 07:41:55 RHSVM1 smbd[3113]:   BACKTRACE: 17 stack frames:
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #0 smbd(log_stack_trace+0x1a) [0x7f2135eb54fa]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #1 smbd(smb_panic+0x2b) [0x7f2135eb55cb]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #2 smbd(+0x41a054) [0x7f2135ea6054]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #3 /lib64/libc.so.6(+0x3178032960) [0x7f2131d5e960]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #4 /usr/lib64/libglusterfs.so.0(+0x317905f522) [0x7f21333a5522]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #5 /usr/lib64/libglusterfs.so.0(yyparse+0x47e) [0x7f21333a5b0e]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #6 /usr/lib64/libglusterfs.so.0(glusterfs_graph_construct+0x38a) [0x7f21333a66ca]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #7 /usr/lib64/libgfapi.so.0(mgmt_getspec_cbk+0x4e0) [0x7f21335cd440]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #8 /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f213313ef35]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #9 /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x147) [0x7f213313ff37]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #10 /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28) [0x7f213313b838]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #11 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0x8be6) [0x7f21248eebe6]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #12 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0xa4fd) [0x7f21248f04fd]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #13 /usr/lib64/libglusterfs.so.0(+0x317905e8c7) [0x7f21333a48c7]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #14 /usr/lib64/libgfapi.so.0(+0x5834) [0x7f21335cb834]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #15 /lib64/libpthread.so.0(+0x3178807851) [0x7f21304dc851]
Sep  2 07:41:55 RHSVM1 smbd[3113]:    #16 /lib64/libc.so.6(clone+0x6d) [0x7f2131e1494d]
Sep  2 07:41:55 RHSVM1 smbd[3113]: [2013/09/02 07:41:55.266403,  0] lib/fault.c:372(dump_core)


Expected results:
Smbd should not crash.

Additional info:
Since debug-info was not available will try to reproduce the issue.

Comment 2 Lalatendu Mohanty 2013-09-02 14:53:50 UTC

I too have faced issue while doing remove-brick operations. But this is an intermittent issue.

ep  2 10:33:02 dhcp159-136 smbd[348]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.696364,  0] lib/fault.c:51(fault_report)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   ===============================================================
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.696476,  0] lib/util.c:1117(smb_panic)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   PANIC (pid 348): internal error
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.739896,  0] lib/util.c:1221(log_stack_trace)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   BACKTRACE: 18 stack frames:
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #0 smbd(log_stack_trace+0x1a) [0x7fd4493d64fa]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #1 smbd(smb_panic+0x2b) [0x7fd4493d65cb]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #2 smbd(+0x41a054) [0x7fd4493c7054]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #3 /lib64/libc.so.6(+0x3ff1832960) [0x7fd44527f960]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #4 /lib64/libpthread.so.0(pthread_mutex_lock+0) [0x7fd4439ff220]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #5 /usr/lib64/libglusterfs.so.0(iobuf_get2+0x42) [0x7fd4468a5b32]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #6 /usr/lib64/libgfapi.so.0(mgmt_submit_request+0x14f) [0x7fd446aeda8f]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #7 /usr/lib64/libgfapi.so.0(glfs_volfile_fetch+0x113) [0x7fd446aedc43]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #8 /usr/lib64/libgfapi.so.0(mgmt_cbk_spec+0x10) [0x7fd446aede50]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #9 /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_cbk+0x132) [0x7fd44665fc12]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #10 /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1b8) [0x7fd446660fa8]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #11 /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28) [0x7fd44665c838]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #12 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0x8be6) [0x7fd43801ebe6]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #13 /usr/lib64/glusterfs/3.4.0.30rhs/rpc-transport/socket.so(+0xa4fd) [0x7fd4380204fd]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #14 /usr/lib64/libglusterfs.so.0(+0x3ff245e8c7) [0x7fd4468c58c7]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #15 /usr/lib64/libgfapi.so.0(+0x5834) [0x7fd446aec834]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #16 /lib64/libpthread.so.0(+0x3ff2007851) [0x7fd4439fd851]
Sep  2 10:33:02 dhcp159-136 smbd[348]:    #17 /lib64/libc.so.6(clone+0x6d) [0x7fd44533594d]
Sep  2 10:33:02 dhcp159-136 smbd[348]: [2013/09/02 10:33:02.740650,  0] lib/fault.c:372(dump_core)
Sep  2 10:33:02 dhcp159-136 smbd[348]:   dumping core in /var/log/core
Sep  2 10:33:02 dhcp159-136 smbd[348]:

Comment 3 Lalatendu Mohanty 2013-09-02 15:15:55 UTC

After looking more into the back-trace and comparing my command history, it seems this samba crash is different then what Surabhi has found, hence logging a new bug.

Comment 4 Raghavendra Talur 2013-09-05 11:12:47 UTC

Posted patch for review at:
https://code.engineering.redhat.com/gerrit/#/c/12517/
https://code.engineering.redhat.com/gerrit/#/c/12519/

Comment 5 surabhi 2013-09-06 11:25:37 UTC

Verified the issue with latest version:

glusterfs-server-3.4.0.31rhs-1.el6rhs.x86_64


Following steps doesn't generate core.

1Mount a volume via cifs on linux client
2.Start I/o's
3.Do a add-brick operation on the same volume
4 start rebalance
5.Do another add brick operation

Comment 6 Scott Haines 2013-09-23 22:32:17 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.