1001614 – File operations caused smbd process to crash

Bug 1001614 - File operations caused smbd process to crash

Summary: File operations caused smbd process to crash

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat
Component:	samba
Sub Component:
Version:	2.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Raghavendra Talur
QA Contact:	Lalatendu Mohanty
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1002511
TreeView+	depends on / blocked

Reported:	2013-08-27 11:52 UTC by Lalatendu Mohanty
Modified:	2013-09-23 22:32 UTC (History)
CC List:	6 users (show)
Fixed In Version:	glusterfs-3.4.0.29rhs-1
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1002511 (view as bug list)
Environment:
Last Closed:	2013-09-23 22:32:16 UTC
Target Upstream Version:
Dependent Products:

Attachments	(Terms of Use)
Perl script used to create files (2.26 KB, application/x-perl) 2013-08-27 11:55 UTC, Lalatendu Mohanty	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Lalatendu Mohanty 2013-08-27 11:52:32 UTC

Description of problem:

I was creating some files on a win7 client using a Perl script on a samba-rhs share and after sometime the script failed and I saw a core file got created. 

However I am not able to reproduce the issue in another attempt. I guess I have hit some corner issue and wanted to log the bug with available information
 
Version-Release number of selected component (if applicable):

root@bvt-rhs1 core]# rpm -qa | grep samba
samba-doc-3.6.9-159.1.el6rhs.x86_64
samba-winbind-3.6.9-159.1.el6rhs.x86_64
samba-glusterfs-3.6.9-159.1.el6rhs.x86_64
samba-swat-3.6.9-159.1.el6rhs.x86_64
samba-winbind-krb5-locator-3.6.9-159.1.el6rhs.x86_64
samba-domainjoin-gui-3.6.9-159.1.el6rhs.x86_64
samba-common-3.6.9-159.1.el6rhs.x86_64
samba-3.6.9-159.1.el6rhs.x86_64
samba-client-3.6.9-159.1.el6rhs.x86_64
samba-winbind-devel-3.6.9-159.1.el6rhs.x86_64
samba4-libs-4.0.0-55.el6.rc4.x86_64
samba-winbind-clients-3.6.9-159.1.el6rhs.x86_64


Gluster version:
glusterfs-3.4.0.22rhs-2.el6rhs.x86_64

How reproducible:

Intermittent 

Steps to Reproduce:
1. Create a volume and start it 
2. Mount it on Win7 client 
3. Create files on the on the samba share in win7client and try to read the files from another windows client.

Actual results:

Below are the messages from /var/log/messages

#################################

[2013/08/27 15:10:38.395766,  0] lib/fault.c:48(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   INTERNAL ERROR: Signal 11 in pid 8761 (3.6.9-159.1.el6rhs)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.395929,  0] lib/fault.c:50(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.396136,  0] lib/fault.c:51(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   ===============================================================
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.396250,  0] lib/util.c:1117(smb_panic)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   PANIC (pid 8761): internal error
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.401721,  0] lib/util.c:1221(log_stack_trace)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   BACKTRACE: 25 stack frames:
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #0 smbd(log_stack_trace+0x1a) [0x7f622b7de58a]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #1 smbd(smb_panic+0x2b) [0x7f622b7de65b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #2 smbd(+0x41a0e4) [0x7f622b7cf0e4]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #3 /lib64/libc.so.6(+0x39dda32920) [0x7f6227688920]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0x59) [0x7f6228efc609]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x80) [0x7f6228efc7c0]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x46) [0x7f6228efcc06]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x58) [0x7f6228efcd78]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x73) [0x7f6228efd3e3]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #9 /usr/lib64/libgfapi.so.0(glfs_close+0x62) [0x7f6228efb4e2]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #10 smbd(fd_close+0x43) [0x7f622b50f383]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #11 smbd(close_file+0x223) [0x7f622b515573]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #12 smbd(reply_close+0x8b) [0x7f622b4e0a8b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #13 smbd(+0x178414) [0x7f622b52d414]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #14 smbd(+0x17882b) [0x7f622b52d82b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #15 smbd(+0x178c45) [0x7f622b52dc45]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #16 smbd(run_events_poll+0x377) [0x7f622b7ed907]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #17 smbd(smbd_process+0x86d) [0x7f622b52b78d]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #18 smbd(+0x69502f) [0x7f622ba4a02f]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #19 smbd(run_events_poll+0x377) [0x7f622b7ed907]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #20 smbd(+0x438dbf) [0x7f622b7eddbf]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #21 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f622802149d]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #22 smbd(main+0xf3b) [0x7f622ba4b32b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #23 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f6227674cdd]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #24 smbd(+0xf4b99) [0x7f622b4a9b99]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.406829,  0] lib/fault.c:372(dump_core)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   dumping core in /var/log/core

########################################
The script failed while writing files
 
C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Creating directory at W:\
Creating files in W:\......
Terminating on signal SIGINT(2)

C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Creating directory at W:\
Creating files in W:\......
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Terminating on signal SIGINT(2)

C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Expected results:

It should not fail

Additional info:

Comment 2 Lalatendu Mohanty 2013-08-27 11:55:38 UTC

Created attachment 790930 [details]
Perl script used to create files

Attached Perl script used to create files

perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Comment 3 Christopher R. Hertel 2013-08-28 02:46:20 UTC

Compare the sequence of calls to the stack trace in Bug #1000545.  This is probably the same bug.  This BZ, however, shows us that the core dump was caused by SIGSEGV.

It appears that glfs_refresh_inode_safe() is passing a bad pointer to something in glibc.

See Bug #1000545.  Please mark this as a duplicate if you agree that these are the same.

Comment 4 Lalatendu Mohanty 2013-08-28 12:46:45 UTC

The Issue again got reproduced today.

Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   ===============================================================
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274029,  0] lib/fault.c:48(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   INTERNAL ERROR: Signal 11 in pid 4363 (3.6.9-160.1.el6rhs)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274209,  0] lib/fault.c:50(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274409,  0] lib/fault.c:51(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   ===============================================================
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274620,  0] lib/util.c:1117(smb_panic)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   PANIC (pid 4363): internal error
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.280478,  0] lib/util.c:1221(log_stack_trace)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   BACKTRACE: 27 stack frames:
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #0 smbd(log_stack_trace+0x1a) [0x7f0d69a8d4fa]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #1 smbd(smb_panic+0x2b) [0x7f0d69a8d5cb]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #2 smbd(+0x41a054) [0x7f0d69a7e054]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #3 /lib64/libc.so.6(+0x38f3832920) [0x7f0d65934920]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0xab) [0x7f0d671a9f0d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x19b) [0x7f0d671ab2c0]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x65) [0x7f0d671ab7aa]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x5e) [0x7f0d671ab830]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x3f) [0x7f0d671ab8bc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #9 /usr/lib64/libgfapi.so.0(glfs_close+0x66) [0x7f0d671a3e43]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #10 smbd(fd_close+0x43) [0x7f0d697be393]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #11 smbd(close_file+0x223) [0x7f0d697c4583]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #12 smbd(file_close_conn+0x4c) [0x7f0d697682fc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #13 smbd(close_cnum+0x2c) [0x7f0d697dd4cc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #14 smbd(reply_tdis+0x1d) [0x7f0d6978f99d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #15 smbd(+0x178424) [0x7f0d697dc424]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #16 smbd(+0x17883b) [0x7f0d697dc83b]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #17 smbd(+0x178c55) [0x7f0d697dcc55]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #18 smbd(run_events_poll+0x377) [0x7f0d69a9c877]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #19 smbd(smbd_process+0x86d) [0x7f0d697da79d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #20 smbd(+0x694f9f) [0x7f0d69cf8f9f]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #21 smbd(run_events_poll+0x377) [0x7f0d69a9c877]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #22 smbd(+0x438d2f) [0x7f0d69a9cd2f]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #23 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f0d662cd49d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #24 smbd(main+0xf3b) [0x7f0d69cfa29b]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #25 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f0d65920cdd]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #26 smbd(+0xf4ba9) [0x7f0d69758ba9]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.281771,  0] lib/fault.c:372(dump_core)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   dumping core in /var/log/core

Comment 5 Raghavendra Talur 2013-08-29 20:11:33 UTC

Patch posted for review at
https://code.engineering.redhat.com/gerrit/#/c/12213/

Comment 6 Lalatendu Mohanty 2013-09-03 10:08:14 UTC

I am not seeing this crash any-more with latest gluster packages i.e. glusterfs-server-3.4.0.30rhs-2.el6rhs.x86_64 and samba-common-3.6.9-160.3.

Hence Marking this bug verified

Comment 7 Scott Haines 2013-09-23 22:32:16 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.