Bug 1001614 - File operations caused smbd process to crash
File operations caused smbd process to crash
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: samba (Show other bugs)
2.1
Unspecified Unspecified
high Severity unspecified
: ---
: ---
Assigned To: Raghavendra Talur
Lalatendu Mohanty
:
Depends On:
Blocks: 1002511
  Show dependency treegraph
 
Reported: 2013-08-27 07:52 EDT by Lalatendu Mohanty
Modified: 2013-09-23 18:32 EDT (History)
6 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.29rhs-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1002511 (view as bug list)
Environment:
Last Closed: 2013-09-23 18:32:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Perl script used to create files (2.26 KB, application/x-perl)
2013-08-27 07:55 EDT, Lalatendu Mohanty
no flags Details

  None (edit)
Description Lalatendu Mohanty 2013-08-27 07:52:32 EDT
Description of problem:

I was creating some files on a win7 client using a Perl script on a samba-rhs share and after sometime the script failed and I saw a core file got created. 

However I am not able to reproduce the issue in another attempt. I guess I have hit some corner issue and wanted to log the bug with available information
 
Version-Release number of selected component (if applicable):

root@bvt-rhs1 core]# rpm -qa | grep samba
samba-doc-3.6.9-159.1.el6rhs.x86_64
samba-winbind-3.6.9-159.1.el6rhs.x86_64
samba-glusterfs-3.6.9-159.1.el6rhs.x86_64
samba-swat-3.6.9-159.1.el6rhs.x86_64
samba-winbind-krb5-locator-3.6.9-159.1.el6rhs.x86_64
samba-domainjoin-gui-3.6.9-159.1.el6rhs.x86_64
samba-common-3.6.9-159.1.el6rhs.x86_64
samba-3.6.9-159.1.el6rhs.x86_64
samba-client-3.6.9-159.1.el6rhs.x86_64
samba-winbind-devel-3.6.9-159.1.el6rhs.x86_64
samba4-libs-4.0.0-55.el6.rc4.x86_64
samba-winbind-clients-3.6.9-159.1.el6rhs.x86_64


Gluster version:
glusterfs-3.4.0.22rhs-2.el6rhs.x86_64

How reproducible:

Intermittent 

Steps to Reproduce:
1. Create a volume and start it 
2. Mount it on Win7 client 
3. Create files on the on the samba share in win7client and try to read the files from another windows client.

Actual results:

Below are the messages from /var/log/messages

#################################

[2013/08/27 15:10:38.395766,  0] lib/fault.c:48(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   INTERNAL ERROR: Signal 11 in pid 8761 (3.6.9-159.1.el6rhs)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.395929,  0] lib/fault.c:50(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.396136,  0] lib/fault.c:51(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   ===============================================================
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.396250,  0] lib/util.c:1117(smb_panic)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   PANIC (pid 8761): internal error
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.401721,  0] lib/util.c:1221(log_stack_trace)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   BACKTRACE: 25 stack frames:
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #0 smbd(log_stack_trace+0x1a) [0x7f622b7de58a]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #1 smbd(smb_panic+0x2b) [0x7f622b7de65b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #2 smbd(+0x41a0e4) [0x7f622b7cf0e4]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #3 /lib64/libc.so.6(+0x39dda32920) [0x7f6227688920]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0x59) [0x7f6228efc609]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x80) [0x7f6228efc7c0]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x46) [0x7f6228efcc06]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x58) [0x7f6228efcd78]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x73) [0x7f6228efd3e3]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #9 /usr/lib64/libgfapi.so.0(glfs_close+0x62) [0x7f6228efb4e2]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #10 smbd(fd_close+0x43) [0x7f622b50f383]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #11 smbd(close_file+0x223) [0x7f622b515573]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #12 smbd(reply_close+0x8b) [0x7f622b4e0a8b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #13 smbd(+0x178414) [0x7f622b52d414]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #14 smbd(+0x17882b) [0x7f622b52d82b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #15 smbd(+0x178c45) [0x7f622b52dc45]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #16 smbd(run_events_poll+0x377) [0x7f622b7ed907]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #17 smbd(smbd_process+0x86d) [0x7f622b52b78d]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #18 smbd(+0x69502f) [0x7f622ba4a02f]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #19 smbd(run_events_poll+0x377) [0x7f622b7ed907]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #20 smbd(+0x438dbf) [0x7f622b7eddbf]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #21 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f622802149d]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #22 smbd(main+0xf3b) [0x7f622ba4b32b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #23 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f6227674cdd]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #24 smbd(+0xf4b99) [0x7f622b4a9b99]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.406829,  0] lib/fault.c:372(dump_core)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   dumping core in /var/log/core

########################################
The script failed while writing files
 
C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Creating directory at W:\
Creating files in W:\......
Terminating on signal SIGINT(2)

C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Creating directory at W:\
Creating files in W:\......
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Terminating on signal SIGINT(2)

C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Expected results:

It should not fail

Additional info:
Comment 2 Lalatendu Mohanty 2013-08-27 07:55:38 EDT
Created attachment 790930 [details]
Perl script used to create files

Attached Perl script used to create files

perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3
Comment 3 Christopher R. Hertel 2013-08-27 22:46:20 EDT
Compare the sequence of calls to the stack trace in Bug #1000545.  This is probably the same bug.  This BZ, however, shows us that the core dump was caused by SIGSEGV.

It appears that glfs_refresh_inode_safe() is passing a bad pointer to something in glibc.

See Bug #1000545.  Please mark this as a duplicate if you agree that these are the same.
Comment 4 Lalatendu Mohanty 2013-08-28 08:46:45 EDT
The Issue again got reproduced today.

Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   ===============================================================
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274029,  0] lib/fault.c:48(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   INTERNAL ERROR: Signal 11 in pid 4363 (3.6.9-160.1.el6rhs)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274209,  0] lib/fault.c:50(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274409,  0] lib/fault.c:51(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   ===============================================================
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274620,  0] lib/util.c:1117(smb_panic)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   PANIC (pid 4363): internal error
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.280478,  0] lib/util.c:1221(log_stack_trace)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   BACKTRACE: 27 stack frames:
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #0 smbd(log_stack_trace+0x1a) [0x7f0d69a8d4fa]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #1 smbd(smb_panic+0x2b) [0x7f0d69a8d5cb]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #2 smbd(+0x41a054) [0x7f0d69a7e054]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #3 /lib64/libc.so.6(+0x38f3832920) [0x7f0d65934920]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0xab) [0x7f0d671a9f0d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x19b) [0x7f0d671ab2c0]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x65) [0x7f0d671ab7aa]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x5e) [0x7f0d671ab830]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x3f) [0x7f0d671ab8bc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #9 /usr/lib64/libgfapi.so.0(glfs_close+0x66) [0x7f0d671a3e43]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #10 smbd(fd_close+0x43) [0x7f0d697be393]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #11 smbd(close_file+0x223) [0x7f0d697c4583]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #12 smbd(file_close_conn+0x4c) [0x7f0d697682fc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #13 smbd(close_cnum+0x2c) [0x7f0d697dd4cc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #14 smbd(reply_tdis+0x1d) [0x7f0d6978f99d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #15 smbd(+0x178424) [0x7f0d697dc424]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #16 smbd(+0x17883b) [0x7f0d697dc83b]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #17 smbd(+0x178c55) [0x7f0d697dcc55]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #18 smbd(run_events_poll+0x377) [0x7f0d69a9c877]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #19 smbd(smbd_process+0x86d) [0x7f0d697da79d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #20 smbd(+0x694f9f) [0x7f0d69cf8f9f]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #21 smbd(run_events_poll+0x377) [0x7f0d69a9c877]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #22 smbd(+0x438d2f) [0x7f0d69a9cd2f]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #23 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f0d662cd49d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #24 smbd(main+0xf3b) [0x7f0d69cfa29b]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #25 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f0d65920cdd]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #26 smbd(+0xf4ba9) [0x7f0d69758ba9]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.281771,  0] lib/fault.c:372(dump_core)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   dumping core in /var/log/core
Comment 5 Raghavendra Talur 2013-08-29 16:11:33 EDT
Patch posted for review at
https://code.engineering.redhat.com/gerrit/#/c/12213/
Comment 6 Lalatendu Mohanty 2013-09-03 06:08:14 EDT
I am not seeing this crash any-more with latest gluster packages i.e. glusterfs-server-3.4.0.30rhs-2.el6rhs.x86_64 and samba-common-3.6.9-160.3.

Hence Marking this bug verified
Comment 7 Scott Haines 2013-09-23 18:32:16 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.