Bug 1001614

Summary:

File operations caused smbd process to crash

Product:

Red Hat Gluster Storage

Reporter:

Lalatendu Mohanty <lmohanty>

Component:

samba

Assignee:

Raghavendra Talur <rtalur>

Status:

CLOSED ERRATA

QA Contact:

Lalatendu Mohanty <lmohanty>

Severity:

unspecified

Docs Contact:

Priority:

high

Version:

2.1

CC:

amarts, crh, sbhaloth, sdharane, spalai, vagarwal

Target Milestone:

---

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

glusterfs-3.4.0.29rhs-1

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Clones:

1002511 (view as bug list)

Environment:

Last Closed:

2013-09-23 22:32:16 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Bug Depends On:

Bug Blocks:

1002511

Attachments:

Description	Flags
Perl script used to create files	none

Description Lalatendu Mohanty 2013-08-27 11:52:32 UTC

Description of problem:

I was creating some files on a win7 client using a Perl script on a samba-rhs share and after sometime the script failed and I saw a core file got created. 

However I am not able to reproduce the issue in another attempt. I guess I have hit some corner issue and wanted to log the bug with available information
 
Version-Release number of selected component (if applicable):

root@bvt-rhs1 core]# rpm -qa | grep samba
samba-doc-3.6.9-159.1.el6rhs.x86_64
samba-winbind-3.6.9-159.1.el6rhs.x86_64
samba-glusterfs-3.6.9-159.1.el6rhs.x86_64
samba-swat-3.6.9-159.1.el6rhs.x86_64
samba-winbind-krb5-locator-3.6.9-159.1.el6rhs.x86_64
samba-domainjoin-gui-3.6.9-159.1.el6rhs.x86_64
samba-common-3.6.9-159.1.el6rhs.x86_64
samba-3.6.9-159.1.el6rhs.x86_64
samba-client-3.6.9-159.1.el6rhs.x86_64
samba-winbind-devel-3.6.9-159.1.el6rhs.x86_64
samba4-libs-4.0.0-55.el6.rc4.x86_64
samba-winbind-clients-3.6.9-159.1.el6rhs.x86_64


Gluster version:
glusterfs-3.4.0.22rhs-2.el6rhs.x86_64

How reproducible:

Intermittent 

Steps to Reproduce:
1. Create a volume and start it 
2. Mount it on Win7 client 
3. Create files on the on the samba share in win7client and try to read the files from another windows client.

Actual results:

Below are the messages from /var/log/messages

#################################

[2013/08/27 15:10:38.395766,  0] lib/fault.c:48(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   INTERNAL ERROR: Signal 11 in pid 8761 (3.6.9-159.1.el6rhs)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.395929,  0] lib/fault.c:50(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.396136,  0] lib/fault.c:51(fault_report)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   ===============================================================
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.396250,  0] lib/util.c:1117(smb_panic)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   PANIC (pid 8761): internal error
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.401721,  0] lib/util.c:1221(log_stack_trace)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   BACKTRACE: 25 stack frames:
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #0 smbd(log_stack_trace+0x1a) [0x7f622b7de58a]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #1 smbd(smb_panic+0x2b) [0x7f622b7de65b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #2 smbd(+0x41a0e4) [0x7f622b7cf0e4]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #3 /lib64/libc.so.6(+0x39dda32920) [0x7f6227688920]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0x59) [0x7f6228efc609]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x80) [0x7f6228efc7c0]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x46) [0x7f6228efcc06]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x58) [0x7f6228efcd78]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x73) [0x7f6228efd3e3]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #9 /usr/lib64/libgfapi.so.0(glfs_close+0x62) [0x7f6228efb4e2]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #10 smbd(fd_close+0x43) [0x7f622b50f383]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #11 smbd(close_file+0x223) [0x7f622b515573]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #12 smbd(reply_close+0x8b) [0x7f622b4e0a8b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #13 smbd(+0x178414) [0x7f622b52d414]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #14 smbd(+0x17882b) [0x7f622b52d82b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #15 smbd(+0x178c45) [0x7f622b52dc45]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #16 smbd(run_events_poll+0x377) [0x7f622b7ed907]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #17 smbd(smbd_process+0x86d) [0x7f622b52b78d]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #18 smbd(+0x69502f) [0x7f622ba4a02f]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #19 smbd(run_events_poll+0x377) [0x7f622b7ed907]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #20 smbd(+0x438dbf) [0x7f622b7eddbf]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #21 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f622802149d]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #22 smbd(main+0xf3b) [0x7f622ba4b32b]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #23 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f6227674cdd]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:    #24 smbd(+0xf4b99) [0x7f622b4a9b99]
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]: [2013/08/27 15:10:38.406829,  0] lib/fault.c:372(dump_core)
Aug 27 15:10:38 bvt-rhs1 GlusterFS[8761]:   dumping core in /var/log/core

########################################
The script failed while writing files
 
C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Creating directory at W:\
Creating files in W:\......
Terminating on signal SIGINT(2)

C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Creating directory at W:\
Creating files in W:\......
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Cannot open file: Invalid argument
print() on closed filehandle FH at Win-CreateDirTreeNFiles.pl line 72.
Terminating on signal SIGINT(2)

C:\Users\hobbit2\Desktop>perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Expected results:

It should not fail

Additional info:

Comment 2 Lalatendu Mohanty 2013-08-27 11:55:38 UTC

Created attachment 790930 [details]
Perl script used to create files

Attached Perl script used to create files

perl Win-CreateDirTreeNFiles.pl W:\ 300 100 150 3 3

Comment 3 Christopher R. Hertel 2013-08-28 02:46:20 UTC

Compare the sequence of calls to the stack trace in Bug #1000545.  This is probably the same bug.  This BZ, however, shows us that the core dump was caused by SIGSEGV.

It appears that glfs_refresh_inode_safe() is passing a bad pointer to something in glibc.

See Bug #1000545.  Please mark this as a duplicate if you agree that these are the same.

Comment 4 Lalatendu Mohanty 2013-08-28 12:46:45 UTC

The Issue again got reproduced today.

Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   ===============================================================
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274029,  0] lib/fault.c:48(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   INTERNAL ERROR: Signal 11 in pid 4363 (3.6.9-160.1.el6rhs)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274209,  0] lib/fault.c:50(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274409,  0] lib/fault.c:51(fault_report)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   ===============================================================
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.274620,  0] lib/util.c:1117(smb_panic)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   PANIC (pid 4363): internal error
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.280478,  0] lib/util.c:1221(log_stack_trace)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   BACKTRACE: 27 stack frames:
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #0 smbd(log_stack_trace+0x1a) [0x7f0d69a8d4fa]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #1 smbd(smb_panic+0x2b) [0x7f0d69a8d5cb]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #2 smbd(+0x41a054) [0x7f0d69a7e054]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #3 /lib64/libc.so.6(+0x38f3832920) [0x7f0d65934920]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0xab) [0x7f0d671a9f0d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x19b) [0x7f0d671ab2c0]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x65) [0x7f0d671ab7aa]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x5e) [0x7f0d671ab830]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x3f) [0x7f0d671ab8bc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #9 /usr/lib64/libgfapi.so.0(glfs_close+0x66) [0x7f0d671a3e43]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #10 smbd(fd_close+0x43) [0x7f0d697be393]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #11 smbd(close_file+0x223) [0x7f0d697c4583]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #12 smbd(file_close_conn+0x4c) [0x7f0d697682fc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #13 smbd(close_cnum+0x2c) [0x7f0d697dd4cc]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #14 smbd(reply_tdis+0x1d) [0x7f0d6978f99d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #15 smbd(+0x178424) [0x7f0d697dc424]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #16 smbd(+0x17883b) [0x7f0d697dc83b]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #17 smbd(+0x178c55) [0x7f0d697dcc55]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #18 smbd(run_events_poll+0x377) [0x7f0d69a9c877]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #19 smbd(smbd_process+0x86d) [0x7f0d697da79d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #20 smbd(+0x694f9f) [0x7f0d69cf8f9f]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #21 smbd(run_events_poll+0x377) [0x7f0d69a9c877]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #22 smbd(+0x438d2f) [0x7f0d69a9cd2f]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #23 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f0d662cd49d]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #24 smbd(main+0xf3b) [0x7f0d69cfa29b]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #25 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f0d65920cdd]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:    #26 smbd(+0xf4ba9) [0x7f0d69758ba9]
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]: [2013/08/28 15:04:07.281771,  0] lib/fault.c:372(dump_core)
Aug 28 15:04:07 bvt-rhs1 GlusterFS[4363]:   dumping core in /var/log/core

Comment 5 Raghavendra Talur 2013-08-29 20:11:33 UTC

Patch posted for review at
https://code.engineering.redhat.com/gerrit/#/c/12213/

Comment 6 Lalatendu Mohanty 2013-09-03 10:08:14 UTC

I am not seeing this crash any-more with latest gluster packages i.e. glusterfs-server-3.4.0.30rhs-2.el6rhs.x86_64 and samba-common-3.6.9-160.3.

Hence Marking this bug verified

Comment 7 Scott Haines 2013-09-23 22:32:16 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html