Bug 1401386

Summary: [SMB] Backtrace observed on SMB v1.0 while running small files workload
Product: Red Hat Gluster Storage Reporter: Karan Sandha <ksandha>
Component: sambaAssignee: Sachin Prabhu <sprabhu>
Status: CLOSED NOTABUG QA Contact: Vivek Das <vdas>
Severity: high Docs Contact: Karan Sandha <ksandha>
Priority: unspecified    
Version: rhgs-3.1CC: gdeschner, rhs-smb, sprabhu, vdas
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-18 11:25:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Karan Sandha 2016-12-05 06:02:46 UTC 
Description of problem:
Backtrace seen in the server logs while running the small file workload.  

Version-Release number of selected component (if applicable):
[root@gqas003 smbd]# gluster --version
glusterfs 3.7.9 built on Aug 24 2016 02:50:45
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.


How reproducible:
Tried once only

Logs and cores are place at  :- 
 rhsqe-repo.lab.eng.blr.redhat.com:/var/www/html/sosreports/<bug>

Steps to Reproduce:
1. Create setup using 4 servers and 4 clients
2. Mount it using SMBv1.0 ,1:1 , and run the small file work load using below command.
3. python /small-files/smallfile/smallfile_cli.py --operation create --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`"


python /small-files/smallfile/smallfile_cli.py --operation read --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`"


python /small-files/smallfile/smallfile_cli.py --operation append --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`"


python /small-files/smallfile/smallfile_cli.py --operation rename --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`"


python /small-files/smallfile/smallfile_cli.py --operation delete-renamed --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`"


python /small-files/smallfile/smallfile_cli.py --operation mkdir --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`"


python /small-files/smallfile/smallfile_cli.py --operation rmdir --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`"


<Snip>

2016/12/02 12:24:44.022873,  0] ../source3/modules/vfs_glusterfs.c:258(vfs_gluster_connect)
  testvol: Initialized volume from server localhost
[2016/12/03 18:21:39.321055,  0] ../lib/util/fault.c:78(fault_report)
  ===============================================================
[2016/12/03 18:21:39.342668,  0] ../lib/util/fault.c:79(fault_report)
  INTERNAL ERROR: Signal 11 in pid 27859 (4.4.5)
  Please read the Trouble-Shooting section of the Samba HOWTO
[2016/12/03 18:21:39.342713,  0] ../lib/util/fault.c:81(fault_report)
  ===============================================================
[2016/12/03 18:21:39.342740,  0] ../source3/lib/util.c:791(smb_panic_s3)
  PANIC (pid 27859): internal error
[2016/12/03 18:21:39.389539,  0] ../source3/lib/util.c:902(log_stack_trace)
  BACKTRACE: 30 stack frames:
   #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f1dcb887eaa]
   #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f1dcb887f80]
   #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f1dcdd7a57f]
   #3 /lib64/libsamba-util.so.0(+0x24796) [0x7f1dcdd7a796]
   #4 /lib64/libpthread.so.0(+0xf370) [0x7f1dcdfdb370]
   #5 /lib64/libglusterfs.so.0(+0x4fdb4) [0x7f1db4583db4]
   #6 /lib64/libglusterfs.so.0(+0x4ff0c) [0x7f1db4583f0c]
   #7 /lib64/libglusterfs.so.0(inode_table_destroy+0xf5) [0x7f1db4585df5]
   #8 /lib64/libglusterfs.so.0(inode_table_destroy_all+0x51) [0x7f1db4585ec1]
   #9 /lib64/libgfapi.so.0(glfs_fini+0x3cf) [0x7f1db4c4b2ef]
   #10 /usr/lib64/samba/vfs/glusterfs.so(+0x4076) [0x7f1db4e6b076]
   #11 /usr/lib64/samba/libsmbd-base-samba4.so(close_cnum+0x121) [0x7f1dcd930111]
   #12 /usr/lib64/samba/libsmbd-base-samba4.so(smbXsrv_tcon_disconnect+0x104) [0x7f1dcd95e714]
   #13 /usr/lib64/samba/libsmbd-base-samba4.so(reply_tdis+0x7c) [0x7f1dcd8e217c]
   #14 /usr/lib64/samba/libsmbd-base-samba4.so(+0x132e70) [0x7f1dcd928e70]
   #15 /usr/lib64/samba/libsmbd-base-samba4.so(+0x134c10) [0x7f1dcd92ac10]
   #16 /usr/lib64/samba/libsmbd-base-samba4.so(+0x136301) [0x7f1dcd92c301]
   #17 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f1dcb89d39c]
   #18 /lib64/libsmbconf.so.0(+0x355f0) [0x7f1dcb89d5f0]
   #19 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f1dca2c440d]
   #20 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f1dca2c45ab]
   #21 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x701) [0x7f1dcd92d651]
   #22 /usr/sbin/smbd(+0xc304) [0x7f1dce416304]
   #23 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f1dcb89d39c]
   #24 /lib64/libsmbconf.so.0(+0x355f0) [0x7f1dcb89d5f0]
   #25 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f1dca2c440d]
   #26 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f1dca2c45ab]
   #27 /usr/sbin/smbd(main+0x15d4) [0x7f1dce411ad4]
   #28 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f1dc9f20b35]
   #29 /usr/sbin/smbd(+0x7ea9) [0x7f1dce411ea9]
[2016/12/03 18:21:39.389878,  0] ../source3/lib/dumpcore.c:303(dump_core)
  dumping core in /var/log/samba/cores/smbd

</snip>


Actual results:
Trace observed in the logs.
Small file workload completed. 

Expected results:
No errors/warnings/trace should be observed.


Additional info:
I executed the run on 3.1.3 + RHEL 7.3
Comment 2 Karan Sandha 2016-12-05 06:16:45 UTC 
The Volume Info:-
[root@gqas003 log]# gluster volume info
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 17ae8b69-77d9-4ce8-9a23-9aac13e7d9f2
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas003.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
nfs.disable: off
client.event-threads: 4
server.event-threads: 4
cluster.lookup-optimize: on
server.allow-insecure: on
performance.stat-prefetch: off
performance.readdir-ahead: on