Hide Forgot
Description of problem: Backtrace seen in the server logs while running the small file workload. Version-Release number of selected component (if applicable): [root@gqas003 smbd]# gluster --version glusterfs 3.7.9 built on Aug 24 2016 02:50:45 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. How reproducible: Tried once only Logs and cores are place at :- rhsqe-repo.lab.eng.blr.redhat.com:/var/www/html/sosreports/<bug> Steps to Reproduce: 1. Create setup using 4 servers and 4 clients 2. Mount it using SMBv1.0 ,1:1 , and run the small file work load using below command. 3. python /small-files/smallfile/smallfile_cli.py --operation create --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`" python /small-files/smallfile/smallfile_cli.py --operation read --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`" python /small-files/smallfile/smallfile_cli.py --operation append --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`" python /small-files/smallfile/smallfile_cli.py --operation rename --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`" python /small-files/smallfile/smallfile_cli.py --operation delete-renamed --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`" python /small-files/smallfile/smallfile_cli.py --operation mkdir --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`" python /small-files/smallfile/smallfile_cli.py --operation rmdir --threads 8 --file-size 64 --files 1000 --top /gluster-mount --pause 10000 --host-set "`echo $CLIENT | tr ' ' ','`" <Snip> 2016/12/02 12:24:44.022873, 0] ../source3/modules/vfs_glusterfs.c:258(vfs_gluster_connect) testvol: Initialized volume from server localhost [2016/12/03 18:21:39.321055, 0] ../lib/util/fault.c:78(fault_report) =============================================================== [2016/12/03 18:21:39.342668, 0] ../lib/util/fault.c:79(fault_report) INTERNAL ERROR: Signal 11 in pid 27859 (4.4.5) Please read the Trouble-Shooting section of the Samba HOWTO [2016/12/03 18:21:39.342713, 0] ../lib/util/fault.c:81(fault_report) =============================================================== [2016/12/03 18:21:39.342740, 0] ../source3/lib/util.c:791(smb_panic_s3) PANIC (pid 27859): internal error [2016/12/03 18:21:39.389539, 0] ../source3/lib/util.c:902(log_stack_trace) BACKTRACE: 30 stack frames: #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f1dcb887eaa] #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f1dcb887f80] #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f1dcdd7a57f] #3 /lib64/libsamba-util.so.0(+0x24796) [0x7f1dcdd7a796] #4 /lib64/libpthread.so.0(+0xf370) [0x7f1dcdfdb370] #5 /lib64/libglusterfs.so.0(+0x4fdb4) [0x7f1db4583db4] #6 /lib64/libglusterfs.so.0(+0x4ff0c) [0x7f1db4583f0c] #7 /lib64/libglusterfs.so.0(inode_table_destroy+0xf5) [0x7f1db4585df5] #8 /lib64/libglusterfs.so.0(inode_table_destroy_all+0x51) [0x7f1db4585ec1] #9 /lib64/libgfapi.so.0(glfs_fini+0x3cf) [0x7f1db4c4b2ef] #10 /usr/lib64/samba/vfs/glusterfs.so(+0x4076) [0x7f1db4e6b076] #11 /usr/lib64/samba/libsmbd-base-samba4.so(close_cnum+0x121) [0x7f1dcd930111] #12 /usr/lib64/samba/libsmbd-base-samba4.so(smbXsrv_tcon_disconnect+0x104) [0x7f1dcd95e714] #13 /usr/lib64/samba/libsmbd-base-samba4.so(reply_tdis+0x7c) [0x7f1dcd8e217c] #14 /usr/lib64/samba/libsmbd-base-samba4.so(+0x132e70) [0x7f1dcd928e70] #15 /usr/lib64/samba/libsmbd-base-samba4.so(+0x134c10) [0x7f1dcd92ac10] #16 /usr/lib64/samba/libsmbd-base-samba4.so(+0x136301) [0x7f1dcd92c301] #17 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f1dcb89d39c] #18 /lib64/libsmbconf.so.0(+0x355f0) [0x7f1dcb89d5f0] #19 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f1dca2c440d] #20 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f1dca2c45ab] #21 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x701) [0x7f1dcd92d651] #22 /usr/sbin/smbd(+0xc304) [0x7f1dce416304] #23 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f1dcb89d39c] #24 /lib64/libsmbconf.so.0(+0x355f0) [0x7f1dcb89d5f0] #25 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f1dca2c440d] #26 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f1dca2c45ab] #27 /usr/sbin/smbd(main+0x15d4) [0x7f1dce411ad4] #28 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f1dc9f20b35] #29 /usr/sbin/smbd(+0x7ea9) [0x7f1dce411ea9] [2016/12/03 18:21:39.389878, 0] ../source3/lib/dumpcore.c:303(dump_core) dumping core in /var/log/samba/cores/smbd </snip> Actual results: Trace observed in the logs. Small file workload completed. Expected results: No errors/warnings/trace should be observed. Additional info: I executed the run on 3.1.3 + RHEL 7.3
The Volume Info:- [root@gqas003 log]# gluster volume info Volume Name: testvol Type: Distributed-Replicate Volume ID: 17ae8b69-77d9-4ce8-9a23-9aac13e7d9f2 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gqas003.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0 Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1 Brick3: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2 Brick4: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3 Options Reconfigured: nfs.disable: off client.event-threads: 4 server.event-threads: 4 cluster.lookup-optimize: on server.allow-insecure: on performance.stat-prefetch: off performance.readdir-ahead: on