Hide Forgot
Description of problem: ******************************* smbd process crashes and there are multiple core dumps while doing following operations in parallel. Create a 2x2 volume and do multiple(3) cifs mount of the volume. 1.From mount point1: /mnt/cifs1/ ./iozone -i 0 -w -+n -c -C -e -s 200M -r 64k -t 50 -F /mnt/cifs5/file{1..50}.ioz 2.From smbclient execute the following: for i in `seq 1 50`; do echo file.ioz; done | xargs -n 1 -P 50 -I {} smbclient -msmb3 //10.70.X.Y/gluster-testvol/ foobar -c "get {} /dev/null" 3.From mount point2 : /mnt/cifs2 Do rename of files that are being created by iozone. 4.From mount point3: /mnt/cifs3 Execute ll ll on the mount point hangs. On the servers smbd process crashes and dumps multiple cores: Trace is as follows: Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.946414, 0] ../source3/modules/vfs_glusterfs.c:258(vfs_gluster_connect) Aug 31 18:01:27 dhcp47-64 smbd[443]: testvol: Initialized volume from server localhost Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.954382, 0] ../source3/smbd/oplock.c:193(update_num_read_oplocks) Aug 31 18:01:27 dhcp47-64 smbd[443]: PANIC: assert failed at ../source3/smbd/oplock.c(193): d->num_share_modes == 1 Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.954468, 0] ../source3/lib/util.c:791(smb_panic_s3) Aug 31 18:01:27 dhcp47-64 smbd[443]: PANIC (pid 443): assert failed: d->num_share_modes == 1 Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.955187, 0] ../source3/lib/util.c:902(log_stack_trace) Aug 31 18:01:27 dhcp47-64 smbd[443]: BACKTRACE: 28 stack frames: Aug 31 18:01:27 dhcp47-64 smbd[443]: #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f8709347eaa] Aug 31 18:01:27 dhcp47-64 smbd[443]: #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f8709347f80] Aug 31 18:01:27 dhcp47-64 smbd[443]: #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f870b83957f] Aug 31 18:01:27 dhcp47-64 smbd[443]: #3 /usr/lib64/samba/libsmbd-base-samba4.so(update_num_read_oplocks+0xf9) [0x7f870b428489] Aug 31 18:01:27 dhcp47-64 smbd[443]: #4 /usr/lib64/samba/libsmbd-base-samba4.so(+0x1173c6) [0x7f870b3cd3c6] Aug 31 18:01:27 dhcp47-64 smbd[443]: #5 /usr/lib64/samba/libsmbd-base-samba4.so(+0x119e29) [0x7f870b3cfe29] Aug 31 18:01:27 dhcp47-64 smbd[443]: #6 /usr/lib64/samba/libsmbd-base-samba4.so(create_file_default+0x1cf) [0x7f870b3d0e7f] Aug 31 18:01:27 dhcp47-64 smbd[443]: #7 /usr/lib64/samba/libsmbd-base-samba4.so(+0x1fbc1e) [0x7f870b4b1c1e] Aug 31 18:01:27 dhcp47-64 smbd[443]: #8 /usr/lib64/samba/libsmbd-base-samba4.so(smb_vfs_call_create_file+0xd8) [0x7f870b3d7b28] Aug 31 18:01:27 dhcp47-64 smbd[443]: #9 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_do_setfilepathinfo+0x1ec6) [0x7f870b3bd7e6] Aug 31 18:01:27 dhcp47-64 smbd[443]: #10 /usr/lib64/samba/libsmbd-base-samba4.so(+0x108ebb) [0x7f870b3beebb] Aug 31 18:01:27 dhcp47-64 smbd[443]: #11 /usr/lib64/samba/libsmbd-base-samba4.so(reply_trans2+0x8cd) [0x7f870b3c212d] Aug 31 18:01:27 dhcp47-64 smbd[443]: #12 /usr/lib64/samba/libsmbd-base-samba4.so(+0x132e90) [0x7f870b3e8e90] Aug 31 18:01:27 dhcp47-64 smbd[443]: #13 /usr/lib64/samba/libsmbd-base-samba4.so(+0x134c30) [0x7f870b3eac30] Aug 31 18:01:27 dhcp47-64 smbd[443]: #14 /usr/lib64/samba/libsmbd-base-samba4.so(+0x136321) [0x7f870b3ec321] Aug 31 18:01:27 dhcp47-64 smbd[443]: #15 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f870935d39c] Aug 31 18:01:27 dhcp47-64 smbd[443]: #16 /lib64/libsmbconf.so.0(+0x355f0) [0x7f870935d5f0] Aug 31 18:01:27 dhcp47-64 smbd[443]: #17 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f8707d8440d] Aug 31 18:01:27 dhcp47-64 smbd[443]: #18 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8707d845ab] Aug 31 18:01:27 dhcp47-64 smbd[443]: #19 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x701) [0x7f870b3ed671] Aug 31 18:01:27 dhcp47-64 smbd[443]: #20 /usr/sbin/smbd(+0xc304) [0x7f870bed7304] Aug 31 18:01:27 dhcp47-64 smbd[443]: #21 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f870935d39c] Aug 31 18:01:27 dhcp47-64 smbd[443]: #22 /lib64/libsmbconf.so.0(+0x355f0) [0x7f870935d5f0] Aug 31 18:01:27 dhcp47-64 smbd[443]: #23 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f8707d8440d] Aug 31 18:01:27 dhcp47-64 smbd[443]: #24 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8707d845ab] Aug 31 18:01:27 dhcp47-64 smbd[443]: #25 /usr/sbin/smbd(main+0x15d4) [0x7f870bed2ad4] Aug 31 18:01:27 dhcp47-64 smbd[443]: #26 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f87079e0b15] Aug 31 18:01:27 dhcp47-64 smbd[443]: #27 /usr/sbin/smbd(+0x7ea9) [0x7f870bed2ea9] Version-Release number of selected component (if applicable): samba-common-libs-4.4.3-8.el7rhgs.x86_64 samba-common-4.4.3-8.el7rhgs.noarch samba-vfs-glusterfs-4.4.3-8.el7rhgs.x86_64 samba-debuginfo-4.4.3-8.el7rhgs.x86_64 samba-libs-4.4.3-8.el7rhgs.x86_64 samba-client-libs-4.4.3-8.el7rhgs.x86_64 samba-4.4.3-8.el7rhgs.x86_64 samba-client-4.4.3-8.el7rhgs.x86_64 samba-common-tools-4.4.3-8.el7rhgs.x86_64 How reproducible: Tried twice with the similar tests. Steps to Reproduce: 1.As explained above in the description 2. 3. Actual results: ******************* ll on cifs mount hangs as smbd process crashes and dumps too many cores. Expected results: ****************** ll on mount point should not hang , and smbd process should not crash. Additional info: **************** core file and logs will be updated shortly.
Patches are available upstream, but only one of the the two is contained in the upstream version 4.4.3 we are using. Will be automatically contained when we rebase to 4.4.5 (or newer).
samba-4.4.5-1.el7rhgs.x86_64 1.From mount point1: /mnt/cifs1/ ./iozone -i 0 -w -+n -c -C -e -s 200M -r 64k -t 50 -F /mnt/cifs5/file{1..50}.ioz 2.From smbclient execute the following: for i in `seq 1 50`; do echo file.ioz; done | xargs -n 1 -P 50 -I {} smbclient -msmb3 //10.70.X.Y/gluster-testvol/ foobar -c "get {} /dev/null" 3.From mount point2 : /mnt/cifs2 Do rename of files that are being created by iozone. 4.From mount point3: /mnt/cifs3 Execute ll Performed the same test and no hangs/crash or issues seen. Also performed tests with multiple clients doing dd on file1 and ll from different mount points and clients.No issues are seen. Marking the BZ verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1913.html