Bug 1372007 - SMB: Smbd crashes when multiple operations are executed simultaneously from multiple cifs clients
Summary: SMB: Smbd crashes when multiple operations are executed simultaneously from m...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: samba
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: RHGS 3.1.3 Async
Assignee: Jose A. Rivera
QA Contact: surabhi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-31 16:46 UTC by surabhi
Modified: 2016-09-22 05:19 UTC (History)
4 users (show)

Fixed In Version: samba-4.4.5-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-22 05:19:28 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1913 0 normal SHIPPED_LIVE gluster-smb bug fix update 2016-09-22 09:18:05 UTC
Samba Project 11844 0 None None None 2016-09-01 06:55:15 UTC

Description surabhi 2016-08-31 16:46:11 UTC
Description of problem:
*******************************

smbd process crashes and there are multiple core dumps while doing following operations in parallel.

Create a 2x2 volume and do multiple(3) cifs mount of the volume.

1.From mount point1: /mnt/cifs1/
./iozone -i 0 -w -+n -c -C -e -s 200M -r 64k -t 50 -F /mnt/cifs5/file{1..50}.ioz
2.From smbclient execute the following:
for i in `seq 1 50`; do echo file.ioz; done  | xargs -n 1 -P 50 -I {} smbclient -msmb3 //10.70.X.Y/gluster-testvol/ foobar -c "get {} /dev/null"
3.From mount point2 : /mnt/cifs2
Do rename of files that are being created by iozone.
4.From mount point3: /mnt/cifs3
Execute ll

ll on the mount point hangs.

On the servers smbd process crashes and dumps multiple cores:

Trace is as follows:

Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.946414,  0] ../source3/modules/vfs_glusterfs.c:258(vfs_gluster_connect)
Aug 31 18:01:27 dhcp47-64 smbd[443]:  testvol: Initialized volume from server localhost
Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.954382,  0] ../source3/smbd/oplock.c:193(update_num_read_oplocks)
Aug 31 18:01:27 dhcp47-64 smbd[443]:  PANIC: assert failed at ../source3/smbd/oplock.c(193): d->num_share_modes == 1
Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.954468,  0] ../source3/lib/util.c:791(smb_panic_s3)
Aug 31 18:01:27 dhcp47-64 smbd[443]:  PANIC (pid 443): assert failed: d->num_share_modes == 1
Aug 31 18:01:27 dhcp47-64 smbd[443]: [2016/08/31 18:01:27.955187,  0] ../source3/lib/util.c:902(log_stack_trace)
Aug 31 18:01:27 dhcp47-64 smbd[443]:  BACKTRACE: 28 stack frames:
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f8709347eaa]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f8709347f80]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f870b83957f]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #3 /usr/lib64/samba/libsmbd-base-samba4.so(update_num_read_oplocks+0xf9) [0x7f870b428489]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #4 /usr/lib64/samba/libsmbd-base-samba4.so(+0x1173c6) [0x7f870b3cd3c6]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #5 /usr/lib64/samba/libsmbd-base-samba4.so(+0x119e29) [0x7f870b3cfe29]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #6 /usr/lib64/samba/libsmbd-base-samba4.so(create_file_default+0x1cf) [0x7f870b3d0e7f]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #7 /usr/lib64/samba/libsmbd-base-samba4.so(+0x1fbc1e) [0x7f870b4b1c1e]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #8 /usr/lib64/samba/libsmbd-base-samba4.so(smb_vfs_call_create_file+0xd8) [0x7f870b3d7b28]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #9 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_do_setfilepathinfo+0x1ec6) [0x7f870b3bd7e6]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #10 /usr/lib64/samba/libsmbd-base-samba4.so(+0x108ebb) [0x7f870b3beebb]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #11 /usr/lib64/samba/libsmbd-base-samba4.so(reply_trans2+0x8cd) [0x7f870b3c212d]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #12 /usr/lib64/samba/libsmbd-base-samba4.so(+0x132e90) [0x7f870b3e8e90]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #13 /usr/lib64/samba/libsmbd-base-samba4.so(+0x134c30) [0x7f870b3eac30]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #14 /usr/lib64/samba/libsmbd-base-samba4.so(+0x136321) [0x7f870b3ec321]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #15 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f870935d39c]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #16 /lib64/libsmbconf.so.0(+0x355f0) [0x7f870935d5f0]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #17 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f8707d8440d]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #18 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8707d845ab]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #19 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x701) [0x7f870b3ed671]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #20 /usr/sbin/smbd(+0xc304) [0x7f870bed7304]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #21 /lib64/libsmbconf.so.0(run_events_poll+0x16c) [0x7f870935d39c]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #22 /lib64/libsmbconf.so.0(+0x355f0) [0x7f870935d5f0]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #23 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f8707d8440d]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #24 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8707d845ab]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #25 /usr/sbin/smbd(main+0x15d4) [0x7f870bed2ad4]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #26 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f87079e0b15]
Aug 31 18:01:27 dhcp47-64 smbd[443]:   #27 /usr/sbin/smbd(+0x7ea9) [0x7f870bed2ea9]


Version-Release number of selected component (if applicable):

samba-common-libs-4.4.3-8.el7rhgs.x86_64
samba-common-4.4.3-8.el7rhgs.noarch
samba-vfs-glusterfs-4.4.3-8.el7rhgs.x86_64
samba-debuginfo-4.4.3-8.el7rhgs.x86_64
samba-libs-4.4.3-8.el7rhgs.x86_64
samba-client-libs-4.4.3-8.el7rhgs.x86_64
samba-4.4.3-8.el7rhgs.x86_64
samba-client-4.4.3-8.el7rhgs.x86_64
samba-common-tools-4.4.3-8.el7rhgs.x86_64

How reproducible:
Tried twice with the similar tests.

Steps to Reproduce:
1.As explained above in the description
2.
3.

Actual results:
*******************
ll on cifs mount hangs as smbd process crashes and dumps too many cores.

Expected results:
******************
ll on mount point should not hang , and smbd process should not crash.


Additional info:
****************
core file and logs will be updated shortly.

Comment 2 Michael Adam 2016-09-01 06:57:29 UTC
Patches are available upstream, but only one of the the two is contained in the upstream version 4.4.3 we are using. Will be automatically contained when we rebase to 4.4.5 (or newer).

Comment 6 surabhi 2016-09-13 07:56:58 UTC
samba-4.4.5-1.el7rhgs.x86_64

1.From mount point1: /mnt/cifs1/
./iozone -i 0 -w -+n -c -C -e -s 200M -r 64k -t 50 -F /mnt/cifs5/file{1..50}.ioz
2.From smbclient execute the following:
for i in `seq 1 50`; do echo file.ioz; done  | xargs -n 1 -P 50 -I {} smbclient -msmb3 //10.70.X.Y/gluster-testvol/ foobar -c "get {} /dev/null"
3.From mount point2 : /mnt/cifs2
Do rename of files that are being created by iozone.
4.From mount point3: /mnt/cifs3
Execute ll

Performed the same test and no hangs/crash or issues seen.

Also performed tests with multiple clients doing dd on file1 and ll from different mount points and clients.No issues are seen.

Marking the BZ verified.

Comment 8 errata-xmlrpc 2016-09-22 05:19:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1913.html


Note You need to log in before you can comment on or make changes to this bug.