Bug 1602824

Summary: SMBD crashes when streams_attr VFS is used with Gluster VFS
Product: [Fedora] Fedora Reporter: ryan
Component: sambaAssignee: Guenther Deschner <gdeschner>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 30CC: abokovoy, anoopcs, asn, atumball, bugs, dan.ragnar, gdeschner, jarrpa, joao.bauto, jstephen, lmohanty, madam, pgurusid, sbose, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: samba-4.10.5-1.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1636246 (view as bug list) Environment:
Last Closed: 2019-07-06 04:08:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1636246    
Attachments:
Description Flags
SMB log from server showing the panics
none
Debug level 10 log of working test
none
Debug level 10 log of failed test01 none

Description ryan 2018-07-18 15:05:03 UTC
Created attachment 1459735 [details]
SMB log from server showing the panics

Description of problem:
SMBD thread panics when streams_xattr is used before gluster VFS

Version-Release number of selected component (if applicable):
Samba (tested with): 4.7.7, 4.7.8, 4.8.3
Gluster: 3.12.11

How reproducible:
Can reproduce everytime


Steps to Reproduce:
1.In Samba, set log level to 1
2.In Samba, create a share with the following:

[share]
vfs objects = fruit streams_xattr glusterfs
guest ok = yes
read only = no
glusterfs:volume = myVol01
path =  "/data/share01"

3.Restart Samba service
4.Tail the Samba log file at /var/log/samba
5.Connect to the share via an OS X client (Tested with 10.13.4)
6. Browse share at finder level

Actual results:
SMBD thread panics and kills clients connection

Expected results:
Share lists normally 


Additional info:
Don't happen from a Windows system

Comment 1 ryan 2018-09-18 11:58:35 UTC
Same issue here

Comment 2 Shyamsundar 2018-10-23 14:54:02 UTC
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.

Comment 3 joao.bauto 2019-01-10 17:14:19 UTC
I'm getting this issue with Gluster 5.2 and Samba 4.8.3 however it only kills the connection once a user begins copying data to the share.

Putting streams_xattr after glusterfs prevents the issue but its painly slow when list a directory.

[2019/01/10 17:04:09.313042,  0] ../lib/util/fault.c:261(log_stack_trace)
  BACKTRACE: 28 stack frames:
   #0 /lib64/libsamba-util.so.0(log_stack_trace+0x1a) [0x7f8fab1e5aaa]
   #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f8fa90e0930]
   #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f8fab1e5b8f]
   #3 /lib64/libsamba-util.so.0(+0x24dc6) [0x7f8fab1e5dc6]
   #4 /lib64/libpthread.so.0(+0xf6d0) [0x7f8fab64e6d0]
   #5 /usr/lib64/samba/vfs/glusterfs.so(+0x43c3) [0x7f8f900173c3]
   #6 /usr/lib64/samba/libsmbd-base-samba4.so(is_posix_locked+0xd2) [0x7f8faae39642]
   #7 /usr/lib64/samba/libsmbd-base-samba4.so(brl_locktest+0x184) [0x7f8faae37174]
   #8 /usr/lib64/samba/libsmbd-base-samba4.so(strict_lock_check_default+0x7b) [0x7f8faae3255b]
   #9 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_smb2_request_process_read+0x51e) [0x7f8faae04e1e]
   #10 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_smb2_request_dispatch+0x164c) [0x7f8faadf75bc]
   #11 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_smb2_request_dispatch_immediate+0x5d) [0x7f8faadf8a2d]
   #12 /lib64/libtevent.so.0(tevent_common_loop_immediate+0xda) [0x7f8fa7adc9ea]
   #13 /lib64/libtevent.so.0(+0xa5fd) [0x7f8fa7ae15fd]
   #14 /lib64/libtevent.so.0(+0x8c07) [0x7f8fa7adfc07]
   #15 /lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f8fa7adbffd]
   #16 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8fa7adc22b]
   #17 /lib64/libtevent.so.0(+0x8ba7) [0x7f8fa7adfba7]
   #18 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x5e1) [0x7f8faade6851]
   #19 /usr/sbin/smbd(+0xcdd0) [0x558d16af5dd0]
   #20 /lib64/libtevent.so.0(+0xa83b) [0x7f8fa7ae183b]
   #21 /lib64/libtevent.so.0(+0x8c07) [0x7f8fa7adfc07]
   #22 /lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f8fa7adbffd]
   #23 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f8fa7adc22b]
   #24 /lib64/libtevent.so.0(+0x8ba7) [0x7f8fa7adfba7]
   #25 /usr/sbin/smbd(main+0x16d7) [0x558d16af0d37]
   #26 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8fa772c445]
   #27 /usr/sbin/smbd(+0x83f0) [0x558d16af13f0]
[2019/01/10 17:04:09.313331,  0] ../source3/lib/dumpcore.c:315(dump_core)

Comment 4 ryan 2019-05-23 10:14:05 UTC
I've carried out some more testing with Samba 4.9.8 and two the different Gluster VFS plugins and have found the standard 'glusterfs' causes and issue when used in conjunction with fruit.
SMB.conf below:
------------------------------------------------------------------------------------------------------------

[global]
security = ADS
workgroup = MAGENTA
realm = MAGENTA.LOCAL
netbios name = DEVCLUSTER01
max protocol = SMB3
min protocol = SMB2
ea support = yes
clustering = yes
server signing = no
max log size = 10000
glusterfs:loglevel = 5
log file = /var/log/samba/log-%M.smbd
logging = file
log level = 3
template shell = /sbin/nologin
winbind offline logon = false
winbind refresh tickets = yes
winbind enum users = Yes
winbind enum groups = Yes
allow trusted domains = yes
passdb backend = tdbsam
idmap cache time = 604800
idmap negative cache time = 300
winbind cache time = 604800
idmap config magenta:backend = rid
idmap config magenta:range = 10000-999999
idmap config * : backend = tdb
idmap config * : range = 3000-7999
guest account = nobody
map to guest = bad user
force directory mode = 0777
force create mode = 0777
create mask = 0777
directory mask = 0777
hide unreadable = no
store dos attributes = no
unix extensions = no
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
glusterfs:volfile_server = localhost
kernel share modes = No
strict locking = auto
oplocks = yes
durable handles = yes
kernel oplocks = no
posix locking = no
level2 oplocks = no
readdir_attr:aapl_rsize = yes
readdir_attr:aapl_finder_info = no
readdir_attr:aapl_max_access = no

[VFS_Apple_Tests]
read only = no
guest ok = yes
vfs objects = catia fruit streams_xattr glusterfs
glusterfs:volume = mcv01
path = "/data"
recycle:keeptree = yes
recycle:directory_mode = 0770
recycle:versions = yes
recycle:repository = .recycle
recycle:subdir_mode = 0777
valid users = "nobody" @"audio" "MAGENTA\r.launchbury"

[FUSE_Apple_Tests]
read only = no
guest ok = yes
vfs objects = catia fruit streams_xattr glusterfs_fuse
path = "/mnt/mcv01/data"
recycle:repository = .recycle
recycle:keeptree = yes
recycle:versions = yes
recycle:directory_mode = 0770
recycle:subdir_mode = 0777

------------------------------------------------------------------------------------------------------------

Testing:
Using Finder and OS X 10.14.4

- Connect to each share
- Try to create a folder in sub-folder of each share

Result: 
- Share 'VFS_Apple_Tests' produced error -50
- Share 'FUSE_Apple_Tests' worked as expected

Debug level 10 logs attached

Comment 5 ryan 2019-05-23 10:16:14 UTC
Created attachment 1572423 [details]
Debug level 10 log of working test

Comment 6 ryan 2019-05-23 10:16:42 UTC
Created attachment 1572424 [details]
Debug level 10 log of failed test01

Comment 7 Guenther Deschner 2019-06-19 16:12:49 UTC
Converting this bug to Samba (where it belongs) and to Fedora because there we can provide an updated package containing a bugfix.

Comment 8 Fedora Update System 2019-06-20 20:04:48 UTC
FEDORA-2019-8015e5dc40 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-8015e5dc40

Comment 9 Fedora Update System 2019-06-22 06:04:26 UTC
samba-4.10.5-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-8015e5dc40

Comment 10 ryan 2019-07-03 15:59:43 UTC
I've recreated our setup on some Fedora 30 VMs and can confirm the issue has been resolved.

Many thanks,
Ryan

Comment 11 Guenther Deschner 2019-07-03 16:06:48 UTC
Thank you very much for testing it, much appreciated!

Comment 12 ryan 2019-07-03 16:12:21 UTC
No problem!

When QA get's approval, will this replace the current 4.10.5 package in the repos, or will it be in 4.10.6?

Best,
Ryan

Comment 13 Guenther Deschner 2019-07-04 08:52:45 UTC
It will be part of the official 4.10.5 package and also be in 4.10.6 (which is currently delayed, was due yesterday...)

Comment 14 ryan 2019-07-04 11:11:44 UTC
Great, thanks!

Comment 15 Fedora Update System 2019-07-06 04:08:51 UTC
samba-4.10.5-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.