Bug 1732376

Summary: Windows client fails to copy large file to GlusterFS volume share with fruit and streams_xattr VFS modules via Samba
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Anoop C S <anoopcs>
Component: md-cacheAssignee: Anoop C S <anoopcs>
Status: CLOSED CURRENTRELEASE QA Contact: Vivek Das <vdas>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.5CC: amukherj, gdeschner, madam, rcyriac, rhs-bugs, rhs-smb, sheggodu, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-6.0-13 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1726205 Environment:
Last Closed: 2019-11-06 10:19:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1726205, 1743782, 1785228    
Bug Blocks: 1696809    
Attachments:
Description Flags
windows error dialogue box none

Description Anoop C S 2019-07-23 09:14:38 UTC 
Created attachment 1592831 [details]
windows error dialogue box

+++ This bug was initially created as a clone of Bug #1726205 +++

Description of problem:
Windows client errors out while copying large file into GlusterFS volume share configured with fruit and streams_xattr VFS modules. See attachment for error message.

Version-Release number of selected component (if applicable):
master

How reproducible:
Always

Steps to Reproduce:
1. Create a basic distribute-replicate volume
2. Enable "group samba" volume set on the volume
3. Set up a Samba share with fruit and streams_xattr VFS modules
   vfs objects = fruit streams_xattr glusterfs
4. Connect to the share from a Windows client
5. Try to copy a large file(probably with size > 600M) into share

Actual results:
Windows client fails to copy large file with error(see attachment).

Expected results:
Copy completes successfully.

Additional info(root cause):
Problem lies in md-cache layer where it fails to update cache for xattrs with null value("\0"). Following steps reproduce the core issue on a plain FUSE mount:

# touch /mnt/glusterfs/foobar

# setfattr -n "user.DosStream.Zone.Identifier:\$DATA" -v "\0"
/mnt/glusterfs/foobar 

# echo $?
0

# getfattr -d -m . -e hex /mnt/glusterfs/foobar 
getfattr: Removing leading '/' from absolute path names
# file: mnt/glusterfs/foobar
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f74
3a733000
/mnt/glusterfs/foobar: user.DosStream.Zone.Identifier:$DATA: No such
attribute

# getfattr -d -m . -e hex /brick/brick1/foobar 
getfattr: Removing leading '/' from absolute path names
# file: brick/brick1/foobar
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c757374657264
5f627269636b5f743a733000
trusted.gfid=0xde7d450691b24107b0c03fac58d9e49e
trusted.gfid2path.17f514a2c19aaa57=0x30303030303030302d303030302d303030
302d303030302d3030303030303030303030312f666f6f626172
user.DosStream.Zone.Identifier:$DATA=0x00

# gluster v set vol performance.cache-samba-metadata off
volume set: success

# getfattr -d -m . -e hex /mnt/glusterfs/foobar 
getfattr: Removing leading '/' from absolute path names
# file: mnt/glusterfs/foobar
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f74
3a733000
user.DosStream.Zone.Identifier:$DATA=0x00

--- Additional comment from  on 2019-07-04 16:22:06 IST ---

Just raised this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1727062

Not sure if there are any similarities?

--- Additional comment from Anoop C S on 2019-07-05 17:11:08 IST ---
Comment 8 Vivek Das 2019-09-11 06:05:38 UTC 
Followed the below steps to reproduce and was not able to hit the issue with version 
"samba-4.9.8-108.el7rhgs.x86_64"
"glusterfs-fuse-6.0-13.el7rhgs.x86_64"

1. Create a distribute-replicate volume
2. Enable "group samba" volume set on the volume
3. Set up a Samba share with fruit and streams_xattr VFS modules
   vfs objects = fruit streams_xattr glusterfs
4. Connect to the share from a Windows client
5. Try to copy a large file(probably with size > 600M) into share

I copied a 2 GB file from local to share, within share and was copied successfully.
I tried from command prompt as well as manually.

Volume Name: volkswagen
Type: Distributed-Replicate
Volume ID: 39ebb750-95a7-4160-b4d2-a774289f5f31
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: dhcp43-166.lab.eng.blr.redhat.com:/bricks/brick0/volkswagen_brick0
Brick2: dhcp42-105.lab.eng.blr.redhat.com:/bricks/brick0/volkswagen_brick1
Brick3: dhcp42-196.lab.eng.blr.redhat.com:/bricks/brick0/volkswagen_brick2
Brick4: dhcp42-42.lab.eng.blr.redhat.com:/bricks/brick0/volkswagen_brick3
Brick5: dhcp43-166.lab.eng.blr.redhat.com:/bricks/brick1/volkswagen_brick4
Brick6: dhcp42-105.lab.eng.blr.redhat.com:/bricks/brick1/volkswagen_brick5
Brick7: dhcp42-196.lab.eng.blr.redhat.com:/bricks/brick1/volkswagen_brick6
Brick8: dhcp42-42.lab.eng.blr.redhat.com:/bricks/brick1/volkswagen_brick7
Brick9: dhcp43-166.lab.eng.blr.redhat.com:/bricks/brick2/volkswagen_brick8
Options Reconfigured:
performance.parallel-readdir: on
performance.readdir-ahead: on
performance.nl-cache-timeout: 600
performance.nl-cache: on
network.inode-lru-limit: 200000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
performance.cache-samba-metadata: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
user.cifs: enable
cluster.brick-multiplex: enable


Windows 10

Testparm

# Global parameters
[global]
	clustering = Yes
	ctdbd socket = /var/run/ctdb/ctdbd.socket
	disable spoolss = Yes
	kernel change notify = No
	load printers = No
	log file = /var/log/samba/log.%m
	max log size = 50
	security = USER
	server string = Samba Server Version %v
	show add printer wizard = No
	workgroup = MYGROUP
	fruit:aapl = yes
	idmap config * : backend = tdb
	aio read size = 4096
	cups options = raw
	include = /etc/samba/ctdb.conf
	kernel share modes = No
	map archive = No
	posix locking = No
	printing = bsd


[gluster-volkswagen]
	comment = For samba share of volume volkswagen
	guest ok = Yes
	path = /
	read only = No
	vfs objects = fruit streams_xattr glusterfs
	fruit:encoding = native
	glusterfs:loglevel = 7
	glusterfs:logfile = /var/log/samba/glusterfs-volkswagen-fruit.%M.log
	glusterfs:volume = volkswagen