Bug 1175088

Summary:	Files creates are very slow in directories already having files in it.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Raghavendra Talur <rtalur>
Component:	samba	Assignee:	Raghavendra Talur <rtalur>
Status:	CLOSED ERRATA	QA Contact:	surabhi <sbhaloth>
Severity:	high	Docs Contact:
Priority:	high
Version:	rhgs-3.0	CC:	divya, jarrpa, nlevinki, rcyriac, sbhaloth, surs
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.0.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	samba-3.6.509-169.4.el6rhs	Doc Type:	Bug Fix
Doc Text:	Previously, creating a new file over the SMB protocol, took a long time if the parent directory had many files in it. This was due to a bug in an optimization made to help Samba to ignore case comparison of requested file name to every entry in the directory. With this fix, the time taken to create a new file over the SMB protocol takes lesser time than before, even if the parent directory had many files in it.	Story Points:	---
Clone Of:
Clones:	1180130 (view as bug list)		Environment:
Last Closed:	2015-01-15 13:43:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1086681, 1162694, 1322735

Description Raghavendra Talur 2014-12-17 06:41:50 UTC

Description of problem:
When a directory has many files in it and a new file create is attempted over smb, we see that it takes a long time to complete.

Version-Release number of selected component (if applicable):



How reproducible:
Always


Steps to Reproduce:
1. Touch 120000 files in a directory in a gluster volume
2. Touch one more file and note the time taken.
3. Compare the above time with time required to create one new file over fuse now.


Actual results:
Create file in a dir having 120,000 files in it: 14.097 seconds.

Expected results:
It should not take that long.

Comment 1 Raghavendra Talur 2014-12-17 06:47:55 UTC

Root cause:

Before creating a new file, it must be validated that no such file exists.
As smb clients may be case insensitive, Samba has to make sure no file exists which has a case-insensitive match to the filename for which create request has come.

Brute force method of such validation in Samba is to do a readdir of the parent dir and case-insensitive string matching. However, this is a hugh performance hit if the dir already contains a lot of files.

Optimization for such cases is the get_real_filename API that the lower filesystem can provide. If this call exists, Samba pushes the responsibility of finding the real filename for a given string.

The key used for get_real_filename was not same in samba-glusterfs plugin and glusterfs hence causing the optimization to not work.

Comment 3 Jose A. Rivera 2014-12-17 21:57:34 UTC

Incorporated patch into a new build of Samba.

Comment 4 Sachidananda Urs 2014-12-19 10:31:54 UTC

Verified on build:  samba-3.6.509-169.4.el6rhs

I do not see any performance issues.

Comment 5 Raghavendra Talur 2014-12-19 10:52:04 UTC

This bug existed only for SMB access.

Steps to verify:
1. Have version of Samba < the fixed in version.
2. Create many files in a dir.
3. Add "case sensitive = yes" line in smb.conf under the share definition for this volume, say "gluster-vol1"
4. kill -HUP <pid of smbd> or just service restart smb
5. Create a single file in the same dir using cifs mount and record time taken.
6. Update samba
7. service restart smb and make sure the case sensitive line still exists in smb.conf, if not add it again.
8. Create one more file on the cifs mount and check the time taken.

Comment 6 surabhi 2014-12-19 11:53:13 UTC

[gluster-verify-vol]
	comment = For samba share of volume verify-vol
	path = /
	read only = No
	guest ok = Yes
	case sensitive = Yes
	vfs objects = glusterfs
	glusterfs:loglevel = 7
	glusterfs:logfile = /var/log/samba/glusterfs-verify-vol.%M.log
	glusterfs:volume = verify-vol


[root@dhcp159-63 per]# time touch file_perf_file11

real	0m0.025s
user	0m0.000s
sys	0m0.002s
[root@dhcp159-63 per]# time touch file_perf_file11

real	0m0.023s
user	0m0.001s
sys	0m0.001s

The time to create file in a directory on cifs mount where already 120000 file exists, it took much lesser time than before.

Marking the BZ as verified.

Comment 7 Divya 2015-01-08 07:32:14 UTC

Raghavendra,

Please review the edited doc text and sign-off.

Comment 8 Raghavendra Talur 2015-01-08 13:04:53 UTC

This bug will be fixed for 2.1.6.
Bug is https://bugzilla.redhat.com/show_bug.cgi?id=1180130

Comment 9 Raghavendra Talur 2015-01-13 09:04:20 UTC

The doc text looks good to me.

Comment 10 errata-xmlrpc 2015-01-15 13:43:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0038.html