Bug 983507 - Fileop tests fail on smb mount
Fileop tests fail on smb mount
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: samba (Show other bugs)
2.1
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Susant Kumar Palai
surabhi
: TestBlocker
Depends On:
Blocks: 956495
  Show dependency treegraph
 
Reported: 2013-07-11 06:52 EDT by surabhi
Modified: 2013-09-23 18:29 EDT (History)
10 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.19rhs-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-23 18:29:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description surabhi 2013-07-11 06:52:20 EDT
Description of problem:
Fileop is failing on a 6x2 distributed replicate volume on smb mount in a ctdb setup.

Version-Release number of selected component (if applicable):

[root@dhcp159-205 glusterfs]# rpm -qa | grep glusterfs
glusterfs-devel-3.4.0.12rhs.beta3-1.el6rhs.x86_64
glusterfs-server-3.4.0.12rhs.beta3-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.12rhs.beta3-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.12rhs.beta3-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.12rhs.beta3-1.el6rhs.x86_64
glusterfs-3.4.0.12rhs.beta3-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.12rhs.beta3-1.el6rhs.x86_64


How reproducible:
I tried three times and it failed.

Steps to Reproduce:
1. create a ctdb setup.
2. create a dis-rep volume and do a cifs mount on the client with virtual ip corresponding to the physical ip on which the volume is created.
3. From the mount point run fileop.

Actual results:
Fileop failed .

Expected results:
Fileop should pass.

Additional info:
For the first time fileop ran for ~1 hour an then failed.
Second time for about 2 hours
Third time it ran for almost 4 hours and failed.
Comment 3 surabhi 2013-07-15 08:02:33 EDT
Tried the fileop test on nfs mount and fuse mount as well ,it passed on both.
But takes 4-5 hours to finish.
Comment 5 surabhi 2013-07-26 06:13:15 EDT
Executed the test on following version:

glusterfs-3.4.0.12rhs.beta6-1.el6rhs.x86_64
samba-3.6.9-155.5.el6rhs.x86_64
2.6.32-358.14.1.el6.x86_64

Performed the same steps and fileop failed again.
Comment 6 Raghavendra Talur 2013-08-07 06:27:48 EDT
Surabhi,

Can you tell what was the error message from fileop?
eg, what operations failed.
Comment 7 surabhi 2013-08-07 08:29:22 EDT
Talur,

Here is the output from fileop log,


read:    Files =     27000 Total Time = 13.466583252 seconds
         Avg read(s)/sec      =      2004.96 ( 0.000498762 seconds/op)
         Best read(s)/sec     =      4236.67 ( 0.000236034 seconds/op)
         Worst read(s)/sec    =       240.79 ( 0.004153013 seconds/op)

access:  Files =     27000 Total Time = 44.049818754 seconds
         Avg access(s)/sec    =       612.94 ( 0.001631475 seconds/op)
         Best access(s)/sec   =      1057.03 ( 0.000946045 seconds/op)
         Worst access(s)/sec  =        77.97 ( 0.012825966 seconds/op)

chmod:   Files =     27000 Total Time = 118.453602791 seconds
         Avg chmod(s)/sec     =       227.94 ( 0.004387170 seconds/op)
         Best chmod(s)/sec    =       375.80 ( 0.002660990 seconds/op)
         Worst chmod(s)/sec   =        24.99 ( 0.040018797 seconds/op)

readdir: Files =       900 Total Time = 12.854415417 seconds
         Avg readdir(s)/sec   =        70.01 ( 0.014282684 seconds/op)
         Best readdir(s)/sec  =        97.70 ( 0.010235071 seconds/op)
         Worst readdir(s)/sec =        35.54 ( 0.028141022 seconds/op)

Link failed

Not included all messages but the last ones with error.
Comment 8 Raghavendra Talur 2013-08-07 10:32:51 EDT
It could possibly be the case that fileop is doing some test related to hard link
and with unix extensions being off the behaviour may be different than what it expects.
I have asked Chris to comment here if that is the case.
Comment 9 Ben Turner 2013-08-07 18:10:24 EDT
I am still seeing this on fs sanity runs on glusterfs-3.4.0.17rhs-1.el6rhs.x86_64.  To note I don't have ctdb configured, just regular samba.
Comment 10 Susant Kumar Palai 2013-08-08 02:06:36 EDT
Hi Ben,
can you please specify whether you used fuse mount(older method) or vfs module(new one) ?
Comment 11 surabhi 2013-08-08 05:08:49 EDT
Talur,

With the version glusterfs-3.4.0.16rhs-1.el6rhs.x86_64
I have tried the test with both the options unix extensions = yes
and unix extensions = no the test failed with the same link fail error.


read:    Files =     27000 Total Time = 10.803651094 seconds
         Avg read(s)/sec      =      2499.16 ( 0.000400135 seconds/op)
         Best read(s)/sec     =    262144.00 ( 0.000003815 seconds/op)
         Worst read(s)/sec    =       110.21 ( 0.009073973 seconds/op)

access:  Files =     27000 Total Time = 66.434024096 seconds
         Avg access(s)/sec    =       406.42 ( 0.002460519 seconds/op)
         Best access(s)/sec   =       661.46 ( 0.001511812 seconds/op)
         Worst access(s)/sec  =       133.78 ( 0.007474899 seconds/op)

chmod:   Files =     27000 Total Time = 281.460999250 seconds
         Avg chmod(s)/sec     =        95.93 ( 0.010424481 seconds/op)
         Best chmod(s)/sec    =       151.47 ( 0.006602049 seconds/op)
         Worst chmod(s)/sec   =        44.66 ( 0.022388935 seconds/op)

readdir: Files =       900 Total Time = 19.414461851 seconds
         Avg readdir(s)/sec   =        46.36 ( 0.021571624 seconds/op)
         Best readdir(s)/sec  =        60.36 ( 0.016567945 seconds/op)
         Worst readdir(s)/sec =        27.50 ( 0.036361933 seconds/op)

Link failed
Comment 12 surabhi 2013-08-08 05:51:55 EDT
This time even I didn't run on ctdb setup, it failed on regular samba vfs.
Comment 13 Raghavendra Talur 2013-08-08 07:06:34 EDT
Thanks Surabhi

It is bug with hard link in gfapi.

Example:

10.70.42.194#touch new
10.70.42.194#ls -l new
-rwxr-xr-x 1 root root 0 Aug  8 11:00 new
10.70.42.194#ls -li
total 0
13127611488624932007 -rwxr-xr-x 1 root root 0 Aug  8 11:00 new
10.70.42.194#ln new new2
ln: creating hard link `new2' => `new': Operation not supported
10.70.42.194#ls -li
total 0
13127611488624932007 -rwxr-xr-x 0 root root 0 Aug  8 11:00 new
13127611488624932007 -rwxr-xr-x 0 root root 0 Aug  8 11:00 new2

link gets created but we get a operation not supported error.

Susant is looking into it.
Comment 14 Christopher R. Hertel 2013-08-11 14:58:15 EDT
I have discussed this bug with one of the CIFS kernel client developers.  Under SMBv1, support for hard links is a feature of the Unix Extensions for SMB and is, therefore, only supported by a very limited set of clients.  Key among those is the Linux Kernel CIFS client.  By default, the kernel client negotiates the Unix Extensions with the Samba server, so hard links *should* be enabled in this scenario.

From an SMB perspective, this problem currently only impacts support for hard links via the Linux CIFS Kernel client when the Unix Extensions are negotiated over SMB1.  Hard links are not otherwise supported, so this is not a critical fix for SMB in Big Bend.

However, this will be an important fix for future releases, particularly if/when other access methods start using libgfapi.

Note that newer versions of Windows (NTFS) do support hard links, and that support for the Windows CreateHardLink function is available in SMB3.  Fixing this will be important for future SMB3 support.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365006.aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363860.aspx
Comment 15 Jeff Layton 2013-08-12 08:24:07 EDT
FWIW, creating a hardlink works just fine between the Linux CIFS client and samba when exporting a "normal" filesystem, both with and without unix extensions.
Comment 16 Christopher R. Hertel 2013-08-12 13:31:34 EDT
Previous tests show that creation of hardlinks works between the Linux CIFS kernel client and Samba when exporting file systems other than Gluster (e.g. XFS or EXT4).  As indicated above, in comment #13, the problem has been localized to libgfapi, the interface to the Gluster client.

The remaining open question has to do with the priority of the bug, with respect to others that currently require fixing.
Comment 17 Amar Tumballi 2013-08-13 14:53:11 EDT
bug 996063 is now fixed in upstream. patch downstream @ https://code.engineering.redhat.com/gerrit/#/c/11407/
Comment 18 Ben Turner 2013-08-15 17:21:51 EDT
Fileop is now passing for me on glusterfs-3.4.0.19rhs-2.el6rhs.x86_64:

INFO: Switching to /gluster-mount/run6691_fileop
INFO: Currently in directory /gluster-mount/run6691_fileop

Fileop:  Working in ., File size is 1,  Output is in Ops/sec. (A=Avg, B=Best, W=Worst)
mkdir:   Dirs =     27930 Total Time = 431.059383631 seconds
         Avg mkdir(s)/sec     =        64.79 ( 0.015433562 seconds/op)
         Best mkdir(s)/sec    =       156.25 ( 0.006400108 seconds/op)
         Worst mkdir(s)/sec   =         0.27 ( 3.740863085 seconds/op)

chdir:   Dirs =     27930 Total Time = 36.934317827 seconds
         Avg chdir(s)/sec     =       756.21 ( 0.001322389 seconds/op)
         Best chdir(s)/sec    =      1519.68 ( 0.000658035 seconds/op)
         Worst chdir(s)/sec   =         0.86 ( 1.164689064 seconds/op)

rmdir:   Dirs =     27930 Total Time = 364.032657862 seconds
         Avg rmdir(s)/sec     =        76.72 ( 0.013033751 seconds/op)
         Best rmdir(s)/sec    =       129.37 ( 0.007730007 seconds/op)
         Worst rmdir(s)/sec   =         0.42 ( 2.371411085 seconds/op)

create:  Files =     27000 Total Time = 271.101895571 seconds
         Avg create(s)/sec    =        99.59 ( 0.010040811 seconds/op)
         Best create(s)/sec   =       164.91 ( 0.006063938 seconds/op)
         Worst create(s)/sec  =         0.45 ( 2.199863911 seconds/op)

write:   Files =     27000 Total Time =  9.480272532 seconds
         Avg write(s)/sec     =      2848.02 ( 0.000351121 seconds/op)
         Best write(s)/sec    =      5433.04 ( 0.000184059 seconds/op)
         Worst write(s)/sec   =        14.76 ( 0.067746878 seconds/op)

close:   Files =     27000 Total Time = 26.621973038 seconds
         Avg close(s)/sec     =      1014.20 ( 0.000985999 seconds/op)
         Best close(s)/sec    =      3744.91 ( 0.000267029 seconds/op)
         Worst close(s)/sec   =         0.63 ( 1.577756882 seconds/op)

stat:    Files =     27000 Total Time = 22.073333025 seconds
         Avg stat(s)/sec      =      1223.20 ( 0.000817531 seconds/op)
         Best stat(s)/sec     =      1996.34 ( 0.000500917 seconds/op)
         Worst stat(s)/sec    =         6.67 ( 0.149971962 seconds/op)

open:    Files =     27000 Total Time = 37.427981853 seconds
         Avg open(s)/sec      =       721.39 ( 0.001386222 seconds/op)
         Best open(s)/sec     =       971.80 ( 0.001029015 seconds/op)
         Worst open(s)/sec    =         4.95 ( 0.201858044 seconds/op)

read:    Files =     27000 Total Time =  5.365317583 seconds
         Avg read(s)/sec      =      5032.32 ( 0.000198715 seconds/op)
         Best read(s)/sec     =      8924.05 ( 0.000112057 seconds/op)
         Worst read(s)/sec    =       965.10 ( 0.001036167 seconds/op)

access:  Files =     27000 Total Time = 20.328207016 seconds
         Avg access(s)/sec    =      1328.20 ( 0.000752897 seconds/op)
         Best access(s)/sec   =      1858.35 ( 0.000538111 seconds/op)
         Worst access(s)/sec  =       445.02 ( 0.002247095 seconds/op)

chmod:   Files =     27000 Total Time = 38.459837675 seconds
         Avg chmod(s)/sec     =       702.03 ( 0.001424438 seconds/op)
         Best chmod(s)/sec    =      1014.10 ( 0.000986099 seconds/op)
         Worst chmod(s)/sec   =       264.19 ( 0.003785133 seconds/op)

readdir: Files =       900 Total Time =  9.273023367 seconds
         Avg readdir(s)/sec   =        97.06 ( 0.010303359 seconds/op)
         Best readdir(s)/sec  =       108.15 ( 0.009246111 seconds/op)
         Worst readdir(s)/sec =        15.03 ( 0.066555023 seconds/op)

link:    Files =     27000 Total Time = 197.369472027 seconds
         Avg link(s)/sec      =       136.80 ( 0.007309980 seconds/op)
         Best link(s)/sec     =       220.71 ( 0.004530907 seconds/op)
         Worst link(s)/sec    =         0.44 ( 2.255393982 seconds/op)

unlink:  Files =     27000 Total Time = 98.380872250 seconds
         Avg unlink(s)/sec    =       274.44 ( 0.003643736 seconds/op)
         Best unlink(s)/sec   =       512.81 ( 0.001950026 seconds/op)
         Worst unlink(s)/sec  =         0.51 ( 1.962489128 seconds/op)

delete:  Files =     27000 Total Time = 97.155341148 seconds
         Avg delete(s)/sec    =       277.91 ( 0.003598346 seconds/op)
         Best delete(s)/sec   =       509.95 ( 0.001960993 seconds/op)
         Worst delete(s)/sec  =         0.43 ( 2.331680059 seconds/op)


real	29m28.201s
user	0m3.084s
sys	0m30.649s

I don't run on ctdb setups though, could one of the ctdb people re run before we mark as verified?
Comment 19 surabhi 2013-08-16 23:39:58 EDT
glusterfs-3.4.0.19rhs-2.el6rhs.x86_64
I ran it on ctdb setup and normal smb setup the test passed on both.
/mnt/smb_cifs/run10963
fileop
executing fileop
start:15:30:16

real    47m20.089s
user    0m2.084s
sys     0m29.210s
end:16:17:36
1
Total 1 tests were successful
So moving it to verified.
Comment 20 Scott Haines 2013-09-23 18:29:50 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.