improve gfapi performance relative to FUSE access.
Upstream patch: http://review.gluster.org/#/c/6131/
Please provide steps to verify.
Created attachment 817340 [details] test program to create and read small files Use a program similar to the attached code to test small file read performance (run it once with read_new() commented to create data set, then run it with write_new() commented to test just read perf)
https://code.engineering.redhat.com/gerrit/#/c/14861/
removing this from u1
Patches for this have been sent using the blocking bug 1025205. Patch posted for U1 at https://code.engineering.redhat.com/gerrit/#/c/15014/2 Patch posted for U2 at https://code.engineering.redhat.com/gerrit/15261
Tested with the attached program. The write performance has gone down on glusterfs 3.4.0.39rhs and read performance seem to be better. performance numbers are listed below... Followed the steps from comment #5 glusterfs 3.4.0.39rhs: write: [root@upgrade-1 exmp]# time ./smfile real 0m11.173s user 0m2.363s sys 0m5.845s read [root@upgrade-1 exmp]# time ./smfile real 0m3.403s user 0m1.162s sys 0m1.602s read-write time [root@upgrade-1 exmp]# time ./smfile real 0m16.525s user 0m4.237s sys 0m9.265s ========================= glusterfs 3.4.0.35rhs write: [root@boggs exmp]# time ./smfile real 0m6.670s user 0m2.381s sys 0m2.136s read: [root@boggs exmp]# time ./smfile real 0m4.235s user 0m1.930s sys 0m1.428s read-write [root@boggs exmp]# time ./smfile real 0m11.553s user 0m4.515s sys 0m3.553s
|| readCommented || writeCommented || readWriteEnabled || V35(rpm) V35+patch V39 || V35(rpm) V35+patch V39 || V35(rpm) V35+patch V39 real || 0m4.035s 0m3.322s 0m3.118s || 0m1.185s 0m1.151s 0m1.067s || 0m4.655s 0m4.184s 0m4.155s user || 0m1.316s 0m1.086s 0m1.000s || 0m0.366s 0m0.421s 0m0.340s || 0m1.526s 0m1.409s 0m1.361s sys || 0m0.824s 0m0.647s 0m0.522s || 0m0.249s 0m0.211s 0m0.203s || 0m0.898s 0m0.749s 0m0.771s I did quite a few runs of the program and I found performance to be better than v35. Write performance did increase for "user" time, where it increased from 0m0.366s to 0m0.421s. This is for a local, distribute volume. Need to see what else could have caused the poor write performance on Sachidananda's setup.
Last comment came out jumbled. Here is the same output. Build 35 ----------------------------------------------------------------------- root.42.211[Nov-07-2013- 8:49:29] >time ./readCommented real 0m4.035s user 0m1.316s sys 0m0.824s root.42.211[Nov-07-2013- 8:49:57] >time ./writeCommented real 0m1.185s user 0m0.366s sys 0m0.249s root.42.211[Nov-07-2013- 8:50:04] >time ./readWriteEnabled real 0m4.655s user 0m1.526s sys 0m0.898s ------------------------------------------------------------------ Build 35 + patch ----------------------------------------------------------------- root.42.211[Nov-07-2013- 9:36:35] >time ./readCommented real 0m3.322s user 0m1.086s sys 0m0.647s root.42.211[Nov-07-2013- 9:36:44] >time ./writeCommented real 0m1.151s user 0m0.421s sys 0m0.211s root.42.211[Nov-07-2013- 9:36:50] >time ./readWriteEnabled real 0m4.184s user 0m1.409s sys 0m0.749s ----------------------------------------------------------------- git tag 39 compiled ----------------------------------------------------------------- root.42.211[Nov-07-2013- 9:44:32] >time ./readCommented time ./ real 0m3.118s user 0m1.000s sys 0m0.522s root.42.211[Nov-07-2013- 9:44:41] >time ./writeCommented time real 0m1.067s user 0m0.340s sys 0m0.203s root.42.211[Nov-07-2013- 9:44:45] >time ./readWriteEnabled real 0m4.155s user 0m1.361s sys 0m0.771s ---------------------------------------------------------------- I did quite a few runs of the program and I found performance to be better than v35. "user" time for Write did increase from 0m0.366s to 0m0.421s with patch though. This is for a local, distribute volume. Need to see what else could have caused the poor write performance on Sachidananda's setup.
I tried on a distributed replicate setup (4 x 2), can you please login to: 10.70.37.137 - 3.4.0.35rhs 10.70.43.41 - 3.4.0.39rhs And take a look. I am using the program from Comment #5 And can be found in example-1
After testing this on 3 different setups with different versions(build 35, 39, 42 ,43), we are sure that write performance has not reduced because of this patch. I acknowledge that we need to find the reason why write performance is slow on Sachidananda's setup but we need not hold this bug in assigned because of that. A clean install of glusterfs build 43(with the patch) with 2x2 setup gives the same performance as 2x2 on a build 35(without the patch), hence moving this bug to ON_QA.
I still see some huge degradation in performance in latest release Vs build 35: These are the average results of around 12 runs: FOR glusterfs 3.4.0.44rhs built on Nov 13 2013 08:03:02 [root@upgrade-1 example-1]# time ./smfile-ro real 0m2.236s user 0m0.612s sys 0m1.132s [root@upgrade-1 example-1]# time ./smfile-wo real 0m10.783s user 0m2.485s sys 0m6.026s [root@upgrade-1 example-1]# time ./smfile-rw real 0m26.793s user 0m5.344s sys 0m14.657s [root@upgrade-1 example-1]# ============================================================ FOR glusterfs 3.4.0.35rhs built on Oct 15 2013 14:06:04 [root@boggs example-1]# time ./smfile-ro real 0m2.421s user 0m0.967s sys 0m0.786s [root@boggs example-1]# time ./smfile-wo real 0m6.686s user 0m2.523s sys 0m2.071s [root@boggs example-1]# time ./smfile-rw real 0m13.028s user 0m4.875s sys 0m3.905s Login details are same as mentioned in Comment 14 Please login and take a look. This is a fresh setup... please login and take a look.
As mentioned in my previous comment, I suspect it has something to with the environment. Can we try one of the two tests here to get more details? a. upgrade all machines in "boggs' gluster pool" to latest gluster and check the times. b. downgrade all machines in "upgrade-1's gluster pool" to gluster build 35 and check the times. This would help us to get a comparison on same set of machines.
We need to do an apple to apple comparison, which means we need to test with the patch and without the patch for a particular build. This means dev and QE need to work together to try the 44 build and a custom build without the patch in question.
Sachidananda provided his setup to test on. Here are the results. ========================================== Test1: build35(does not have the patch) --------------------------------------------------------------- [root@upgrade-1 ~]# rpm -qa | grep gluster glusterfs-libs-3.4.0.35rhs-1.el6rhs.x86_64 gluster-swift-account-1.8.0-6.11.el6rhs.noarch glusterfs-api-3.4.0.35rhs-1.el6rhs.x86_64 vdsm-gluster-4.10.2-23.0.1.el6rhs.noarch gluster-swift-1.8.0-6.11.el6rhs.noarch glusterfs-fuse-3.4.0.35rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.35rhs-1.el6rhs.x86_64 samba-glusterfs-3.6.9-160.3.el6rhs.x86_64 gluster-swift-object-1.8.0-6.11.el6rhs.noarch gluster-swift-container-1.8.0-6.11.el6rhs.noarch glusterfs-3.4.0.35rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.35rhs-1.el6rhs.x86_64 gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch glusterfs-rdma-3.4.0.35rhs-1.el6rhs.x86_64 gluster-swift-plugin-1.8.0-7.el6rhs.noarch [root@upgrade-1 ~]# time ./writePerformance real 0m20.605s user 0m4.667s sys 0m9.999s ========================================= Test2: build44(has the patch) [root@upgrade-1 ~]# rpm -qa | grep gluster glusterfs-3.4.0.44rhs-1.el6rhs.x86_64 glusterfs-api-devel-3.4.0.44rhs-1.el6rhs.x86_64 glusterfs-debuginfo-3.4.0.44rhs-1.el6rhs.x86_64 gluster-swift-account-1.8.0-6.11.el6rhs.noarch vdsm-gluster-4.10.2-23.0.1.el6rhs.noarch glusterfs-libs-3.4.0.44rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.44rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.44rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.44rhs-1.el6rhs.x86_64 glusterfs-devel-3.4.0.44rhs-1.el6rhs.x86_64 gluster-swift-1.8.0-6.11.el6rhs.noarch samba-glusterfs-3.6.9-160.3.el6rhs.x86_64 gluster-swift-object-1.8.0-6.11.el6rhs.noarch gluster-swift-container-1.8.0-6.11.el6rhs.noarch glusterfs-fuse-3.4.0.44rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.44rhs-1.el6rhs.x86_64 gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch gluster-swift-plugin-1.8.0-7.el6rhs.noarch [root@upgrade-1 ~]# time ./writePerformance real 0m15.589s user 0m3.583s sys 0m8.137s [root@upgrade-1 ~]# time ./writePerformance real 0m15.032s user 0m3.537s sys 0m8.415s [root@upgrade-1 ~]# time ./writePerformance real 0m17.114s user 0m3.802s sys 0m8.248s [root@upgrade-1 ~]# time ./writePerformance real 0m16.764s user 0m3.866s sys 0m8.104s [root@upgrade-1 ~]# time ./writePerformance real 0m19.429s user 0m3.739s sys 0m7.891s With this data clearly indicating that the patch is superior in performance.
Tested on new setup and with cleanup between .44 and .35 on the setup. The read performance is better. Write-perf almost the same / slightly better in some cases. .44 ~ 5.2secs and .35 ~ 5.4secs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html