Bug 1406723 - [Perf] : significant Performance regression seen with disperse volume when compared with 3.1.3
Summary: [Perf] : significant Performance regression seen with disperse volume when co...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: disperse
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: RHGS 3.2.0
Assignee: Ashish Pandey
QA Contact: Ambarish
URL:
Whiteboard:
Depends On:
Blocks: 1351528 1408809
TreeView+ depends on / blocked
 
Reported: 2016-12-21 10:37 UTC by nchilaka
Modified: 2017-03-23 05:58 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.8.4-17
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1408809 (view as bug list)
Environment:
Last Closed: 2017-03-23 05:58:45 UTC


Attachments (Terms of Use)
3.2 numbers (2.22 KB, text/plain)
2016-12-21 13:16 UTC, nchilaka
no flags Details
3.1.3 numbers (6.85 KB, text/plain)
2016-12-21 13:16 UTC, nchilaka
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC
Red Hat Bugzilla 1408655 None None None Never
Red Hat Bugzilla 1419859 None None None Never

Internal Links: 1408655 1419859

Description nchilaka 2016-12-21 10:37:56 UTC
I did a comparison b/w 3.1.3's 3.7.9-12 build and 3.2's 3.8.4-9 build on EC volume
I observed a significant drop in performance and the numbers are as below:






Setup Info:
creation of 10k zero byte files
3.1.3==>took 1min 12sec
3.2==>took 3min 09 sec
--->that is almost 60% drop


More numbers will be published soon

Comment 2 nchilaka 2016-12-21 10:59:23 UTC
below are the numbers
(in both cases, quotas was enabled, uss was turned on)
2x(4+2) volume on rhel7.3
File Operation		3.1.3	3.2
touch to create new file 10000 files		1min 12sec	3min 9sec
linux untar of kernel image 4.9		25min 23Sec	43min 15sec
ls -lRt of untarred directory		51Sec	59.3Sec
rm -rf of the 10k files		50sec	1min 15sec
Stat * of the folder hosting 10000 files		8sec	14sec

Comment 3 nchilaka 2016-12-21 11:26:16 UTC
Operation	3.1.3	3.2	Drop in Performance
touch	72	189	61.9047619048
untar	1523	2595	41.3102119461
ls	51	59	13.5593220339
rm -rf	50	75	33.3333333333
stat	8	14	42.8571428571

Comment 4 nchilaka 2016-12-21 11:26:53 UTC
3.1.3 numbers:
Was run in 3.2 time frame
Setup info
Ec volume build (3.7.9-12) 2nd async build after GA
rhel 7.3 
6 VMs each of 8GB

Client:
16GB RHEL7.3
 
root@dhcp35-37 ~]# gluster v info
 
Volume Name: disperse
Type: Distributed-Disperse
Volume ID: ccede272-2cde-4b55-be94-51581289eb56
Status: Started
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.35.37:/rhs/brick2/disperse
Brick2: 10.70.35.116:/rhs/brick2/disperse
Brick3: 10.70.35.239:/rhs/brick2/disperse
Brick4: 10.70.35.135:/rhs/brick2/disperse
Brick5: 10.70.35.8:/rhs/brick2/disperse
Brick6: 10.70.35.196:/rhs/brick2/disperse
Brick7: 10.70.35.37:/rhs/brick3/disperse
Brick8: 10.70.35.116:/rhs/brick3/disperse
Brick9: 10.70.35.239:/rhs/brick3/disperse
Brick10: 10.70.35.135:/rhs/brick3/disperse
Brick11: 10.70.35.8:/rhs/brick3/disperse
Brick12: 10.70.35.196:/rhs/brick3/disperse
Options Reconfigured:
features.uss: enable
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on
[root@dhcp35-37 ~]# gluster v status
Status of volume: disperse
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.37:/rhs/brick2/disperse      49154     0          Y       22200
Brick 10.70.35.116:/rhs/brick2/disperse     49154     0          Y       21974
Brick 10.70.35.239:/rhs/brick2/disperse     49154     0          Y       21982
Brick 10.70.35.135:/rhs/brick2/disperse     49154     0          Y       21966
Brick 10.70.35.8:/rhs/brick2/disperse       49154     0          Y       21998
Brick 10.70.35.196:/rhs/brick2/disperse     49154     0          Y       21999
Brick 10.70.35.37:/rhs/brick3/disperse      49155     0          Y       22219
Brick 10.70.35.116:/rhs/brick3/disperse     49155     0          Y       21993
Brick 10.70.35.239:/rhs/brick3/disperse     49155     0          Y       22001
Brick 10.70.35.135:/rhs/brick3/disperse     49155     0          Y       21985
Brick 10.70.35.8:/rhs/brick3/disperse       49155     0          Y       22017
Brick 10.70.35.196:/rhs/brick3/disperse     49155     0          Y       22018
Snapshot Daemon on localhost                49156     0          Y       22343
NFS Server on localhost                     2049      0          Y       22353
Self-heal Daemon on localhost               N/A       N/A        Y       22244
Quota Daemon on localhost                   N/A       N/A        Y       22298
Snapshot Daemon on 10.70.35.135             49156     0          Y       22089
NFS Server on 10.70.35.135                  2049      0          Y       22097
Self-heal Daemon on 10.70.35.135            N/A       N/A        Y       22012
Quota Daemon on 10.70.35.135                N/A       N/A        Y       22055
Snapshot Daemon on 10.70.35.196             49156     0          Y       22123
NFS Server on 10.70.35.196                  2049      0          Y       22131
Self-heal Daemon on 10.70.35.196            N/A       N/A        Y       22045
Quota Daemon on 10.70.35.196                N/A       N/A        Y       22086
Snapshot Daemon on 10.70.35.239             49156     0          Y       22105
NFS Server on 10.70.35.239                  2049      0          Y       22114
Self-heal Daemon on 10.70.35.239            N/A       N/A        Y       22028
Quota Daemon on 10.70.35.239                N/A       N/A        Y       22069
Snapshot Daemon on 10.70.35.116             49156     0          Y       22097
NFS Server on 10.70.35.116                  2049      0          Y       22105
Self-heal Daemon on 10.70.35.116            N/A       N/A        Y       22020
Quota Daemon on 10.70.35.116                N/A       N/A        Y       22061
Snapshot Daemon on 10.70.35.8               49156     0          Y       22121
NFS Server on 10.70.35.8                    2049      0          Y       22129
Self-heal Daemon on 10.70.35.8              N/A       N/A        Y       22048
Quota Daemon on 10.70.35.8                  N/A       N/A        Y       22088
 
Task Status of Volume disperse
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-37 ~]# cat /etc/redhat-*
cat: /etc/redhat-access-insights: Is a directory
Red Hat Enterprise Linux Server release 7.3 (Maipo)
Red Hat Gluster Storage Server 3.1 Update 3
[root@dhcp35-37 ~]# rpm -qa|grep gluster
glusterfs-libs-3.7.9-12.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-fuse-3.7.9-12.el7rhgs.x86_64
glusterfs-client-xlators-3.7.9-12.el7rhgs.x86_64
glusterfs-server-3.7.9-12.el7rhgs.x86_64
python-gluster-3.7.9-12.el7rhgs.noarch
glusterfs-api-3.7.9-12.el7rhgs.x86_64
glusterfs-geo-replication-3.7.9-12.el7rhgs.x86_64
vdsm-gluster-4.17.33-1.el7rhgs.noarch
glusterfs-3.7.9-12.el7rhgs.x86_64
glusterfs-cli-3.7.9-12.el7rhgs.x86_64
gluster-nagios-addons-0.2.7-1.el7rhgs.x86_64
[root@dhcp35-37 ~]# 

####### FINDINGS###########################
Fuse mount:
(all bricks are up)
touch of 10k files in /rootOFvol/dir1/ ===>took 1min 12sec
[root@rhs-client45 dir1]# date;touch file{1..10000};date
Wed Dec 21 15:40:31 IST 2016
Wed Dec 21 15:41:43 IST 2016

post above step immediate ls -lRt on root ===>took less than 1sec  to start displaying and o/p was completed in 1sec
real	0m0.920s
user	0m0.084s
sys	0m0.126s

post above step immediate find * on root ===>took less than 1sec to start displaying and o/p was completed in 1sec
real	0m0.863s
user	0m0.014s
sys	0m0.021s


post above step immediate stat * on /rootOfvol/dir1/===>took about 1 sec to respond and 8 sec for completing the total o/p
real	0m8.358s
user	0m0.482s
sys	0m0.651s

post above step rm -rf on root ===>took about 50.720s
[root@rhs-client45 disperse]# ls
dir1
[root@rhs-client45 disperse]# time rm -rf *
real	0m50.711s
user	0m0.045s
sys	0m0.841s


Linux Untar:
downloaded 4.9 kernel image(size was 89MB) in /rootOFvol/dir2/ ===>the untar folder size was 695MB===>untar took about 25m  23sec as below
real	25m23.259s
user	0m14.129s
sys	0m23.510s


ls -lRt of the untarred folder took in total to complete as below 
real	0m50.975s
user	0m0.701s
sys	0m1.292s

Comment 5 nchilaka 2016-12-21 11:27:08 UTC
3.2 numbers:
Setup info
Ec volume build (3.8.4-9)
rhel 7.3 
6 VMs each of 8GB

Client:
16GB RHEL7.3
 
Volume Name: disperse
Type: Distributed-Disperse
Volume ID: ef4f768e-4b10-4c81-8053-adafaa1183db
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.35.37:/rhs/brick2/disperse
Brick2: 10.70.35.116:/rhs/brick2/disperse
Brick3: 10.70.35.239:/rhs/brick2/disperse
Brick4: 10.70.35.135:/rhs/brick2/disperse
Brick5: 10.70.35.8:/rhs/brick2/disperse
Brick6: 10.70.35.196:/rhs/brick2/disperse
Brick7: 10.70.35.37:/rhs/brick3/disperse
Brick8: 10.70.35.116:/rhs/brick3/disperse
Brick9: 10.70.35.239:/rhs/brick3/disperse
Brick10: 10.70.35.135:/rhs/brick3/disperse
Brick11: 10.70.35.8:/rhs/brick3/disperse
Brick12: 10.70.35.196:/rhs/brick3/disperse
Options Reconfigured:
features.uss: enable
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

####### FINDINGS###########################
Fuse mount:
(all bricks are up)
touch of 10k files in /rootOFvol/dir1/ ===>took 3min 09Sec
post above step immediate ls -lRt on root ===>took less than 1sec  to start displaying and o/p was completed in 1sec
post above step immediate find * on root ===>took less than 1sec to start displaying and o/p was completed in 1sec
post above step immediate stat * on /rootOfvol/dir1/===>took about 1 sec to respond and 14 sec for completing the total o/p
post above step rm -rf on root ===>took about 1min 15.720s
Linux Untar:
downloaded 4.9 kernel image(size was 89MB) in /rootOFvol/dir2/ ===>the untar folder size was 695MB===>untar took about 43m 15 sec as below
Real  43m15.552s
user    0m16.014s
sys     0m35.983s

ls -lRt of the untarred folder took in total to complete as below
real    0m59.373s

Comment 7 nchilaka 2016-12-21 13:15:47 UTC
For actual numbers you can refer to 
https://docs.google.com/spreadsheets/d/1T0pqXuL8mnIMMATwNGVWVTuJLSwziQvwi7LGr0LSgnk/edit#gid=0

Also attaching 3.2 and 3.1.3 numbers seperately

Comment 8 nchilaka 2016-12-21 13:16:12 UTC
Created attachment 1234395 [details]
3.2 numbers

Comment 9 nchilaka 2016-12-21 13:16:44 UTC
Created attachment 1234396 [details]
3.1.3 numbers

Comment 10 nchilaka 2016-12-21 13:29:04 UTC
I have rerun the measurements without enabling quota/uss and have been updated in 
https://docs.google.com/spreadsheets/d/1T0pqXuL8mnIMMATwNGVWVTuJLSwziQvwi7LGr0LSgnk/edit#gid=0
I still see the degradation

Comment 11 Ambarish 2016-12-26 09:32:09 UTC
**Perf Data on physical machines** :

*Setup/Environment details*: 


Testbed : 12*(4+2),6 servers,6 workload generating clients.

Benchmark : 3.1.3 with io-threads enabled.

3.2 testing was done with io-threads enabled and mdcache parameters set


***********
OBSERVATION
***********

**FUSE** :

1. Creates :

3.1.3 : **Perf Data on physical machines** :

*Setup/Environment details*: 


Testbed : 12*(4+2),6 servers,6 workload generating clients.

Benchmark : 3.1.3 with io-threads enabled.

3.2 testing was done with io-threads enabled and mdcache parameters set


***********
OBSERVATION
***********

---------
Creates 
---------

3.1.3 : 3445 files/sec
3.2   : 1841 files/sec 

Regression -  -46%

--------
Renames 
--------

3.1.3 : 724 files/sec
3.2   : 592 files/sec

Regression : -18%

mkdir regression on FUSE and gNFS is tracked via https://bugzilla.redhat.com/show_bug.cgi?id=1408655

Comment 14 Atin Mukherjee 2016-12-27 12:10:30 UTC
Upstream patch : http://review.gluster.org/#/c/16298/

Comment 15 Atin Mukherjee 2017-03-03 06:48:11 UTC
A new upstream patch https://review.gluster.org/#/c/16821/ is posted with a different alternative.

Comment 18 Ambarish 2017-03-09 15:42:22 UTC
Small file workloads are well within the regression threshold my runs are allowed to have.

Verified on 3.8.4-18.

Comment 20 errata-xmlrpc 2017-03-23 05:58:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.