Bug 799854 - [glusterfs-3.3.0qa25]: performance - Create operations very slow in comparison with qa24
[glusterfs-3.3.0qa25]: performance - Create operations very slow in compariso...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: core (Show other bugs)
pre-release
Unspecified Unspecified
high Severity urgent
: ---
: ---
Assigned To: Vijay Bellur
:
Depends On:
Blocks: 817967
  Show dependency treegraph
 
Reported: 2012-03-05 04:11 EST by Rahul C S
Modified: 2015-12-01 11:45 EST (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-24 13:29:52 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
performance comparison (2.22 KB, text/plain)
2012-03-05 04:11 EST, Rahul C S
no flags Details
Profile output (13.87 KB, application/octet-stream)
2012-03-08 04:25 EST, Rahul C S
no flags Details

  None (edit)
Description Rahul C S 2012-03-05 04:11:49 EST
Created attachment 567534 [details]
performance comparison

Description of problem:
As we can see from the numbers below, create operations have taken a hit & have been 4 times slower (minimum).

run132 - glusterfs - 3.3.0qa24 - distrep - (quota off, gsync off)
run140 - glusterfs - 3.3.0qa25 - distrep - (quota off, gsync off)

Operations                      RUN132  RUN140
-------------------------       ------- -------
emptyfiles_create               399     2102
emptyfiles_delete               286     270
smallfiles_create               740     6962
smallfiles_rewrite              681     6686
smallfiles_read                 457     459
smallfiles_reread               447     444
smallfiles_delete               318     311
largefile_create                21      21
largefile_rewrite               21      21
largefile_read                  9       9
largefile_reread                9       9
largefile_delete                0       0
directory_crawl_create          389     1695
directory_crawl                 142     140
directory_recrawl               135     131
metadata_modify                 400     388
directory_crawl_delete          310     287

I have attached the other numbers with quota & geo-rep enabled. They are slow as well.
Comment 1 Vijay Bellur 2012-03-05 12:34:20 EST
Can you please disable md-cache and schedule a performance run?
Comment 2 Amar Tumballi 2012-03-06 01:19:04 EST
Also want a separate test again with 3.3.0qa25 with 'volume profile' turned on, and I want 'profile info' after the test.
Comment 3 Rahul C S 2012-03-07 06:21:29 EST
I ran the performance runs with md-cache off, & still saw similar results & thought of not running the whole suite since there was no improvement. So, I am running the same with profile info. I will update the results once it is over. I am running only emptyfiles create & delete operations.
Comment 4 Rahul C S 2012-03-08 04:25:53 EST
Created attachment 568548 [details]
Profile output

I have attached the output of profiling during emptyfiles create & delete operations were happening. 

qa25 emptyfiles create & delete time:
emptyfiles_create               3992
emptyfiles_delete               265
Comment 5 Rahul C S 2012-03-21 07:43:34 EDT
Some more analysis with qa31.

performance analysis.

1000 emptyfiles:

Fuse mount - disrep volume->xfs backend with inodesize 512

Operations (xlator xceptions) - Time in secs
-----------------------------   ------------
Create/delete (normal)        - 41/2.6
Create/delete (md-cache off)  - 40/2.5 [all other perf xlators enabled]
Create/delete (quick-read off)- 32/2.3 [all other perf xlators enabled]
Create/delete (io-cache off)  - 21/2.5 [all other perf xlators enabled]
Create/delete (read-ahead off)- 25/2.4 [all other perf xlators enabled]
Create/delete (write-behind off)- 4/2.3[all other perf xlators enabled]
Create/delete (flush-behind off)- 4/2.3[all other perf xlators enabled]

After flush-behind is reset back to on(default), the
performance improves & create/delete times are 18s/2.5s

Create/delete (all perf-xlators xcept write-behind off)- 13/2.2
Create/delete (all perf-xlators off)- 4/2.3

Nfs mount - disrep
Create/delete (normal)        - 2.8/1.9
Comment 7 Vijay Bellur 2012-04-19 15:06:56 EDT
Patch accepted as part of bug 795635
Comment 8 Anush Shetty 2012-04-23 02:37:29 EDT
Verified with 3.3.0qa36

Note You need to log in before you can comment on or make changes to this bug.