Bug 799854
| Summary: | [glusterfs-3.3.0qa25]: performance - Create operations very slow in comparison with qa24 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Rahul C S <rahulcs> | ||||||
| Component: | core | Assignee: | Vijay Bellur <vbellur> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | pre-release | CC: | ashetty, gluster-bugs, rfortier, vbellur | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | glusterfs-3.4.0 | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2013-07-24 17:29:52 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 817967 | ||||||||
| Attachments: |
|
||||||||
Can you please disable md-cache and schedule a performance run? Also want a separate test again with 3.3.0qa25 with 'volume profile' turned on, and I want 'profile info' after the test. I ran the performance runs with md-cache off, & still saw similar results & thought of not running the whole suite since there was no improvement. So, I am running the same with profile info. I will update the results once it is over. I am running only emptyfiles create & delete operations. Created attachment 568548 [details]
Profile output
I have attached the output of profiling during emptyfiles create & delete operations were happening.
qa25 emptyfiles create & delete time:
emptyfiles_create 3992
emptyfiles_delete 265
Some more analysis with qa31. performance analysis. 1000 emptyfiles: Fuse mount - disrep volume->xfs backend with inodesize 512 Operations (xlator xceptions) - Time in secs ----------------------------- ------------ Create/delete (normal) - 41/2.6 Create/delete (md-cache off) - 40/2.5 [all other perf xlators enabled] Create/delete (quick-read off)- 32/2.3 [all other perf xlators enabled] Create/delete (io-cache off) - 21/2.5 [all other perf xlators enabled] Create/delete (read-ahead off)- 25/2.4 [all other perf xlators enabled] Create/delete (write-behind off)- 4/2.3[all other perf xlators enabled] Create/delete (flush-behind off)- 4/2.3[all other perf xlators enabled] After flush-behind is reset back to on(default), the performance improves & create/delete times are 18s/2.5s Create/delete (all perf-xlators xcept write-behind off)- 13/2.2 Create/delete (all perf-xlators off)- 4/2.3 Nfs mount - disrep Create/delete (normal) - 2.8/1.9 Patch accepted as part of bug 795635 Verified with 3.3.0qa36 |
Created attachment 567534 [details] performance comparison Description of problem: As we can see from the numbers below, create operations have taken a hit & have been 4 times slower (minimum). run132 - glusterfs - 3.3.0qa24 - distrep - (quota off, gsync off) run140 - glusterfs - 3.3.0qa25 - distrep - (quota off, gsync off) Operations RUN132 RUN140 ------------------------- ------- ------- emptyfiles_create 399 2102 emptyfiles_delete 286 270 smallfiles_create 740 6962 smallfiles_rewrite 681 6686 smallfiles_read 457 459 smallfiles_reread 447 444 smallfiles_delete 318 311 largefile_create 21 21 largefile_rewrite 21 21 largefile_read 9 9 largefile_reread 9 9 largefile_delete 0 0 directory_crawl_create 389 1695 directory_crawl 142 140 directory_recrawl 135 131 metadata_modify 400 388 directory_crawl_delete 310 287 I have attached the other numbers with quota & geo-rep enabled. They are slow as well.