Created attachment 567534 [details] performance comparison Description of problem: As we can see from the numbers below, create operations have taken a hit & have been 4 times slower (minimum). run132 - glusterfs - 3.3.0qa24 - distrep - (quota off, gsync off) run140 - glusterfs - 3.3.0qa25 - distrep - (quota off, gsync off) Operations RUN132 RUN140 ------------------------- ------- ------- emptyfiles_create 399 2102 emptyfiles_delete 286 270 smallfiles_create 740 6962 smallfiles_rewrite 681 6686 smallfiles_read 457 459 smallfiles_reread 447 444 smallfiles_delete 318 311 largefile_create 21 21 largefile_rewrite 21 21 largefile_read 9 9 largefile_reread 9 9 largefile_delete 0 0 directory_crawl_create 389 1695 directory_crawl 142 140 directory_recrawl 135 131 metadata_modify 400 388 directory_crawl_delete 310 287 I have attached the other numbers with quota & geo-rep enabled. They are slow as well.
Can you please disable md-cache and schedule a performance run?
Also want a separate test again with 3.3.0qa25 with 'volume profile' turned on, and I want 'profile info' after the test.
I ran the performance runs with md-cache off, & still saw similar results & thought of not running the whole suite since there was no improvement. So, I am running the same with profile info. I will update the results once it is over. I am running only emptyfiles create & delete operations.
Created attachment 568548 [details] Profile output I have attached the output of profiling during emptyfiles create & delete operations were happening. qa25 emptyfiles create & delete time: emptyfiles_create 3992 emptyfiles_delete 265
Some more analysis with qa31. performance analysis. 1000 emptyfiles: Fuse mount - disrep volume->xfs backend with inodesize 512 Operations (xlator xceptions) - Time in secs ----------------------------- ------------ Create/delete (normal) - 41/2.6 Create/delete (md-cache off) - 40/2.5 [all other perf xlators enabled] Create/delete (quick-read off)- 32/2.3 [all other perf xlators enabled] Create/delete (io-cache off) - 21/2.5 [all other perf xlators enabled] Create/delete (read-ahead off)- 25/2.4 [all other perf xlators enabled] Create/delete (write-behind off)- 4/2.3[all other perf xlators enabled] Create/delete (flush-behind off)- 4/2.3[all other perf xlators enabled] After flush-behind is reset back to on(default), the performance improves & create/delete times are 18s/2.5s Create/delete (all perf-xlators xcept write-behind off)- 13/2.2 Create/delete (all perf-xlators off)- 4/2.3 Nfs mount - disrep Create/delete (normal) - 2.8/1.9
Patch accepted as part of bug 795635
Verified with 3.3.0qa36