Description of problem: With fixes [1][2] small-file workload [3] showed around 54% of regression in read performance: with patches [1][2]: 1856 without patches [1][2]: 3053.8 files/sec Note that with patches [1][2], even though open-behind is present in the stack, opens sent before read are actually sent to bricks as we turn off anonymous-fds and set read-after-open to true. Since every read is witnessed by open-behind (as its a parent of quick-read and reads are absorbed only by quick-read), open-behind does an actual open on bricks. Without patches [1][2] none of open, read, flush and release were sent to bricks. With patches [1][2] open, flush, release are sent to bricks (reads are absorbed by quick-read). This extra fops has caused the performance regression. One fix I can think of is to change the relative order of open-behind and quick-read (quick read as parent of open-behind), so that reads are not witnessed by open-behind. Preliminary test runs show the regression indeed is fixed by this solution (keeping read-after-open on and use-anonymous-fd no). I'll post the average numbers after tests complete [1] https://review.gluster.org/r/Ifa52d8ff017f115e83247f3396b9d27f0295ce3f [2] https://review.gluster.org/r/Iaf130db71ce61ac37269f422e348a45f6ae6e82c [3] https://github.com/distributed-system-analysis/smallfile Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
REVIEW: https://review.gluster.org/21857 (performance/ob: make open-behind as a child of quick-read) posted (#1) for review on master by Raghavendra G
Following are the runs from small-file workload no-fixes-default-config | fixes-ob-without-anon-fd-child-of-qr | fixes-ob-with-anon-fd-child-of-qr | | append sum=7076.66 count=5 avg=1415.33 |append sum=7135.91 count=5 avg=1427.18 |append sum=7052.17 count=5 avg=1410.43 chmod sum=9146.32 count=5 avg=1829.26 |chmod sum=9207.25 count=5 avg=1841.45 |chmod sum=9245.51 count=5 avg=1849.10 create sum=13032.1 count=5 avg=2606.42 |create sum=12853.2 count=5 avg=2570.64 |create sum=13307.8 count=5 avg=2661.56 delete-renamed sum=8483.5 count=5 avg=1696.70 |delete-renamed sum=8676.86 count=5 avg=1735.37 |delete-renamed sum=8634.49 count=5 avg=1726.89 ls-l sum=40362.8 count=5 avg=8072.56 |ls-l sum=38522 count=5 avg=7704.40 |ls-l sum=40140.9 count=5 avg=8028.18 mkdir sum=3419.6 count=5 avg=683.92 |mkdir sum=3639.08 count=5 avg=727.81 |mkdir sum=3708.42 count=5 avg=741.68 read sum=16703.9 count=5 avg=3340.78 |read sum=16340.8 count=5 avg=3268.16 |read sum=16074.7 count=5 avg=3214.94 rename sum=9828.65 count=5 avg=1965.73 |rename sum=10049.7 count=5 avg=2009.94 |rename sum=9983.86 count=5 avg=1996.77 rmdir sum=2341.03 count=5 avg=468.20 |rmdir sum=2419.4 count=5 avg=483.88 |rmdir sum=2424.71 count=5 avg=484.94 stat sum=16763.5 count=5 avg=3352.70 |stat sum=16677.8 count=5 avg=3335.56 |stat sum=16870.2 count=5 avg=3374.04 Above runs clearly show fixes with 1. qr as a parent of ob 2. use-anonymous-fd turned on in ob give better performance when considered all operations holistically. Just to complete the exercise I am going to schedule a run without any fixes and qr loaded as a parent of ob and update the results.
REVIEW: https://review.gluster.org/21857 (performance/ob: make open-behind as a child of quick-read) posted (#2) for review on master by Amar Tumballi
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-6.0, please open a new bug report. glusterfs-6.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-March/000120.html [2] https://www.gluster.org/pipermail/gluster-users/