Description of problem: ls on a mount point on which files were created takes 5x slower than ls on the same volume from a fresh mount point. git bisect points to commit c9bde3021202f1d5c5a2d19ac05a510fc1f788ac as the culprit. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create a single brick volume and fuse mount it on 2 different mount points (/mnt/1 and /mnt/2) 2.From /mnt/1: for i in {1..20000}; do echo "let's scale" > xfile-$i; done 3.Run the following from both mounts time ls -l |wc Actual results: On /mnt/1: [root@rhgs313-7 g1]# time ls -l |wc 20001 180002 1028906 real 0m7.663s user 0m0.163s sys 0m0.326s On /mnt/2: [root@rhgs313-7 g2]# time ls -l|wc 20001 180002 1028906 real 0m1.351s user 0m0.147s sys 0m0.183s Expected results: ls times should be comparable Additional info: Subsequent ls runs on /mnt/1 do not show improved performance until the client is remounted.
readdir-ahead stores that writes have happened to make sure to handle in-progress prefetch and writes concurrent with that prefetching. But the problem is that information is never cleared or stats updated with new values and hence causing this regression. concurrent writes need more intelligent handling than the current logic.
REVIEW: https://review.gluster.org/21811 (performance/readdir-ahead: update stats from prefetched dentries) posted (#1) for review on master by Raghavendra G
with fix [1]: ============= [root@rgowdapp mnt]# rm -rf * [root@rgowdapp mnt]# for i in {1..4000}; do echo "test" > $i ; done [root@rgowdapp mnt]# time ls -l > /dev/null real 0m0.162s user 0m0.012s sys 0m0.033s [root@rgowdapp mnt]# cd without fix [1]: ================ [root@rgowdapp ~]# umount /mnt/ [root@rgowdapp ~]# mount -t glusterfs localhost:ptop /mnt [root@rgowdapp ~]# cd /mnt [root@rgowdapp mnt]# rm -rf * [root@rgowdapp mnt]# for i in {1..4000}; do echo "test" > $i ; done [root@rgowdapp mnt]# time ls -l > /dev/null real 0m0.789s user 0m0.019s sys 0m0.062s [1] https://review.gluster.org/21811
REVIEW: https://review.gluster.org/21811 (performance/readdir-ahead: update stats from prefetched dentries) posted (#4) for review on master by Raghavendra G
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-6.0, please open a new bug report. glusterfs-6.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-March/000120.html [2] https://www.gluster.org/pipermail/gluster-users/