Description of problem: Any application using glfs handles (like NFS-Ganesha) may need to do lookup on each dirent returned by readdir/readdirp operation to create handles. But since the lookup in glusterfs is very costly operation (as need to be sent to all replica bricks), as the directory gets larger or the replica count increases, the time taken for single readdir operation to complete can take hours of time. To avoid that we need an extended readdirp API which can return handles along with dirent stat as part of its reply. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
REVIEW: https://review.gluster.org/15663 (WIP/gfapi/handleops: Introducing READDIRPLUS_EXT() fop for handleops) posted (#6) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_h_xreaddirplus_r() fop for handleops) posted (#7) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_h_xreaddirplus_r() fop for handleops) posted (#8) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_h_xreaddirplus_r() fop for handleops) posted (#9) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_h_xreaddirplus_r() fop for handleops) posted (#10) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_h_xreaddirplus_r() fop for handleops) posted (#11) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_h_xreaddirplus_r() fop for handleops) posted (#12) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops) posted (#13) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops) posted (#14) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops) posted (#15) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/15663 (gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops) posted (#16) for review on master by soumya k (skoduri)
COMMIT: https://review.gluster.org/15663 committed in master by Jeff Darcy (jeff.us) ------ commit 41000cd0b57a81c4ace4a1d3da0fcc352a11f146 Author: Soumya Koduri <skoduri> Date: Fri Apr 21 16:30:20 2017 +0530 gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops Its known that readdirplus operation fetches stat as well for each of the dirents. But often applications may need extra information, like for eg., NFS-Ganesha which operates on handles needs handles for each of those dirents returned. So this would require extra calls to the backend, in this case LOOKUP (which is very expensive operation) resulting in very low readdir performance. To address that introducing this new API using which applications can make request for any extra information to be returned as part of readdirplus response. Currently this new api returns stat and handles as demanded by application. The synopsis of the API is noted in glfs.h. @todo: * Enhance test script using this new API Below were the perf results on single brick volume with and without these changes - Dataset used - 10*100 directories and each directory containing 100 empty files. I used NFS-Ganesha application to test these changes - >for i in {1..5}; do systemctl restart nfs-ganesha; sleep 10; mount -t nfs -o vers=4 localhost:/brick_vol /mnt; cd /mnt; echo "ITERATION$i"; date; find . > tmp-nfs.log; date; cd /; umount /mnt; sleep 2; done; Without these changes - ITERATION1 Mon Mar 20 17:22:26 IST 2017 Mon Mar 20 17:23:18 IST 2017 ITERATION2 Mon Mar 20 17:23:39 IST 2017 Mon Mar 20 17:24:28 IST 2017 ITERATION3 Mon Mar 20 17:24:49 IST 2017 Mon Mar 20 17:25:36 IST 2017 ITERATION4 Mon Mar 20 17:30:57 IST 2017 Mon Mar 20 17:31:37 IST 2017 ITERATION5 Mon Mar 20 17:31:57 IST 2017 Mon Mar 20 17:32:40 IST 2017 [root@dhcp35-197 /]# On an average ~46.2 sec With these changes applied - ITERATION1 Mon Mar 20 17:35:03 IST 2017 Mon Mar 20 17:35:15 IST 2017 ITERATION2 Mon Mar 20 17:35:36 IST 2017 Mon Mar 20 17:35:46 IST 2017 ITERATION3 Mon Mar 20 17:36:06 IST 2017 Mon Mar 20 17:36:17 IST 2017 ITERATION4 Mon Mar 20 17:41:38 IST 2017 Mon Mar 20 17:41:49 IST 2017 ITERATION5 Mon Mar 20 17:42:10 IST 2017 Mon Mar 20 17:42:20 IST 2017 On an average ~10.8 sec Updates #174 BUG: 1442950 Change-Id: I0f74f74dc62085ca4c4a23c38e3edc84bd850876 Signed-off-by: Soumya Koduri <skoduri> Reviewed-on: https://review.gluster.org/15663 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Niels de Vos <ndevos> CentOS-regression: Gluster Build System <jenkins.org>
REVIEW: https://review.gluster.org/17329 (gfapi: Enhance testscript for glfs_xreaddirplus) posted (#1) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/17329 (gfapi: Enhance testscript for glfs_xreaddirplus) posted (#2) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/17329 (gfapi: Enhance testscript for glfs_xreaddirplus) posted (#3) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/17329 (gfapi: Enhance testscript for glfs_xreaddirplus) posted (#4) for review on master by soumya k (skoduri)
REVIEW: https://review.gluster.org/17329 (gfapi: Enhance testscript for glfs_xreaddirplus) posted (#5) for review on master by soumya k (skoduri)
COMMIT: https://review.gluster.org/17329 committed in master by Niels de Vos (ndevos) ------ commit b27e60a0cbdb42755cd7d4ee3c7e62b57d5881ba Author: Soumya Koduri <skoduri> Date: Thu May 18 14:53:47 2017 +0530 gfapi: Enhance testscript for glfs_xreaddirplus Enhance the script testing glfs_xreaddirplus functionality and also measure the performance difference when compared to using the older method. Change-Id: I590d07c850994afab0a02eb5dccb8342224aa6b7 BUG: 1442950 Signed-off-by: Soumya Koduri <skoduri> Reviewed-on: https://review.gluster.org/17329 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Amar Tumballi <amarts> Reviewed-by: Niels de Vos <ndevos>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.0, please open a new bug report. glusterfs-3.12.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-September/000082.html [2] https://www.gluster.org/pipermail/gluster-users/