Hide Forgot
Description of problem: Create a directory tree with maximum allowed depth and it takes too much time. The difference between time taken to create same depth directory tree in glusterfs-3.2.5 and glusterfs-3.3.0qa22 is huge. Version-Release number of selected component (if applicable): glusterfs-3.3.0qa22 How reproducible: always Steps to Reproduce: 1. Create a 2*2 distributed-replicated volume. 2. On the mountpoint run time mkdir -p "a`perl -e \"print '/a' x 2045\"`" 3. Make note of the time. 4. Run time rm -rf a on the mounpoint and make a note of the time again. Repeat above steps in glusterfs3.2.5 and glusterfs-3.3.0qa22 Actual results: With glusterfs-3.2.5 time taken to mkdir --- 0m36.91s time taken to rm -rf --- 0m50.27s With glusterfs-3.3.0qa22 time taken to mkdir --- 22m40.359s time taken to rm -rf --- 36m31.486s Expected results: The time taken should ideally be same or reasonably more. Additional info: On the master for following are the time taken to mkdir for directory depth of 500 --- 0m20.508s for directory depth of 1000 --- 1m19.908s for directory depth of 750 --- 0m47.938s rm -rf of 750 dirs took 1m11.013s
profile info output after rm -rf and mkdir Brick: 10.1.11.113:/data/bricks/hosdu_brick1 Cumulative Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6044 RELEASEDIR 0.00 169.00 us 169.00 us 169.00 us 1 GETXATTR 0.00 400.00 us 400.00 us 400.00 us 1 STATFS 0.01 35.32 us 12.00 us 163.00 us 8070 READDIR 0.03 46.66 us 12.00 us 200.00 us 12090 ENTRYLK 10.23 98002.68 us 276.00 us 216000.00 us 2015 RMDIR 20.98 100517.35 us 433.00 us 225682.00 us 4030 MKDIR 27.49 65835.76 us 87.00 us 146678.00 us 8060 OPENDIR 41.26 98567.92 us 94.00 us 219327.00 us 8080 LOOKUP Duration: 55241 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: 10.1.11.113:/data/bricks/hosdu_brick1 Interval 1 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 4025 RELEASEDIR 0.01 34.55 us 12.00 us 111.00 us 4030 READDIR 0.03 48.41 us 17.00 us 200.00 us 8060 ENTRYLK 14.13 98002.68 us 276.00 us 216000.00 us 2015 RMDIR 14.63 101515.36 us 460.00 us 225682.00 us 2015 MKDIR 28.43 68710.47 us 100.00 us 146678.00 us 5784 OPENDIR 42.77 103194.26 us 110.00 us 219327.00 us 5793 LOOKUP Duration: 41315 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: 10.1.11.114:/data/bricks/hosdu_brick2 Cumulative Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6044 RELEASEDIR 0.00 165.67 us 117.00 us 218.00 us 3 GETXATTR 0.00 2552.00 us 2552.00 us 2552.00 us 1 STATFS 0.01 42.74 us 11.00 us 195.00 us 8070 READDIR 0.24 550.09 us 14.00 us 120512.00 us 12090 ENTRYLK 7.01 48037.32 us 15.00 us 208357.00 us 4040 READDIRP 7.15 98167.82 us 202.00 us 242430.00 us 2015 RMDIR 14.90 102341.48 us 517.00 us 1479537.00 us 4030 MKDIR 18.95 65085.34 us 66.00 us 143752.00 us 8060 OPENDIR 23.15 63670.41 us 64.00 us 139177.00 us 10065 STAT 28.58 97938.53 us 80.00 us 215019.00 us 8078 LOOKUP Duration: 55241 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: 10.1.11.114:/data/bricks/hosdu_brick2 Interval 1 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 4025 RELEASEDIR 0.01 41.52 us 11.00 us 182.00 us 4030 READDIR 0.19 494.80 us 17.00 us 120512.00 us 8060 ENTRYLK 9.12 54412.69 us 15.00 us 208357.00 us 3518 READDIRP 9.43 98167.82 us 202.00 us 242430.00 us 2015 RMDIR 10.05 104699.27 us 538.00 us 1479537.00 us 2015 MKDIR 18.67 67743.40 us 92.00 us 143752.00 us 5784 OPENDIR 24.35 65549.98 us 72.00 us 139177.00 us 7795 STAT 28.19 102136.64 us 107.00 us 215019.00 us 5792 LOOKUP Duration: 41315 seconds Data Read: 0 bytes Data Written: 0 bytes
CHANGE: http://review.gluster.com/3204 (storage/posix: prefer absolute path handles over GFID handles) merged in master by Vijay Bellur (vijay)
With gluserfs-3.3.0qa37, creation of 2030 level depth directory takes around 30 secs and rm -rf of the same takes around 12 mins. For directory depth of 1900 find returned withing 13 odd minutes. Moving the bug to closed upstream.