Bug 791187

Summary:	[glusterfs-3.3.0qa22] - Creation of deep directory tree takes huge amount of time.
Product:	[Community] GlusterFS	Reporter:	M S Vishwanath Bhat <vbhat>
Component:	core	Assignee:	Anand Avati <aavati>
Status:	CLOSED UPSTREAM	QA Contact:
Severity:	medium	Docs Contact:
Priority:	urgent
Version:	pre-release	CC:	chrisw, gluster-bugs, mzywusko, vbellur, vinaraya
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-04-23 08:03:11 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description M S Vishwanath Bhat 2012-02-16 12:34:13 UTC

Description of problem:
Create a directory tree with maximum allowed depth and it takes too much time. The difference between time taken to create same depth directory tree in glusterfs-3.2.5 and glusterfs-3.3.0qa22 is huge. 

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa22

How reproducible:
always

Steps to Reproduce:
1. Create a 2*2 distributed-replicated volume.
2. On the mountpoint run time mkdir -p "a`perl -e \"print '/a' x 2045\"`"  
3. Make note of the time. 
4. Run time rm -rf a on the mounpoint and make a note of the time again.

Repeat above steps in glusterfs3.2.5 and glusterfs-3.3.0qa22
  
Actual results:

With glusterfs-3.2.5
time taken to mkdir  ---  0m36.91s
time taken to rm -rf --- 0m50.27s

With glusterfs-3.3.0qa22
time taken to mkdir  --- 22m40.359s
time taken to rm -rf --- 36m31.486s

Expected results:
The time taken should ideally be same or reasonably more.

Additional info:

On the master for following are the time taken to mkdir
for directory depth of 500  --- 0m20.508s
for directory depth of 1000 --- 1m19.908s
for directory depth of 750  --- 0m47.938s

rm -rf of 750 dirs took 1m11.013s

Comment 1 M S Vishwanath Bhat 2012-02-17 11:40:02 UTC

profile info output after rm -rf and mkdir


Brick: 10.1.11.113:/data/bricks/hosdu_brick1
Cumulative Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6044  RELEASEDIR
      0.00     169.00 us     169.00 us     169.00 us              1    GETXATTR
      0.00     400.00 us     400.00 us     400.00 us              1      STATFS
      0.01      35.32 us      12.00 us     163.00 us           8070     READDIR
      0.03      46.66 us      12.00 us     200.00 us          12090     ENTRYLK
     10.23   98002.68 us     276.00 us  216000.00 us           2015       RMDIR
     20.98  100517.35 us     433.00 us  225682.00 us           4030       MKDIR
     27.49   65835.76 us      87.00 us  146678.00 us           8060     OPENDIR
     41.26   98567.92 us      94.00 us  219327.00 us           8080      LOOKUP
 
    Duration: 55241 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: 10.1.11.113:/data/bricks/hosdu_brick1
Interval 1 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           4025  RELEASEDIR
      0.01      34.55 us      12.00 us     111.00 us           4030     READDIR
      0.03      48.41 us      17.00 us     200.00 us           8060     ENTRYLK
     14.13   98002.68 us     276.00 us  216000.00 us           2015       RMDIR
     14.63  101515.36 us     460.00 us  225682.00 us           2015       MKDIR
     28.43   68710.47 us     100.00 us  146678.00 us           5784     OPENDIR
     42.77  103194.26 us     110.00 us  219327.00 us           5793      LOOKUP
 
    Duration: 41315 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: 10.1.11.114:/data/bricks/hosdu_brick2
Cumulative Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6044  RELEASEDIR
      0.00     165.67 us     117.00 us     218.00 us              3    GETXATTR
      0.00    2552.00 us    2552.00 us    2552.00 us              1      STATFS
      0.01      42.74 us      11.00 us     195.00 us           8070     READDIR
      0.24     550.09 us      14.00 us  120512.00 us          12090     ENTRYLK
      7.01   48037.32 us      15.00 us  208357.00 us           4040    READDIRP
      7.15   98167.82 us     202.00 us  242430.00 us           2015       RMDIR
     14.90  102341.48 us     517.00 us 1479537.00 us           4030       MKDIR
     18.95   65085.34 us      66.00 us  143752.00 us           8060     OPENDIR
     23.15   63670.41 us      64.00 us  139177.00 us          10065        STAT
     28.58   97938.53 us      80.00 us  215019.00 us           8078      LOOKUP
 
    Duration: 55241 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: 10.1.11.114:/data/bricks/hosdu_brick2
Interval 1 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           4025  RELEASEDIR
      0.01      41.52 us      11.00 us     182.00 us           4030     READDIR
      0.19     494.80 us      17.00 us  120512.00 us           8060     ENTRYLK
      9.12   54412.69 us      15.00 us  208357.00 us           3518    READDIRP
      9.43   98167.82 us     202.00 us  242430.00 us           2015       RMDIR
     10.05  104699.27 us     538.00 us 1479537.00 us           2015       MKDIR
     18.67   67743.40 us      92.00 us  143752.00 us           5784     OPENDIR
     24.35   65549.98 us      72.00 us  139177.00 us           7795        STAT
     28.19  102136.64 us     107.00 us  215019.00 us           5792      LOOKUP
 
    Duration: 41315 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Comment 2 Anand Avati 2012-04-21 08:26:31 UTC

CHANGE: http://review.gluster.com/3204 (storage/posix: prefer absolute path handles over GFID handles) merged in master by Vijay Bellur (vijay)

Comment 3 M S Vishwanath Bhat 2012-04-23 08:03:11 UTC

With gluserfs-3.3.0qa37, creation of 2030 level depth directory takes around 30 secs and rm -rf of the same takes around 12 mins. For directory depth of 1900 find returned withing 13 odd minutes.

Moving the bug to closed upstream.