Bug 1288509 - rm -rf is taking very long time
rm -rf is taking very long time
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity unspecified
: ---
: RHGS 3.1.2
Assigned To: Bug Updates Notification Mailing List
RajeshReddy
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-04 08:00 EST by RajeshReddy
Modified: 2016-09-17 11:34 EDT (History)
11 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-15
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-01 01:00:54 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description RajeshReddy 2015-12-04 08:00:21 EST
Description of problem:
===============
rm -rf is taking very long time 

Version-Release number of selected component (if applicable):
================
glusterfs-server-3.7.5-9


How reproducible:


Steps to Reproduce:
==============
1. Create 2x2 volume and then mount it on client using FUSE and create directory and then create 50k files
2. Attach 2x2 hot bricks to the volume and then create new directory and create around 20k files 
3. Kill all the brick process and restart the volume using force option 
4. While files are getting demoted run rm -rf * but it took more than 2 hours time
Actual results:


Expected results:
==========
Should not take this much time 


Additional info:
=============
[root@rhs-client19 test_tier-tier-dht]# gluster vol info test_tier 
 
Volume Name: test_tier
Type: Tier
Volume ID: 9bca8ffb-d47c-4636-95ab-2cfc58da422e
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick5/test_tier_hot4
Brick2: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick5/test_tier_hot4
Brick3: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick4/test_tier_hot3
Brick4: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick4/test_tier_hot3
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick7/test_tier_hot1
Brick6: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick7/test_tier_hot1
Brick7: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick6/test_tier_hot2
Brick8: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick6/test_tier_hot2
Options Reconfigured:
cluster.tier-mode: test
features.ctr-enabled: on
performance.readdir-ahead: on

Client name:vertigo.lab.eng.blr.redhat.com
Mount:/mnt/test_tier
Comment 2 RajeshReddy 2015-12-04 08:04:26 EST
sosreport available  @/home/repo/sosreports/bug.1288509 on rhsqe-repo.lab.eng.blr.redhat.com
Comment 3 RajeshReddy 2015-12-08 05:53:10 EST
Able to reproduce the issue without Step no 3 ( Kill all the brick process and restart the volume using force option)
Comment 6 RajeshReddy 2015-12-11 07:19:44 EST
I was not running rename operations
Comment 7 Joseph Elwin Fernandes 2015-12-19 23:22:18 EST
http://review.gluster.org/12972
Comment 8 Joseph Elwin Fernandes 2015-12-22 10:32:58 EST
https://code.engineering.redhat.com/gerrit/#/c/64372/1

Options provided in for good performance

gluster vol set features.ctr-sql-db-wal-autocheckpoint 25000
gluster vol set features.ctr-sql-db-cachesize 12500

gluster vol set help for details
Comment 10 RajeshReddy 2015-12-23 08:18:10 EST
To delete 50k files took more than two hour
Comment 11 Dan Lambright 2015-12-23 11:29:25 EST
Can you turn off ctr and rerun the test?

gluster set volume <vol?> features.ctr-enabled off
Comment 13 RajeshReddy 2015-12-24 02:28:01 EST
Tested with build glusterfs-server-3.7.5-13 and after setting features.ctr-sql-db-cachesize: 12500 and features.ctr-sql-db-wal-autocheckpoint: 25000 and removal of 50k files took 6m and clearly giving good performance 

As i need to repeat the same tests with build which having above settings as default values so please move this bug to ON_QA once build having proper default values
Comment 14 Joseph Elwin Fernandes 2015-12-24 09:38:57 EST
Sure. Thanks :)
Comment 15 Joseph Elwin Fernandes 2015-12-24 09:45:24 EST
Sure
Comment 16 Joseph Elwin Fernandes 2015-12-30 22:59:26 EST
https://code.engineering.redhat.com/gerrit/64642
Comment 18 RajeshReddy 2016-01-06 07:45:42 EST
Tested with 3.7.5-14 build and after creation of new tiered volume verified the features.ctr-sql-db-wal-autocheckpoint  and features.ctr-sql-db-cachesize default values and those values are not modified so marking this bug as failed QA 


[root@tettnang afr1x2_tier_bug]# rpm -qa | grep glusterfs 
glusterfs-3.7.5-14.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-14.el7rhgs.x86_64
glusterfs-fuse-3.7.5-14.el7rhgs.x86_64
glusterfs-debuginfo-3.7.5-14.el7rhgs.x86_64
glusterfs-api-3.7.5-14.el7rhgs.x86_64
glusterfs-rdma-3.7.5-14.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-14.el7rhgs.x86_64
glusterfs-server-3.7.5-14.el7rhgs.x86_64
glusterfs-ganesha-3.7.5-14.el7rhgs.x86_64
glusterfs-cli-3.7.5-14.el7rhgs.x86_64
glusterfs-libs-3.7.5-14.el7rhgs.x86_64
glusterfs-api-devel-3.7.5-14.el7rhgs.x86_64
glusterfs-devel-3.7.5-14.el7rhgs.x86_64
[root@tettnang afr1x2_tier_bug]# gluster vol get perform_create all | grep cachesize
features.ctr-sql-db-cachesize           1000                                    
[root@tettnang afr1x2_tier_bug]# gluster vol get perform_create all | grep ctr-sql-db-wal-autocheckpoint
features.ctr-sql-db-wal-autocheckpoint  1000                                    
[root@tettnang afr1x2_tier_bug]#
Comment 19 Joseph Elwin Fernandes 2016-01-06 12:57:40 EST
https://code.engineering.redhat.com/gerrit/64971
Comment 21 RajeshReddy 2016-01-08 06:58:24 EST
Deletion of 50K files took 2m and by default  features.ctr-sql-db-cachesize:  and features.ctr-sql-db-wal-autocheckpoint set to 12500 & 25000 respectively so marking this bug as verified
Comment 23 errata-xmlrpc 2016-03-01 01:00:54 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.