This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1281347 - Though directories are deleted from mount, still able to see them on cold bricks
Though directories are deleted from mount, still able to see them on cold bricks
Status: CLOSED WORKSFORME
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Nithya Balachandran
nchilaka
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-12 06:02 EST by RajeshReddy
Modified: 2016-09-17 11:38 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-14 23:16:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description RajeshReddy 2015-11-12 06:02:26 EST
Description of problem:
================
Though directories are deleted from mount, still able to see them on cold bricks 


Version-Release number of selected component (if applicable):
===================
glusterfs-server-3.7.5-5

How reproducible:


Steps to Reproduce:
==============
1. Create distributed replica volume and attach 4 hot bricks 
2. Mount it on client using FUSE, create ten directories data1 to data10 and create 1k files inside each directory 
3. while files are migrating from hot to cold tried deletion of directories using rm -rf data* but few directories are not getting deleted saying directory is not empty after some time again deleted those directories from mount 

Actual results:
=============
Able to see few directories on the cold tier 


Expected results:
=========
directories should be deleted from cold tier too when those are deleted from mount 


Additional info:
================
[root@rhs-client19 ~]# gluster vol info disrep_tier 
 
Volume Name: disrep_tier
Type: Tier
Volume ID: ea4bd2c2-efd3-4d25-bbc1-8f6d9c75dafc
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick5/tier
Brick2: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick5/tier
Brick3: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick6/tier
Brick4: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick6/tier
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick7/disrep_teri
Brick6: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick7/disrep_teri
Brick7: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick6/disrep_teri
Brick8: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick6/disrep_teri
Options Reconfigured:
features.ctr-enabled: on
performance.readdir-ahead: on


Cold tier 
====
[root@rhs-client18 disrep_teri]# ls -lrth
total 184K
drwxr-xr-x. 2 root root  12K Nov 10 15:19 data4  
drwxr-xr-x. 2 root root  12K Nov 10 15:19 data5
drwxr-xr-x. 2 root root 8.0K Nov 10 15:20 data6
drwxr-xr-x. 2 root root  12K Nov 10 15:28 data7
drwxr-xr-x. 2 root root  12K Nov 10 16:40 data8
drwxr-xr-x. 2 root root  12K Nov 10 16:40 data9
drwxr-xr-x. 2 root root  28K Nov 12 11:29 bug
drwxr-xr-x. 2 root root   33 Nov 12 13:24 big
drwxr-xr-x. 2 root root    6 Nov 12 15:26 test
drwxr-xr-x. 2 root root    6 Nov 12 15:34 down

mount (client)
====
[root@dht-rhs-23 disrep_tier]# pwd
/mnt/disrep_tier
[root@dht-rhs-23 disrep_tier]# ls -lrth
total 2.0K
drwxr-xr-x. 2 root root 58 Nov 12 11:52 bug
drwxr-xr-x. 2 root root  6 Nov 12 11:56 big
drwxr-xr-x. 2 root root  6 Nov 12 13:34 test
drwxr-xr-x. 2 root root  6 Nov 12 16:15 down
[root@dht-rhs-23 disrep_tier]# mount | grep /mnt/disrep_tier
rhs-client19.lab.eng.blr.redhat.com:disrep_tier on /mnt/disrep_tier type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@dht-rhs-23 disrep_tier]# 


Data4 to 9 are there on cold tier but not visible under client mount
Comment 2 RajeshReddy 2015-11-16 06:06:07 EST
After remounting, able to see those directories from the mount 

[root@dht-rhs-23 disrep_tier]# pwd
/mnt/disrep_tier
[root@dht-rhs-23 disrep_tier]# ls -lrth
total 5.5K
drwxr-xr-x. 2 root root  6 Nov 12 11:28 big
drwxr-xr-x. 2 root root 58 Nov 12 11:29 bug
drwxr-xr-x. 2 root root  6 Nov 12 12:22 test
drwxr-xr-x. 2 root root  6 Nov 12 15:50 down
drwxr-xr-x. 2 root root  6 Nov 16 15:45 data4
drwxr-xr-x. 2 root root  6 Nov 16 15:45 data5
drwxr-xr-x. 2 root root  6 Nov 16 15:45 data6
drwxr-xr-x. 2 root root  6 Nov 16 15:45 data7
drwxr-xr-x. 2 root root  6 Nov 16 15:45 data8
drwxr-xr-x. 2 root root  6 Nov 16 15:45 data9
drwxr-xr-x. 4 root root 57 Nov 16 15:50 linux
Comment 4 RajeshReddy 2015-11-26 09:39:57 EST
sosreports are available @ /home/repo/sosreports/bug.1284928 on rhsqe-repo.lab.eng.blr.redhat.com
Comment 5 RajeshReddy 2015-12-04 01:59:19 EST
Updated client sosreport available @ /home/repo/sosreports/bug.1284928 on rhsqe-repo.lab.eng.blr.redhat.com
Comment 6 Nithya Balachandran 2015-12-04 08:28:54 EST
The client logs do not have any errors related to dir deletes. However, there are lots of error messages indicating that some bricks were not reachable. Where any bricks down during these tests?

[2015-11-12 11:45:06.450045] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-disrep_tier-client-2: changing port to 49171 (from 0)
[2015-11-12 11:45:06.455800] E [socket.c:2278:socket_connect_finish] 0-disrep_tier-client-2: connection to 10.70.36.42:49171 failed (Connection refused)
[2015-11-12 11:45:09.460433] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-disrep_tier-client-3: changing port to 49170 (from 0)
[2015-11-12 11:45:09.471182] E [socket.c:2278:socket_connect_finish] 0-disrep_tier-client-3: connection to 10.70.36.43:49170 failed (Connection refused)



[2015-11-12 11:46:31.920475] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-disrep_tier-cold-dht: Found anomalies in /big (gfid = 82d2ddc2-d36c-4f7d-b308-39534e558371). Holes=1 overlaps=0
[2015-11-12 11:46:32.659562] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-disrep_tier-cold-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
[2015-11-12 11:46:32.659626] W [MSGID: 109005] [dht-selfheal.c:1795:dht_selfheal_directory] 0-disrep_tier-cold-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /, gfid = 00000000-0000-0000-0000-000000000001
The message "I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-disrep_tier-replicate-1: no subvolumes up" repeated 19 times between [2015-11-12 11:46:31.906727] and [2015-11-12 11:46:32.690197]
[2015-11-12 11:46:32.690997] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-disrep_tier-cold-dht: Found anomalies in /big (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2015-11-12 11:46:32.691073] W [MSGID: 109005] [dht-selfheal.c:1795:dht_selfheal_directory] 0-disrep_tier-cold-dht: Directory selfheal failed: 1 subvolumes down.Not fixing. path = /big, gfid = 82d2ddc2-d36c-4f7d-b308-39534e558371


[2015-11-12 12:14:14.410012] W [MSGID: 109075] [dht-diskusage.c:49:dht_du_info_cbk] 0-disrep_tier-cold-dht: failed to get disk info from disrep_tier-replicate-1 [Transport endpoint is not connected]



The 

[2015-11-12 12:14:15.799370] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-disrep_tier-cold-dht: no subvolume for hash (value) = 3553706931
[2015-11-12 12:14:15.799485] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-disrep_tier-cold-dht: no subvolume for hash (value) = 3781782680
[2015-11-12 12:14:15.799518] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-disrep_tier-cold-dht: no subvolume for hash (value) = 2593024302
[2015-11-12 12:14:15.799541] W [MSGID: 109011] [dht-layout.c:191:dht_layout_search] 0-disrep_tier-cold-dht: no subvolume for hash (value) = 4122822141
Comment 7 RajeshReddy 2015-12-04 08:44:02 EST
While performing delete operations no brick was down
Comment 9 RajeshReddy 2015-12-14 08:02:21 EST
Tested with glusterfs-server-3.7.5-11, created 10 directories with each directory having 1k files and deleted parent directory and after deletion not able to see any sub directories on both cold and hot tier

Note You need to log in before you can comment on or make changes to this bug.