Bug 1289437 - [Tier]: rm -rf * from client during demotion causes a stale link file to remain in system with attributes as ?????
[Tier]: rm -rf * from client during demotion causes a stale link file to rema...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
x86_64 Linux
unspecified Severity medium
: ---
: RHGS 3.1.2
Assigned To: Mohammed Rafi KC
Rahul Hinduja
: ZStream
Depends On: 1276227
Blocks: 1260783
  Show dependency treegraph
 
Reported: 2015-12-08 01:27 EST by Rahul Hinduja
Modified: 2016-09-17 11:36 EDT (History)
5 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-12
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-01 01:02:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rahul Hinduja 2015-12-08 01:27:26 EST
Description of problem:
=======================

Tried to remove all the files from client using "rm -rf *" while demotions were in-progress. rmdir failed complaing "Directory not empty". The file that is present from the client has all the attributes ???? and at the backend it is the link file in the cold tier:

From client:
============

[root@dj level75]# ls -l 
ls: cannot access 5664bb94%%VV2VS9M5R9: No such file or directory
total 0
?????????? ? ? ? ?            ? 5664bb94%%VV2VS9M5R9
[root@dj level75]# 
[root@dj level75]# pwd
/mnt/master/thread4/level05/level15/level25/level35/level45/level55/level65/level75
[root@dj level75]# 


From Cold tier bricks:
======================

[root@dhcp37-110 level75]# ls -l
total 0
---------T. 2 root root 0 Dec  7 05:17 5664bb94%%VV2VS9M5R9
[root@dhcp37-110 level75]# pwd
/rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75
[root@dhcp37-110 level75]# 


Errors at the Client logs:
===========================

mnt-master.log-20151207:[2015-12-07 17:32:55.063393] I [MSGID: 109070] [dht-common.c:1840:dht_lookup_linkfile_cbk] 2-master-tier-dht: Lookup of /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 on master-hot-dht (following linkfile) failed ,gfid = 00000000-0000-0000-0000-000000000000 [No such file or directory]
mnt-master.log-20151207:[2015-12-07 17:32:55.069227] I [MSGID: 109069] [dht-common.c:1159:dht_lookup_unlink_stale_linkto_cbk] 2-master-tier-dht: Returned with op_ret -1 and op_errno 16 for /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9
[root@dj glusterfs]# 


Errors at the Bricks logs:
=========================

rhs-brick1-ct-b5.log:[2015-12-07 18:15:06.611511] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-master-server: 76189: UNLINK /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 (691747be-864d-41b0-90f0-0932362702b0/5664bb94%%VV2VS9M5R9) ==> (Device or resource busy) [Device or resource busy]
rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789653] I [MSGID: 113030] [posix.c:1552:posix_unlink] 0-master-posix: open-fd-key-status: 0 for /rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9
rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789717] I [MSGID: 113031] [posix.c:1583:posix_unlink] 0-master-posix: linkto_xattr status: 1 for /rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9
rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789784] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-master-server: 76208: UNLINK /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 (691747be-864d-41b0-90f0-0932362702b0/5664bb94%%VV2VS9M5R9) ==> (Device or resource busy) [Device or resource busy]
[root@dhcp37-110 bricks]# 

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-9.el7rhgs.x86_64


How reproducible:
=================

1/1


Steps to Reproduce:
===================
1. Create and start Tier volume both hot and cold tier as DR
2. Mount the volume
3. Set the tier mode to test
4. Start creating lots of data
5. Wait for promotion/demotion to start
6. While demotions are happening, perform rm -rf from client

Actual results:
===============

Link file remain in the cold tier without the data file being in hot tier


Expected results:
=================

Link file or stale link file should also get deleted


Additional info:
Comment 5 Mohammed Rafi KC 2015-12-10 05:16:26 EST
upstream patch : http://review.gluster.org/#/c/12829/
Comment 7 Rahul Hinduja 2015-12-21 04:03:38 EST
Verified with build: glusterfs-3.7.5-12

While promotion and demotiones were inprogress, performed rm -rf * from client having 6k files at 5 level deep directory.

All files got removed from client and backend without any stale file being present in the system. Moving this bug to verified state. 

From Client:
============
[root@dj fuse]# ls
t1  test  thread0  thread1  thread2  thread3  thread4
[root@dj fuse]# 
[root@dj fuse]# 
[root@dj fuse]# rm -rf *
[root@dj fuse]#
[root@dj fuse]# ls -lRt
.:
total 0
[root@dj fuse]#   

From Backend:
=============

Cold Tier bricks:
=================

[root@dhcp37-165 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-165 tiervolume_ct-disp1]# 


[root@dhcp37-133 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-133 tiervolume_ct-disp1]# 


[root@dhcp37-160 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-160 tiervolume_ct-disp1]# 


[root@dhcp37-158 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-158 tiervolume_ct-disp1]# 


[root@dhcp37-110 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-110 tiervolume_ct-disp1]# 


[root@dhcp37-155 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-155 tiervolume_ct-disp1]# 


Hot Tier Bricks:
================

[root@dhcp37-99 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-99 tiervolume_hot]# 


[root@dhcp37-88 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-88 tiervolume_hot]# 


[root@dhcp37-112 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-112 tiervolume_hot]# 


[root@dhcp37-199 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-199 tiervolume_hot]# 


[root@dhcp37-162 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-162 tiervolume_hot]# 


[root@dhcp37-87 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-87 tiervolume_hot]#
Comment 10 errata-xmlrpc 2016-03-01 01:02:11 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.