Bug 1289437

Summary:	[Tier]: rm -rf * from client during demotion causes a stale link file to remain in system with attributes as ?????
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	tier	Assignee:	Mohammed Rafi KC <rkavunga>
Status:	CLOSED ERRATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	nchilaka, rhs-bugs, rkavunga, sankarshan, storage-qa-internal
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.1.2
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.5-12	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-01 06:02:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1276227
Bug Blocks:	1260783

Description Rahul Hinduja 2015-12-08 06:27:26 UTC

Description of problem:
=======================

Tried to remove all the files from client using "rm -rf *" while demotions were in-progress. rmdir failed complaing "Directory not empty". The file that is present from the client has all the attributes ???? and at the backend it is the link file in the cold tier:

From client:
============

[root@dj level75]# ls -l 
ls: cannot access 5664bb94%%VV2VS9M5R9: No such file or directory
total 0
?????????? ? ? ? ?            ? 5664bb94%%VV2VS9M5R9
[root@dj level75]# 
[root@dj level75]# pwd
/mnt/master/thread4/level05/level15/level25/level35/level45/level55/level65/level75
[root@dj level75]# 


From Cold tier bricks:
======================

[root@dhcp37-110 level75]# ls -l
total 0
---------T. 2 root root 0 Dec  7 05:17 5664bb94%%VV2VS9M5R9
[root@dhcp37-110 level75]# pwd
/rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75
[root@dhcp37-110 level75]# 


Errors at the Client logs:
===========================

mnt-master.log-20151207:[2015-12-07 17:32:55.063393] I [MSGID: 109070] [dht-common.c:1840:dht_lookup_linkfile_cbk] 2-master-tier-dht: Lookup of /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 on master-hot-dht (following linkfile) failed ,gfid = 00000000-0000-0000-0000-000000000000 [No such file or directory]
mnt-master.log-20151207:[2015-12-07 17:32:55.069227] I [MSGID: 109069] [dht-common.c:1159:dht_lookup_unlink_stale_linkto_cbk] 2-master-tier-dht: Returned with op_ret -1 and op_errno 16 for /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9
[root@dj glusterfs]# 


Errors at the Bricks logs:
=========================

rhs-brick1-ct-b5.log:[2015-12-07 18:15:06.611511] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-master-server: 76189: UNLINK /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 (691747be-864d-41b0-90f0-0932362702b0/5664bb94%%VV2VS9M5R9) ==> (Device or resource busy) [Device or resource busy]
rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789653] I [MSGID: 113030] [posix.c:1552:posix_unlink] 0-master-posix: open-fd-key-status: 0 for /rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9
rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789717] I [MSGID: 113031] [posix.c:1583:posix_unlink] 0-master-posix: linkto_xattr status: 1 for /rhs/brick1/ct-b5/thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9
rhs-brick1-ct-b5.log:[2015-12-07 18:15:11.789784] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-master-server: 76208: UNLINK /thread4/level05/level15/level25/level35/level45/level55/level65/level75/5664bb94%%VV2VS9M5R9 (691747be-864d-41b0-90f0-0932362702b0/5664bb94%%VV2VS9M5R9) ==> (Device or resource busy) [Device or resource busy]
[root@dhcp37-110 bricks]# 

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-9.el7rhgs.x86_64


How reproducible:
=================

1/1


Steps to Reproduce:
===================
1. Create and start Tier volume both hot and cold tier as DR
2. Mount the volume
3. Set the tier mode to test
4. Start creating lots of data
5. Wait for promotion/demotion to start
6. While demotions are happening, perform rm -rf from client

Actual results:
===============

Link file remain in the cold tier without the data file being in hot tier


Expected results:
=================

Link file or stale link file should also get deleted


Additional info:

Comment 5 Mohammed Rafi KC 2015-12-10 10:16:26 UTC

upstream patch : http://review.gluster.org/#/c/12829/

Comment 7 Rahul Hinduja 2015-12-21 09:03:38 UTC

Verified with build: glusterfs-3.7.5-12

While promotion and demotiones were inprogress, performed rm -rf * from client having 6k files at 5 level deep directory.

All files got removed from client and backend without any stale file being present in the system. Moving this bug to verified state. 

From Client:
============
[root@dj fuse]# ls
t1  test  thread0  thread1  thread2  thread3  thread4
[root@dj fuse]# 
[root@dj fuse]# 
[root@dj fuse]# rm -rf *
[root@dj fuse]#
[root@dj fuse]# ls -lRt
.:
total 0
[root@dj fuse]#   

From Backend:
=============

Cold Tier bricks:
=================

[root@dhcp37-165 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-165 tiervolume_ct-disp1]# 


[root@dhcp37-133 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-133 tiervolume_ct-disp1]# 


[root@dhcp37-160 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-160 tiervolume_ct-disp1]# 


[root@dhcp37-158 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-158 tiervolume_ct-disp1]# 


[root@dhcp37-110 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-110 tiervolume_ct-disp1]# 


[root@dhcp37-155 tiervolume_ct-disp1]# ls -lRt
.:
total 0
[root@dhcp37-155 tiervolume_ct-disp1]# 


Hot Tier Bricks:
================

[root@dhcp37-99 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-99 tiervolume_hot]# 


[root@dhcp37-88 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-88 tiervolume_hot]# 


[root@dhcp37-112 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-112 tiervolume_hot]# 


[root@dhcp37-199 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-199 tiervolume_hot]# 


[root@dhcp37-162 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-162 tiervolume_hot]# 


[root@dhcp37-87 tiervolume_hot]# ls -lRt
.:
total 0
[root@dhcp37-87 tiervolume_hot]#

Comment 10 errata-xmlrpc 2016-03-01 06:02:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html