Bug 1248306

Summary:	tiering: rename fails with "Device or resource busy" error message
Product:	[Community] GlusterFS	Reporter:	Mohammed Rafi KC <rkavunga>
Component:	tiering	Assignee:	Mohammed Rafi KC <rkavunga>
Status:	CLOSED CURRENTRELEASE	QA Contact:	bugs <bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	mainline	CC:	bugs
Target Milestone:	---	Keywords:	Reopened, Triaged
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	glusterfs-3.8rc2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1254437 (view as bug list)		Environment:
Last Closed:	2016-06-16 13:27:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1254437, 1260923

Description Mohammed Rafi KC 2015-07-30 04:52:17 UTC

Description of problem:

On a tiered volume after a file promoted/demoted, then unlink on that file will fail with EBUSY.

Version-Release number of selected component (if applicable):

mainline.

How reproducible:

100%

Steps to Reproduce:
1.create a distributed-replicated volume.
2.attach a distributed-replicated volume.
3.mount the volume and create some files on the mount point
4.let the files demote.
5.Rename any any of the demoted files.

Actual results:

rename fails

Expected results:

rename should succeed.

Additional info:

Comment 1 Mohammed Rafi KC 2015-07-30 05:04:04 UTC

RCA:

A newly created files will always hashed to hot-tier for a tiered volume. When the files was in hot tier, then hashed and cached subvolume will be hot-tier. Once the file is migrated from hot-tier to cold-tier, then subsequent lookup will send a revalidate lookup to hot-tier and it will find out that the file was
actually moved and there is only a link file in the cached subvolume. So dht
will return an ESTALE to fuse. Upon receiving ESTALE for a lookup, fuse
will create a new inode and sent a fresh lookup. This lookup will be
successful, and it will locate the file properly. Then fuse try to link
the inode, but the older inode was already there in inmemory inode cache
with same gfid and that is also shared with fuse kernal. So inode_link
will return the older ionode itself. So the subsequent rename fop will
come to gluster with the older inode. From dht_rename, we will take a
lock on the inode and after successful inodelk on inode dht will send
lookup before creating a link. this lookup will again find out that the
file is a link file, and then dht will think that file is
migrating/migrated during the course, and will send a EBUSY error.

Comment 2 Anand Avati 2015-08-05 13:53:53 UTC

REVIEW: http://review.gluster.org/11768 (dht/tier :unlink fails with EBUSY) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

Comment 3 Anand Avati 2015-08-06 06:48:36 UTC

REVIEW: http://review.gluster.org/11768 (dht/tier :rename fails with EBUSY) posted (#3) for review on master by mohammed rafi  kc (rkavunga)

Comment 4 Anand Avati 2015-08-13 14:32:28 UTC

COMMIT: http://review.gluster.org/11768 committed in master by Dan Lambright (dlambrig) 
------
commit 0ad26041fbf65ab36856a0ad178c32e51bf87319
Author: Mohammed Rafi KC <rkavunga>
Date:   Wed Aug 5 19:20:04 2015 +0530

    dht/tier :rename fails with EBUSY
    
    When the files was in hot tier and the look up was done already, then
    hashed and cached subvolume will be hot-tier. Once the file is moved
    from hot-tier to cold-tier, then subsequent lookup will send a
    revalidate lookup to hot-tier and it will find out that the file was
    actually moved and there is only link in the cached subvolume. So dht
    will return an ESTALE to fuse. Upon receiving ESTALE for a lookup, fuse
    will create a new inode and sent a fresh lookup. This lookup will be
    successful, and it will locate the file properly. Then fuse try to link
    the inode, but the older inode was already there in inmemory inode cache
    with same gfid and that is also shared with fuse kernal. So inode_link
    will return the older ionode itself. So the subsequent rename fop will
    come to gluster with the older inode. From dht_rename, we will take a
    lock on the inode and after successful inodelk on inode dht will send
    lookup before creating a link. this lookup will again find out that the
    file is a link file, and then dht will think that file is
    migrating/migrated in the mean time, and will send EBUSY.
    
    Change-Id: Ib3a01e5b1d7f64514b04bb6234026d049f082679
    BUG: 1248306
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/11768
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Dan Lambright <dlambrig>

Comment 5 Nagaprasad Sathyanarayana 2015-10-25 15:09:15 UTC

Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 6 Niels de Vos 2016-06-16 13:27:16 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user