Bug 1254437 - tiering: rename fails with "Device or resource busy" error message
tiering: rename fails with "Device or resource busy" error message
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: tiering (Show other bugs)
3.7.3
All All
high Severity high
: ---
: ---
Assigned To: Mohammed Rafi KC
bugs@gluster.org
: Triaged
Depends On: 1248306
Blocks: 1260923
  Show dependency treegraph
 
Reported: 2015-08-18 03:14 EDT by Mohammed Rafi KC
Modified: 2015-10-30 13:32 EDT (History)
2 users (show)

See Also:
Fixed In Version: glusterfs-3.7.4
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1248306
Environment:
Last Closed: 2015-09-09 05:39:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mohammed Rafi KC 2015-08-18 03:14:12 EDT
+++ This bug was initially created as a clone of Bug #1248306 +++

Description of problem:

On a tiered volume after a file promoted/demoted, then unlink on that file will fail with EBUSY.

Version-Release number of selected component (if applicable):

mainline.

How reproducible:

100%

Steps to Reproduce:
1.create a distributed-replicated volume.
2.attach a distributed-replicated volume.
3.mount the volume and create some files on the mount point
4.let the files demote.
5.Rename any any of the demoted files.

Actual results:

rename fails

Expected results:

rename should succeed.

Additional info:

--- Additional comment from Mohammed Rafi KC on 2015-07-30 01:04:04 EDT ---

RCA:

A newly created files will always hashed to hot-tier for a tiered volume. When the files was in hot tier, then hashed and cached subvolume will be hot-tier. Once the file is migrated from hot-tier to cold-tier, then subsequent lookup will send a revalidate lookup to hot-tier and it will find out that the file was
actually moved and there is only a link file in the cached subvolume. So dht
will return an ESTALE to fuse. Upon receiving ESTALE for a lookup, fuse
will create a new inode and sent a fresh lookup. This lookup will be
successful, and it will locate the file properly. Then fuse try to link
the inode, but the older inode was already there in inmemory inode cache
with same gfid and that is also shared with fuse kernal. So inode_link
will return the older ionode itself. So the subsequent rename fop will
come to gluster with the older inode. From dht_rename, we will take a
lock on the inode and after successful inodelk on inode dht will send
lookup before creating a link. this lookup will again find out that the
file is a link file, and then dht will think that file is
migrating/migrated during the course, and will send a EBUSY error.

--- Additional comment from Anand Avati on 2015-08-05 09:53:53 EDT ---

REVIEW: http://review.gluster.org/11768 (dht/tier :unlink fails with EBUSY) posted (#2) for review on master by mohammed rafi  kc (rkavunga@redhat.com)

--- Additional comment from Anand Avati on 2015-08-06 02:48:36 EDT ---

REVIEW: http://review.gluster.org/11768 (dht/tier :rename fails with EBUSY) posted (#3) for review on master by mohammed rafi  kc (rkavunga@redhat.com)

--- Additional comment from Anand Avati on 2015-08-13 10:32:28 EDT ---

COMMIT: http://review.gluster.org/11768 committed in master by Dan Lambright (dlambrig@redhat.com) 
------
commit 0ad26041fbf65ab36856a0ad178c32e51bf87319
Author: Mohammed Rafi KC <rkavunga@redhat.com>
Date:   Wed Aug 5 19:20:04 2015 +0530

    dht/tier :rename fails with EBUSY
    
    When the files was in hot tier and the look up was done already, then
    hashed and cached subvolume will be hot-tier. Once the file is moved
    from hot-tier to cold-tier, then subsequent lookup will send a
    revalidate lookup to hot-tier and it will find out that the file was
    actually moved and there is only link in the cached subvolume. So dht
    will return an ESTALE to fuse. Upon receiving ESTALE for a lookup, fuse
    will create a new inode and sent a fresh lookup. This lookup will be
    successful, and it will locate the file properly. Then fuse try to link
    the inode, but the older inode was already there in inmemory inode cache
    with same gfid and that is also shared with fuse kernal. So inode_link
    will return the older ionode itself. So the subsequent rename fop will
    come to gluster with the older inode. From dht_rename, we will take a
    lock on the inode and after successful inodelk on inode dht will send
    lookup before creating a link. this lookup will again find out that the
    file is a link file, and then dht will think that file is
    migrating/migrated in the mean time, and will send EBUSY.
    
    Change-Id: Ib3a01e5b1d7f64514b04bb6234026d049f082679
    BUG: 1248306
    Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com>
    Reviewed-on: http://review.gluster.org/11768
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    Reviewed-by: Dan Lambright <dlambrig@redhat.com>
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Tested-by: Dan Lambright <dlambrig@redhat.com>
Comment 1 Anand Avati 2015-08-19 04:31:32 EDT
REVIEW: http://review.gluster.org/11944 (dht/tier :rename fails with EBUSY) posted (#2) for review on release-3.7 by mohammed rafi  kc (rkavunga@redhat.com)
Comment 2 Anand Avati 2015-08-19 09:26:19 EDT
REVIEW: http://review.gluster.org/11944 (dht/tier :rename fails with EBUSY) posted (#3) for review on release-3.7 by mohammed rafi  kc (rkavunga@redhat.com)
Comment 3 Kaushal 2015-09-09 05:39:52 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.4, please open a new bug report.

glusterfs-3.7.4 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/12496
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.