Description of problem: On a tiered volume after a file promoted/demoted, then unlink on that file will fail with EBUSY. Version-Release number of selected component (if applicable): mainline. How reproducible: 100% Steps to Reproduce: 1.create a distributed-replicated volume. 2.attach a distributed-replicated volume. 3.mount the volume and create some files on the mount point 4.let the files demote. 5.Rename any any of the demoted files. Actual results: rename fails Expected results: rename should succeed. Additional info:
RCA: A newly created files will always hashed to hot-tier for a tiered volume. When the files was in hot tier, then hashed and cached subvolume will be hot-tier. Once the file is migrated from hot-tier to cold-tier, then subsequent lookup will send a revalidate lookup to hot-tier and it will find out that the file was actually moved and there is only a link file in the cached subvolume. So dht will return an ESTALE to fuse. Upon receiving ESTALE for a lookup, fuse will create a new inode and sent a fresh lookup. This lookup will be successful, and it will locate the file properly. Then fuse try to link the inode, but the older inode was already there in inmemory inode cache with same gfid and that is also shared with fuse kernal. So inode_link will return the older ionode itself. So the subsequent rename fop will come to gluster with the older inode. From dht_rename, we will take a lock on the inode and after successful inodelk on inode dht will send lookup before creating a link. this lookup will again find out that the file is a link file, and then dht will think that file is migrating/migrated during the course, and will send a EBUSY error.
REVIEW: http://review.gluster.org/11768 (dht/tier :unlink fails with EBUSY) posted (#2) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/11768 (dht/tier :rename fails with EBUSY) posted (#3) for review on master by mohammed rafi kc (rkavunga)
COMMIT: http://review.gluster.org/11768 committed in master by Dan Lambright (dlambrig) ------ commit 0ad26041fbf65ab36856a0ad178c32e51bf87319 Author: Mohammed Rafi KC <rkavunga> Date: Wed Aug 5 19:20:04 2015 +0530 dht/tier :rename fails with EBUSY When the files was in hot tier and the look up was done already, then hashed and cached subvolume will be hot-tier. Once the file is moved from hot-tier to cold-tier, then subsequent lookup will send a revalidate lookup to hot-tier and it will find out that the file was actually moved and there is only link in the cached subvolume. So dht will return an ESTALE to fuse. Upon receiving ESTALE for a lookup, fuse will create a new inode and sent a fresh lookup. This lookup will be successful, and it will locate the file properly. Then fuse try to link the inode, but the older inode was already there in inmemory inode cache with same gfid and that is also shared with fuse kernal. So inode_link will return the older ionode itself. So the subsequent rename fop will come to gluster with the older inode. From dht_rename, we will take a lock on the inode and after successful inodelk on inode dht will send lookup before creating a link. this lookup will again find out that the file is a link file, and then dht will think that file is migrating/migrated in the mean time, and will send EBUSY. Change-Id: Ib3a01e5b1d7f64514b04bb6234026d049f082679 BUG: 1248306 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/11768 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Reviewed-by: Dan Lambright <dlambrig> Tested-by: NetBSD Build System <jenkins.org> Tested-by: Dan Lambright <dlambrig>
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user