Bug 1293256

Summary: [Tier]: "Bad file descriptor" on removal of symlink only on tiered volume
Product: [Community] GlusterFS Reporter: Mohammed Rafi KC <rkavunga>
Component: tieringAssignee: Mohammed Rafi KC <rkavunga>
Status: CLOSED CURRENTRELEASE QA Contact: bugs <bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: mainlineCC: bugs, nchilaka, rhinduja, rhs-bugs
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.8rc2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1293237
: 1295347 (view as bug list) Environment:
Last Closed: 2016-06-16 13:51:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1295347    

Comment 1 Mohammed Rafi KC 2015-12-21 09:10:35 UTC 
Description of problem:
=======================

rm on symbolic link errors with "Bad file descriptor" from client only on tiered volume. 

[root@dj r1]# touch a
[root@dj r1]# ln -s a b
[root@dj r1]# ls
a  b
[root@dj r1]# rm -rf b
rm: cannot remove ‘b’: Bad file descriptor
[root@dj r1]# ls
a
[root@dj r1]# 


Brick Log Snippet:
==================

[2015-12-20 19:36:21.565019] E [MSGID: 113005] [posix.c:1700:posix_unlink] 0-vol0-posix: post operation fstat failed on fd=-1 [Bad file descriptor]
[2015-12-20 19:36:21.565125] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-vol0-server: 229668: UNLINK /r1/b (1ad39c8b-623b-455a-9914-6139453f4b31/b) ==> (Bad file descriptor) [Bad file descriptor]


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-12.el7rhgs.x86_64


How reproducible:
=================

Always

Steps to Reproduce:
===================
1. Create Tier volume
2. Mount the volume on client
3. Create a file
4. Create symlink to the file
5. Remove symlink from client

Actual results:
===============

rm: cannot remove ‘etc.3/xdg/systemd/user’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K50netconsole’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K03rhnsd’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K90network’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc5.d/S10network’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc5.d/K50netconsole’: Bad file descriptor

[root@mia ~]# file /etc/xdg/systemd/user
/etc/xdg/systemd/user: symbolic link to `/etc/systemd/user'
[root@mia ~]# file /etc/rc.d/rc0.d/K50netconsole
/etc/rc.d/rc0.d/K50netconsole: symbolic link to `../init.d/netconsole'
[root@mia ~]# file /etc/rc.d/rc0.d/K03rhnsd
/etc/rc.d/rc0.d/K03rhnsd: symbolic link to `../init.d/rhnsd'
[root@mia ~]# ls -l /etc/rc.d/rc0.d/K03rhnsd
lrwxrwxrwx. 1 root root 15 Oct 14 07:17 /etc/rc.d/rc0.d/K03rhnsd -> ../init.d/rhnsd
[root@mia ~]# file /etc/rc.d/rc0.d/K90network
/etc/rc.d/rc0.d/K90network: symbolic link to `../init.d/network'
[root@mia ~]# file /etc/rc.d/rc5.d/S10network
/etc/rc.d/rc5.d/S10network: symbolic link to `../init.d/network'
[root@mia ~]# file /etc/rc.d/rc5.d/K50netconsole
/etc/rc.d/rc5.d/K50netconsole: symbolic link to `../init.d/netconsole'
[root@mia ~]#


Expected results:
=================

Symbolic link should also be removed without errors


Additional info:
================

Removal of symlink file on non-tiered volume is successful
Comment 2 Vijay Bellur 2015-12-21 09:11:33 UTC 
REVIEW: http://review.gluster.org/13034 (tier/unlink: open fd for special file for fdstat) posted (#1) for review on master by mohammed rafi  kc (rkavunga)
Comment 3 Vijay Bellur 2015-12-21 17:16:39 UTC 
COMMIT: http://review.gluster.org/13034 committed in master by Dan Lambright (dlambrig) 
------
commit 7237f8f0d72743d2522b7db1be928c5016713c59
Author: Mohammed Rafi KC <rkavunga>
Date:   Mon Dec 21 14:18:26 2015 +0530

    tier/unlink: open fd for special file for fdstat
    
    DUring unlink of a file, dht request stat to see whether
    the file is under migration or not. But in posix_unlink
    currently we are opening for regular files. so the fdstat
    for special files are failing with EBAD
    
    Change-Id: Ic0678e42e7701c3dffb91d98272e664b0fc646b5
    BUG: 1293256
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/13034
    Reviewed-by: Raghavendra G <rgowdapp>
    Reviewed-by: Susant Palai <spalai>
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>
Comment 4 Vijay Bellur 2016-01-04 09:10:13 UTC 
REVIEW: http://review.gluster.org/13160 (tier/unlink: open fd for special file for fdstat) posted (#1) for review on release-3.7 by mohammed rafi  kc (rkavunga)
Comment 5 Niels de Vos 2016-06-16 13:51:54 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user