Bug 1293237

Summary:	[Tier]: "Bad file descriptor" on removal of symlink only on tiered volume
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	tier	Assignee:	Bug Updates Notification Mailing List <rhs-bugs>
Status:	CLOSED ERRATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	dlambrig, nchilaka, rcyriac, rhs-bugs, rkavunga, rmekala, sankarshan, storage-qa-internal
Target Milestone:	---	Keywords:	Regression, ZStream
Target Release:	RHGS 3.1.2
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.5-13	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1293256 (view as bug list)		Environment:
Last Closed:	2016-03-01 06:04:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1293903, 1293963, 1295360
Bug Blocks:	1295347

Description Rahul Hinduja 2015-12-21 07:49:00 UTC

Description of problem:
=======================

rm on symbolic link errors with "Bad file descriptor" from client only on tiered volume. 

[root@dj r1]# touch a
[root@dj r1]# ln -s a b
[root@dj r1]# ls
a  b
[root@dj r1]# rm -rf b
rm: cannot remove ‘b’: Bad file descriptor
[root@dj r1]# ls
a
[root@dj r1]# 


Brick Log Snippet:
==================

[2015-12-20 19:36:21.565019] E [MSGID: 113005] [posix.c:1700:posix_unlink] 0-vol0-posix: post operation fstat failed on fd=-1 [Bad file descriptor]
[2015-12-20 19:36:21.565125] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-vol0-server: 229668: UNLINK /r1/b (1ad39c8b-623b-455a-9914-6139453f4b31/b) ==> (Bad file descriptor) [Bad file descriptor]


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-12.el7rhgs.x86_64


How reproducible:
=================

Always

Steps to Reproduce:
===================
1. Create Tier volume
2. Mount the volume on client
3. Create a file
4. Create symlink to the file
5. Remove symlink from client

Actual results:
===============

rm: cannot remove ‘etc.3/xdg/systemd/user’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K50netconsole’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K03rhnsd’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K90network’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc5.d/S10network’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc5.d/K50netconsole’: Bad file descriptor

[root@mia ~]# file /etc/xdg/systemd/user
/etc/xdg/systemd/user: symbolic link to `/etc/systemd/user'
[root@mia ~]# file /etc/rc.d/rc0.d/K50netconsole
/etc/rc.d/rc0.d/K50netconsole: symbolic link to `../init.d/netconsole'
[root@mia ~]# file /etc/rc.d/rc0.d/K03rhnsd
/etc/rc.d/rc0.d/K03rhnsd: symbolic link to `../init.d/rhnsd'
[root@mia ~]# ls -l /etc/rc.d/rc0.d/K03rhnsd
lrwxrwxrwx. 1 root root 15 Oct 14 07:17 /etc/rc.d/rc0.d/K03rhnsd -> ../init.d/rhnsd
[root@mia ~]# file /etc/rc.d/rc0.d/K90network
/etc/rc.d/rc0.d/K90network: symbolic link to `../init.d/network'
[root@mia ~]# file /etc/rc.d/rc5.d/S10network
/etc/rc.d/rc5.d/S10network: symbolic link to `../init.d/network'
[root@mia ~]# file /etc/rc.d/rc5.d/K50netconsole
/etc/rc.d/rc5.d/K50netconsole: symbolic link to `../init.d/netconsole'
[root@mia ~]#


Expected results:
=================

Symbolic link should also be removed without errors


Additional info:
================

Removal of symlink file on non-tiered volume is successful

Comment 3 Mohammed Rafi KC 2015-12-21 09:12:14 UTC

upstream pach : http://review.gluster.org/13034

Comment 4 Nag Pavan Chilakam 2015-12-21 09:46:43 UTC

Note: the directory fails to get deleted due to this issue, if there is symlinks in the directory
Workaround: retry delete of the directory once all files are deleted.

Comment 11 Rahul Hinduja 2016-01-06 11:59:40 UTC

Verified with build: glusterfs-3.7.5-14.el7rhgs.x86_64

Removal of symbolic and hardlink from tiervolume is successful without any errors. Moving this bug to verified state

[root@dj ~]# mkdir /mnt/fuse
[root@dj ~]# mount -t glusterfs 10.70.37.165:/tiervolume /mnt/fuse/
[root@dj ~]# cd /mnt/fuse/
[root@dj fuse]# 
[root@dj fuse]# touch a
[root@dj fuse]# ls
a
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
[root@dj fuse]# ln -s a b
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
lrwxrwxrwx. 1 root root 1 Jan  6  2016 b -> a
[root@dj fuse]# rm -rf b
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
[root@dj fuse]# ln -s a c
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
lrwxrwxrwx. 1 root root 1 Jan  6  2016 c -> a
[root@dj fuse]# rm c
rm: remove symbolic link ‘c’? y
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
[root@dj fuse]# 
[root@dj fuse]# 
[root@dj fuse]# for i in {1..3}; do cp -rf /etc etc.$i ; done 
[root@dj fuse]# 
[root@dj fuse]# ll
total 336
-rw-r--r--.  1 root root     0 Jan  6 04:36 a
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.1
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.2
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.3
[root@dj fuse]# ln -s a d
[root@dj fuse]# ln a e
[root@dj fuse]# ll
total 336
-rw-r--r--.  2 root root     0 Jan  6 04:36 a
lrwxrwxrwx.  1 root root     1 Jan  6  2016 d -> a
-rw-r--r--.  2 root root     0 Jan  6 04:36 e
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.1
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.2
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.3
[root@dj fuse]# 
[root@dj fuse]# rm -rf *
[root@dj fuse]# ll
total 0
[root@dj fuse]# ls
[root@dj fuse]# 
[root@dj fuse]# touch {1..9}
[root@dj fuse]# for i in {1..9}; do ln -s $i sym.$i ; done
[root@dj fuse]# for i in {1..9}; do ln  $i hard.$i ; done
[root@dj fuse]# ls
1  2  3  4  5  6  7  8  9  hard.1  hard.2  hard.3  hard.4  hard.5  hard.6  hard.7  hard.8  hard.9  sym.1  sym.2  sym.3  sym.4  sym.5  sym.6  sym.7  sym.8  sym.9
[root@dj fuse]# ll
total 0
-rw-r--r--. 2 root root 0 Jan  6 04:44 1
-rw-r--r--. 2 root root 0 Jan  6 04:44 2
-rw-r--r--. 2 root root 0 Jan  6 04:44 3
-rw-r--r--. 2 root root 0 Jan  6 04:44 4
-rw-r--r--. 2 root root 0 Jan  6 04:44 5
-rw-r--r--. 2 root root 0 Jan  6 04:44 6
-rw-r--r--. 2 root root 0 Jan  6 04:44 7
-rw-r--r--. 2 root root 0 Jan  6 04:44 8
-rw-r--r--. 2 root root 0 Jan  6 04:44 9
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.1
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.2
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.3
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.4
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.5
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.6
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.7
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.8
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.9
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.1 -> 1
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.2 -> 2
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.3 -> 3
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.4 -> 4
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.5 -> 5
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.6 -> 6
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.7 -> 7
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.8 -> 8
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.9 -> 9
[root@dj fuse]# rm -rf *
[root@dj fuse]# ll
total 0
[root@dj fuse]# ls
[root@dj fuse]# mount | grep tiervolume 
10.70.37.165:/tiervolume on /mnt/fuse type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@dj fuse]#

Comment 12 Niels de Vos 2016-01-20 09:19:53 UTC

*** Bug 1293798 has been marked as a duplicate of this bug. ***

Comment 14 errata-xmlrpc 2016-03-01 06:04:44 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html