Bug 1293237 - [Tier]: "Bad file descriptor" on removal of symlink only on tiered volume
[Tier]: "Bad file descriptor" on removal of symlink only on tiered volume
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
x86_64 Linux
unspecified Severity urgent
: ---
: RHGS 3.1.2
Assigned To: Bug Updates Notification Mailing List
Rahul Hinduja
: Regression, ZStream
: 1293798 (view as bug list)
Depends On: 1293903 1293963 1295360
Blocks: 1295347
  Show dependency treegraph
 
Reported: 2015-12-21 02:49 EST by Rahul Hinduja
Modified: 2016-09-17 11:40 EDT (History)
8 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-13
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1293256 (view as bug list)
Environment:
Last Closed: 2016-03-01 01:04:44 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rahul Hinduja 2015-12-21 02:49:00 EST
Description of problem:
=======================

rm on symbolic link errors with "Bad file descriptor" from client only on tiered volume. 

[root@dj r1]# touch a
[root@dj r1]# ln -s a b
[root@dj r1]# ls
a  b
[root@dj r1]# rm -rf b
rm: cannot remove ‘b’: Bad file descriptor
[root@dj r1]# ls
a
[root@dj r1]# 


Brick Log Snippet:
==================

[2015-12-20 19:36:21.565019] E [MSGID: 113005] [posix.c:1700:posix_unlink] 0-vol0-posix: post operation fstat failed on fd=-1 [Bad file descriptor]
[2015-12-20 19:36:21.565125] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-vol0-server: 229668: UNLINK /r1/b (1ad39c8b-623b-455a-9914-6139453f4b31/b) ==> (Bad file descriptor) [Bad file descriptor]


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-12.el7rhgs.x86_64


How reproducible:
=================

Always

Steps to Reproduce:
===================
1. Create Tier volume
2. Mount the volume on client
3. Create a file
4. Create symlink to the file
5. Remove symlink from client

Actual results:
===============

rm: cannot remove ‘etc.3/xdg/systemd/user’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K50netconsole’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K03rhnsd’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc0.d/K90network’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc5.d/S10network’: Bad file descriptor
rm: cannot remove ‘etc.3/rc.d/rc5.d/K50netconsole’: Bad file descriptor

[root@mia ~]# file /etc/xdg/systemd/user
/etc/xdg/systemd/user: symbolic link to `/etc/systemd/user'
[root@mia ~]# file /etc/rc.d/rc0.d/K50netconsole
/etc/rc.d/rc0.d/K50netconsole: symbolic link to `../init.d/netconsole'
[root@mia ~]# file /etc/rc.d/rc0.d/K03rhnsd
/etc/rc.d/rc0.d/K03rhnsd: symbolic link to `../init.d/rhnsd'
[root@mia ~]# ls -l /etc/rc.d/rc0.d/K03rhnsd
lrwxrwxrwx. 1 root root 15 Oct 14 07:17 /etc/rc.d/rc0.d/K03rhnsd -> ../init.d/rhnsd
[root@mia ~]# file /etc/rc.d/rc0.d/K90network
/etc/rc.d/rc0.d/K90network: symbolic link to `../init.d/network'
[root@mia ~]# file /etc/rc.d/rc5.d/S10network
/etc/rc.d/rc5.d/S10network: symbolic link to `../init.d/network'
[root@mia ~]# file /etc/rc.d/rc5.d/K50netconsole
/etc/rc.d/rc5.d/K50netconsole: symbolic link to `../init.d/netconsole'
[root@mia ~]#


Expected results:
=================

Symbolic link should also be removed without errors


Additional info:
================

Removal of symlink file on non-tiered volume is successful
Comment 3 Mohammed Rafi KC 2015-12-21 04:12:14 EST
upstream pach : http://review.gluster.org/13034
Comment 4 nchilaka 2015-12-21 04:46:43 EST
Note: the directory fails to get deleted due to this issue, if there is symlinks in the directory
Workaround: retry delete of the directory once all files are deleted.
Comment 11 Rahul Hinduja 2016-01-06 06:59:40 EST
Verified with build: glusterfs-3.7.5-14.el7rhgs.x86_64

Removal of symbolic and hardlink from tiervolume is successful without any errors. Moving this bug to verified state

[root@dj ~]# mkdir /mnt/fuse
[root@dj ~]# mount -t glusterfs 10.70.37.165:/tiervolume /mnt/fuse/
[root@dj ~]# cd /mnt/fuse/
[root@dj fuse]# 
[root@dj fuse]# touch a
[root@dj fuse]# ls
a
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
[root@dj fuse]# ln -s a b
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
lrwxrwxrwx. 1 root root 1 Jan  6  2016 b -> a
[root@dj fuse]# rm -rf b
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
[root@dj fuse]# ln -s a c
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
lrwxrwxrwx. 1 root root 1 Jan  6  2016 c -> a
[root@dj fuse]# rm c
rm: remove symbolic link ‘c’? y
[root@dj fuse]# ll
total 0
-rw-r--r--. 1 root root 0 Jan  6 04:36 a
[root@dj fuse]# 
[root@dj fuse]# 
[root@dj fuse]# for i in {1..3}; do cp -rf /etc etc.$i ; done 
[root@dj fuse]# 
[root@dj fuse]# ll
total 336
-rw-r--r--.  1 root root     0 Jan  6 04:36 a
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.1
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.2
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.3
[root@dj fuse]# ln -s a d
[root@dj fuse]# ln a e
[root@dj fuse]# ll
total 336
-rw-r--r--.  2 root root     0 Jan  6 04:36 a
lrwxrwxrwx.  1 root root     1 Jan  6  2016 d -> a
-rw-r--r--.  2 root root     0 Jan  6 04:36 e
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.1
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.2
drwxr-xr-x. 80 root root 65536 Jan  6  2016 etc.3
[root@dj fuse]# 
[root@dj fuse]# rm -rf *
[root@dj fuse]# ll
total 0
[root@dj fuse]# ls
[root@dj fuse]# 
[root@dj fuse]# touch {1..9}
[root@dj fuse]# for i in {1..9}; do ln -s $i sym.$i ; done
[root@dj fuse]# for i in {1..9}; do ln  $i hard.$i ; done
[root@dj fuse]# ls
1  2  3  4  5  6  7  8  9  hard.1  hard.2  hard.3  hard.4  hard.5  hard.6  hard.7  hard.8  hard.9  sym.1  sym.2  sym.3  sym.4  sym.5  sym.6  sym.7  sym.8  sym.9
[root@dj fuse]# ll
total 0
-rw-r--r--. 2 root root 0 Jan  6 04:44 1
-rw-r--r--. 2 root root 0 Jan  6 04:44 2
-rw-r--r--. 2 root root 0 Jan  6 04:44 3
-rw-r--r--. 2 root root 0 Jan  6 04:44 4
-rw-r--r--. 2 root root 0 Jan  6 04:44 5
-rw-r--r--. 2 root root 0 Jan  6 04:44 6
-rw-r--r--. 2 root root 0 Jan  6 04:44 7
-rw-r--r--. 2 root root 0 Jan  6 04:44 8
-rw-r--r--. 2 root root 0 Jan  6 04:44 9
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.1
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.2
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.3
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.4
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.5
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.6
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.7
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.8
-rw-r--r--. 2 root root 0 Jan  6 04:44 hard.9
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.1 -> 1
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.2 -> 2
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.3 -> 3
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.4 -> 4
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.5 -> 5
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.6 -> 6
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.7 -> 7
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.8 -> 8
lrwxrwxrwx. 1 root root 1 Jan  6  2016 sym.9 -> 9
[root@dj fuse]# rm -rf *
[root@dj fuse]# ll
total 0
[root@dj fuse]# ls
[root@dj fuse]# mount | grep tiervolume 
10.70.37.165:/tiervolume on /mnt/fuse type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@dj fuse]#
Comment 12 Niels de Vos 2016-01-20 04:19:53 EST
*** Bug 1293798 has been marked as a duplicate of this bug. ***
Comment 14 errata-xmlrpc 2016-03-01 01:04:44 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.