Bug 990510 - fd leaks observed while running dbench with "open-behind" volume option set to "on" on a replicate volume
fd leaks observed while running dbench with "open-behind" volume option set t...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
2.1
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Pranith Kumar K
spandura
:
Depends On:
Blocks: 991622
  Show dependency treegraph
 
Reported: 2013-07-31 06:42 EDT by spandura
Modified: 2013-09-23 18:35 EDT (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.15rhs
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 991622 (view as bug list)
Environment:
Last Closed: 2013-09-23 18:35:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description spandura 2013-07-31 06:42:54 EDT
Description of problem:
========================
running dbench on a replicate volume ( 1 x 2 ) with "open-behind" volume option set to "on" causes fd leaks. 

Version-Release number of selected component (if applicable):
===============================================================
root@king [Jul-31-2013-16:03:45] >rpm -qa | grep glusterfs-server
glusterfs-server-3.4.0.13rhs-1.el6rhs.x86_64

root@king [Jul-31-2013-16:03:58] >gluster --version
glusterfs 3.4.0.13rhs built on Jul 28 2013 15:22:56

How reproducible:
================
Often

Steps to Reproduce:
==================
1. create a 1 x 2 replicate volume. Set "open-behind" to "on"

2. Create 2 Fuse mount

3. On both the mount points ,run "dbench -s -F -S -x  --one-byte-write-fix --stat-check 10"

4. Kill dbench after running for about 3 minutes.

5. On both the storage_nodes, do
 ls -l /proc/pid_of_brick(s)/fd|grep deleted 

Actual results:
===================
fd leaks observed on brick process. 

root@king [Jul-31-2013-15:08:12] >gluster v status
Status of volume: vol_rep
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick king:/rhs/bricks/b0				49152	Y	28380
Brick hicks:/rhs/bricks/b1				49152	Y	27937
NFS Server on localhost					2049	Y	28392
Self-heal Daemon on localhost				N/A	Y	28400
NFS Server on hicks					2049	Y	27996
Self-heal Daemon on hicks				N/A	Y	28009
 
Storage_node1 "/proc/<brick_pid>/fd" directory listing.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
root@king [Jul-31-2013-15:10:35] >ls -l /proc/28380/fd
total 0
lr-x------ 1 root root 64 Jul 30 18:00 0 -> /dev/null
l-wx------ 1 root root 64 Jul 30 18:00 1 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:00 10 -> socket:[2364581]
lrwx------ 1 root root 64 Jul 31 14:41 101 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:00 11 -> /dev/urandom
lr-x------ 1 root root 64 Jul 30 18:00 12 -> /rhs/bricks/b0
lrwx------ 1 root root 64 Jul 30 18:00 13 -> socket:[2364729]
lrwx------ 1 root root 64 Jul 30 18:00 14 -> socket:[2364939]
lrwx------ 1 root root 64 Jul 30 18:00 15 -> socket:[2364733]
lrwx------ 1 root root 64 Jul 30 18:00 16 -> socket:[2364921]
lrwx------ 1 root root 64 Jul 30 18:00 17 -> socket:[2364941]
lrwx------ 1 root root 64 Jul 30 18:06 18 -> socket:[2407574]
lrwx------ 1 root root 64 Jul 30 18:06 19 -> socket:[2407575]
l-wx------ 1 root root 64 Jul 30 18:00 2 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:06 20 -> socket:[2410718]
lrwx------ 1 root root 64 Jul 30 18:06 21 -> socket:[2410719]
lr-x------ 1 root root 64 Jul 30 18:06 22 -> socket:[2410972]
lr-x------ 1 root root 64 Jul 30 18:06 23 -> socket:[2410973]
lrwx------ 1 root root 64 Jul 30 18:06 24 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 25 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 26 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 27 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 28 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__43212.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:06 29 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 3 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jul 30 18:06 30 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 31 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 32 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 33 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 34 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 35 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 36 -> /rhs/bricks/b0/clients/client6/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 37 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 38 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 39 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__3F2C4.DB (deleted)
l-wx------ 1 root root 64 Jul 30 18:00 4 -> /var/log/glusterfs/bricks/rhs-bricks-b0.log
lrwx------ 1 root root 64 Jul 30 18:06 40 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 41 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 42 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 43 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 44 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 45 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 46 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 47 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 48 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 49 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 5 -> /var/lib/glusterd/vols/vol_rep/run/king-rhs-bricks-b0.pid
lrwx------ 1 root root 64 Jul 30 18:06 51 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 52 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 54 -> /rhs/bricks/b0/clients/client0/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 55 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 56 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 57 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__52E2B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 58 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 59 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 6 -> socket:[2364565]
lrwx------ 1 root root 64 Jul 31 14:41 60 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 61 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 62 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 63 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 65 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__414F2.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 67 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 69 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 7 -> socket:[2364656]
lrwx------ 1 root root 64 Jul 31 14:41 70 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 71 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 73 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 75 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 76 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 78 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 79 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 8 -> socket:[2364574]
lrwx------ 1 root root 64 Jul 31 14:41 80 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 84 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 85 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 9 -> socket:[2364796]
lrwx------ 1 root root 64 Jul 31 14:41 91 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 92 -> /rhs/bricks/b0/clients/client7/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 99 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)

Storage_node2 "/proc/<brick_pid>/fd" directory listing.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
oot@hicks [Jul-31-2013-15:10:41] >ls -l /proc/27937/fd
total 0
lr-x------ 1 root root 64 Jul 30 18:00 0 -> /dev/null
l-wx------ 1 root root 64 Jul 30 18:00 1 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:00 10 -> socket:[2426084]
lrwx------ 1 root root 64 Jul 31 14:42 101 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:00 11 -> /dev/urandom
lr-x------ 1 root root 64 Jul 30 18:00 12 -> /rhs/bricks/b1
lrwx------ 1 root root 64 Jul 30 18:00 13 -> socket:[2427374]
lrwx------ 1 root root 64 Jul 30 18:00 14 -> socket:[2427427]
lrwx------ 1 root root 64 Jul 30 18:00 15 -> socket:[2426324]
lrwx------ 1 root root 64 Jul 30 18:00 16 -> socket:[2426332]
lrwx------ 1 root root 64 Jul 30 18:00 17 -> socket:[2426334]
lrwx------ 1 root root 64 Jul 30 18:06 18 -> socket:[2469921]
lrwx------ 1 root root 64 Jul 30 18:06 19 -> socket:[2469922]
l-wx------ 1 root root 64 Jul 30 18:00 2 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:06 20 -> socket:[2472781]
lrwx------ 1 root root 64 Jul 30 18:06 21 -> socket:[2472782]
lr-x------ 1 root root 64 Jul 30 18:06 22 -> socket:[2472942]
lr-x------ 1 root root 64 Jul 30 18:06 23 -> socket:[2472943]
lrwx------ 1 root root 64 Jul 30 18:06 24 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 25 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 26 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 27 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:06 28 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 29 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 3 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jul 30 18:06 30 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 31 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 32 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 33 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 34 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 35 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 36 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 37 -> /rhs/bricks/b1/clients/client6/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 38 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 39 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__S31.VAL (deleted)
l-wx------ 1 root root 64 Jul 30 18:00 4 -> /var/log/glusterfs/bricks/rhs-bricks-b1.log
lrwx------ 1 root root 64 Jul 30 18:06 40 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 41 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 42 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 43 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 44 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 45 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 46 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 47 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 48 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 5 -> /var/lib/glusterd/vols/vol_rep/run/hicks-rhs-bricks-b1.pid
lrwx------ 1 root root 64 Jul 30 18:06 50 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 51 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 52 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 53 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 54 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 55 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__52E2B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 56 -> /rhs/bricks/b1/clients/client0/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 57 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 59 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 6 -> socket:[2426068]
lrwx------ 1 root root 64 Jul 31 14:42 60 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 61 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 62 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 63 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 64 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 65 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 66 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 67 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 69 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__414F2.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 7 -> socket:[2427233]
lrwx------ 1 root root 64 Jul 31 14:42 70 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 72 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 73 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 74 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 76 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 8 -> socket:[2426077]
lrwx------ 1 root root 64 Jul 31 14:42 80 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 81 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 85 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 87 -> /rhs/bricks/b1/clients/client7/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 9 -> socket:[2427234]
lrwx------ 1 root root 64 Jul 31 14:42 94 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 98 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)

Expected results:
=================
Once dbench is killed, the brick processes must not have open/unlinked fds 

Additional info:
=================

root@king [Jul-31-2013-14:30:03] >gluster v info
 
Volume Name: vol_rep
Type: Replicate
Volume ID: 0a19be1c-e6ad-44c9-880d-e33774ce4311
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: king:/rhs/bricks/b0
Brick2: hicks:/rhs/bricks/b1
Options Reconfigured:
performance.open-behind: on
Comment 3 Pranith Kumar K 2013-08-05 03:01:44 EDT
The bug is resulted whenever application does open on a file and does either unlink of the file or rename on to this file, replacing this file. This fix does necessary unrefs on the fd objects so that close happens on this opened files even in those cases. Please look at http://review.gluster.org/#/c/5493/3/tests/bugs/bug-991622.t for detailed test cases.
Comment 4 spandura 2013-08-07 08:21:03 EDT
Verified the fix on the build:
===============================
glusterfs 3.4.0.17rhs built on Aug  6 2013 13:47:26

Case 2:-
===========
1. Create a replicate volume (1 x 2). Set the open-behind to on. 

2. Create a fuse mount

3. Execute the following from fuse mount

exec 5>>test_file1
exec 6>>test_file2

echo "In file1" >&5
echo "In file2" >&6

cat ./test_file1
cat ./test_file2

touch ./test_file3

ls -l 
ls -l /proc/self/fd

rm -f ./test_file1
mv test_file3 test_file2

ls -l
ls -l /proc/self/fd

// This will list the fd's for files test_file1 and test_file2 (deleted) 

// Close the fd's.
exec 5>&-
exec 6>&-

Check the /proc/<brick_pid>/fd on both the bricks. 

There should not be any fd's to file test_file1 and test_file2 . 


Bug is fixed.
Comment 5 Scott Haines 2013-09-23 18:35:57 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.