Bug 990510

Summary:	fd leaks observed while running dbench with "open-behind" volume option set to "on" on a replicate volume
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	spandura
Component:	glusterfs	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED ERRATA	QA Contact:	spandura
Severity:	high	Docs Contact:
Priority:	high
Version:	2.1	CC:	amarts, nsathyan, rhs-bugs, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.0.15rhs	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	991622 (view as bug list)		Environment:
Last Closed:	2013-09-23 22:35:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	991622

Description spandura 2013-07-31 10:42:54 UTC

Description of problem:
========================
running dbench on a replicate volume ( 1 x 2 ) with "open-behind" volume option set to "on" causes fd leaks. 

Version-Release number of selected component (if applicable):
===============================================================
root@king [Jul-31-2013-16:03:45] >rpm -qa | grep glusterfs-server
glusterfs-server-3.4.0.13rhs-1.el6rhs.x86_64

root@king [Jul-31-2013-16:03:58] >gluster --version
glusterfs 3.4.0.13rhs built on Jul 28 2013 15:22:56

How reproducible:
================
Often

Steps to Reproduce:
==================
1. create a 1 x 2 replicate volume. Set "open-behind" to "on"

2. Create 2 Fuse mount

3. On both the mount points ,run "dbench -s -F -S -x  --one-byte-write-fix --stat-check 10"

4. Kill dbench after running for about 3 minutes.

5. On both the storage_nodes, do
 ls -l /proc/pid_of_brick(s)/fd|grep deleted 

Actual results:
===================
fd leaks observed on brick process. 

root@king [Jul-31-2013-15:08:12] >gluster v status
Status of volume: vol_rep
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick king:/rhs/bricks/b0				49152	Y	28380
Brick hicks:/rhs/bricks/b1				49152	Y	27937
NFS Server on localhost					2049	Y	28392
Self-heal Daemon on localhost				N/A	Y	28400
NFS Server on hicks					2049	Y	27996
Self-heal Daemon on hicks				N/A	Y	28009
 
Storage_node1 "/proc/<brick_pid>/fd" directory listing.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
root@king [Jul-31-2013-15:10:35] >ls -l /proc/28380/fd
total 0
lr-x------ 1 root root 64 Jul 30 18:00 0 -> /dev/null
l-wx------ 1 root root 64 Jul 30 18:00 1 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:00 10 -> socket:[2364581]
lrwx------ 1 root root 64 Jul 31 14:41 101 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:00 11 -> /dev/urandom
lr-x------ 1 root root 64 Jul 30 18:00 12 -> /rhs/bricks/b0
lrwx------ 1 root root 64 Jul 30 18:00 13 -> socket:[2364729]
lrwx------ 1 root root 64 Jul 30 18:00 14 -> socket:[2364939]
lrwx------ 1 root root 64 Jul 30 18:00 15 -> socket:[2364733]
lrwx------ 1 root root 64 Jul 30 18:00 16 -> socket:[2364921]
lrwx------ 1 root root 64 Jul 30 18:00 17 -> socket:[2364941]
lrwx------ 1 root root 64 Jul 30 18:06 18 -> socket:[2407574]
lrwx------ 1 root root 64 Jul 30 18:06 19 -> socket:[2407575]
l-wx------ 1 root root 64 Jul 30 18:00 2 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:06 20 -> socket:[2410718]
lrwx------ 1 root root 64 Jul 30 18:06 21 -> socket:[2410719]
lr-x------ 1 root root 64 Jul 30 18:06 22 -> socket:[2410972]
lr-x------ 1 root root 64 Jul 30 18:06 23 -> socket:[2410973]
lrwx------ 1 root root 64 Jul 30 18:06 24 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 25 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 26 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 27 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 28 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__43212.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:06 29 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 3 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jul 30 18:06 30 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 31 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 32 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 33 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 34 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 35 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 36 -> /rhs/bricks/b0/clients/client6/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 37 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 38 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 39 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__3F2C4.DB (deleted)
l-wx------ 1 root root 64 Jul 30 18:00 4 -> /var/log/glusterfs/bricks/rhs-bricks-b0.log
lrwx------ 1 root root 64 Jul 30 18:06 40 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 41 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 42 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 43 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 44 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 45 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 46 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 47 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 48 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 49 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 5 -> /var/lib/glusterd/vols/vol_rep/run/king-rhs-bricks-b0.pid
lrwx------ 1 root root 64 Jul 30 18:06 51 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 52 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 54 -> /rhs/bricks/b0/clients/client0/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 55 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 56 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 57 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__52E2B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 58 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 59 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 6 -> socket:[2364565]
lrwx------ 1 root root 64 Jul 31 14:41 60 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 61 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 62 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 63 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 65 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__414F2.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 67 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 69 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 7 -> socket:[2364656]
lrwx------ 1 root root 64 Jul 31 14:41 70 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 71 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 73 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 75 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 76 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 78 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 79 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 8 -> socket:[2364574]
lrwx------ 1 root root 64 Jul 31 14:41 80 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 84 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 85 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 9 -> socket:[2364796]
lrwx------ 1 root root 64 Jul 31 14:41 91 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 92 -> /rhs/bricks/b0/clients/client7/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 99 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)

Storage_node2 "/proc/<brick_pid>/fd" directory listing.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
oot@hicks [Jul-31-2013-15:10:41] >ls -l /proc/27937/fd
total 0
lr-x------ 1 root root 64 Jul 30 18:00 0 -> /dev/null
l-wx------ 1 root root 64 Jul 30 18:00 1 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:00 10 -> socket:[2426084]
lrwx------ 1 root root 64 Jul 31 14:42 101 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:00 11 -> /dev/urandom
lr-x------ 1 root root 64 Jul 30 18:00 12 -> /rhs/bricks/b1
lrwx------ 1 root root 64 Jul 30 18:00 13 -> socket:[2427374]
lrwx------ 1 root root 64 Jul 30 18:00 14 -> socket:[2427427]
lrwx------ 1 root root 64 Jul 30 18:00 15 -> socket:[2426324]
lrwx------ 1 root root 64 Jul 30 18:00 16 -> socket:[2426332]
lrwx------ 1 root root 64 Jul 30 18:00 17 -> socket:[2426334]
lrwx------ 1 root root 64 Jul 30 18:06 18 -> socket:[2469921]
lrwx------ 1 root root 64 Jul 30 18:06 19 -> socket:[2469922]
l-wx------ 1 root root 64 Jul 30 18:00 2 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:06 20 -> socket:[2472781]
lrwx------ 1 root root 64 Jul 30 18:06 21 -> socket:[2472782]
lr-x------ 1 root root 64 Jul 30 18:06 22 -> socket:[2472942]
lr-x------ 1 root root 64 Jul 30 18:06 23 -> socket:[2472943]
lrwx------ 1 root root 64 Jul 30 18:06 24 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 25 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 26 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 27 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:06 28 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 29 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 3 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jul 30 18:06 30 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 31 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 32 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 33 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 34 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 35 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 36 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 37 -> /rhs/bricks/b1/clients/client6/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 38 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 39 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__S31.VAL (deleted)
l-wx------ 1 root root 64 Jul 30 18:00 4 -> /var/log/glusterfs/bricks/rhs-bricks-b1.log
lrwx------ 1 root root 64 Jul 30 18:06 40 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 41 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 42 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 43 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 44 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 45 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 46 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 47 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 48 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 5 -> /var/lib/glusterd/vols/vol_rep/run/hicks-rhs-bricks-b1.pid
lrwx------ 1 root root 64 Jul 30 18:06 50 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 51 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 52 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 53 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 54 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 55 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__52E2B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 56 -> /rhs/bricks/b1/clients/client0/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 57 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 59 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 6 -> socket:[2426068]
lrwx------ 1 root root 64 Jul 31 14:42 60 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 61 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 62 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 63 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 64 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 65 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 66 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 67 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 69 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__414F2.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 7 -> socket:[2427233]
lrwx------ 1 root root 64 Jul 31 14:42 70 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 72 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 73 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 74 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 76 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 8 -> socket:[2426077]
lrwx------ 1 root root 64 Jul 31 14:42 80 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 81 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 85 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 87 -> /rhs/bricks/b1/clients/client7/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 9 -> socket:[2427234]
lrwx------ 1 root root 64 Jul 31 14:42 94 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 98 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)

Expected results:
=================
Once dbench is killed, the brick processes must not have open/unlinked fds 

Additional info:
=================

root@king [Jul-31-2013-14:30:03] >gluster v info
 
Volume Name: vol_rep
Type: Replicate
Volume ID: 0a19be1c-e6ad-44c9-880d-e33774ce4311
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: king:/rhs/bricks/b0
Brick2: hicks:/rhs/bricks/b1
Options Reconfigured:
performance.open-behind: on

Comment 2 Amar Tumballi 2013-08-05 01:52:39 UTC

https://code.engineering.redhat.com/gerrit/#/c/11077/

Comment 3 Pranith Kumar K 2013-08-05 07:01:44 UTC

The bug is resulted whenever application does open on a file and does either unlink of the file or rename on to this file, replacing this file. This fix does necessary unrefs on the fd objects so that close happens on this opened files even in those cases. Please look at http://review.gluster.org/#/c/5493/3/tests/bugs/bug-991622.t for detailed test cases.

Comment 4 spandura 2013-08-07 12:21:03 UTC

Verified the fix on the build:
===============================
glusterfs 3.4.0.17rhs built on Aug  6 2013 13:47:26

Case 2:-
===========
1. Create a replicate volume (1 x 2). Set the open-behind to on. 

2. Create a fuse mount

3. Execute the following from fuse mount

exec 5>>test_file1
exec 6>>test_file2

echo "In file1" >&5
echo "In file2" >&6

cat ./test_file1
cat ./test_file2

touch ./test_file3

ls -l 
ls -l /proc/self/fd

rm -f ./test_file1
mv test_file3 test_file2

ls -l
ls -l /proc/self/fd

// This will list the fd's for files test_file1 and test_file2 (deleted) 

// Close the fd's.
exec 5>&-
exec 6>&-

Check the /proc/<brick_pid>/fd on both the bricks. 

There should not be any fd's to file test_file1 and test_file2 . 


Bug is fixed.

Comment 5 Scott Haines 2013-09-23 22:35:57 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html