990510 – fd leaks observed while running dbench with "open-behind" volume option set to "on" on a replicate volume

Bug 990510 - fd leaks observed while running dbench with "open-behind" volume option set to "on" on a replicate volume

Summary: fd leaks observed while running dbench with "open-behind" volume option set t...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	glusterfs
Sub Component:
Version:	2.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Pranith Kumar K
QA Contact:	spandura
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	991622
TreeView+	depends on / blocked

Reported:	2013-07-31 10:42 UTC by spandura
Modified:	2013-09-23 22:35 UTC (History)
CC List:	4 users (show)
Fixed In Version:	glusterfs-3.4.0.15rhs
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	991622 (view as bug list)
Environment:
Last Closed:	2013-09-23 22:35:57 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description spandura 2013-07-31 10:42:54 UTC

Description of problem:
========================
running dbench on a replicate volume ( 1 x 2 ) with "open-behind" volume option set to "on" causes fd leaks. 

Version-Release number of selected component (if applicable):
===============================================================
root@king [Jul-31-2013-16:03:45] >rpm -qa | grep glusterfs-server
glusterfs-server-3.4.0.13rhs-1.el6rhs.x86_64

root@king [Jul-31-2013-16:03:58] >gluster --version
glusterfs 3.4.0.13rhs built on Jul 28 2013 15:22:56

How reproducible:
================
Often

Steps to Reproduce:
==================
1. create a 1 x 2 replicate volume. Set "open-behind" to "on"

2. Create 2 Fuse mount

3. On both the mount points ,run "dbench -s -F -S -x  --one-byte-write-fix --stat-check 10"

4. Kill dbench after running for about 3 minutes.

5. On both the storage_nodes, do
 ls -l /proc/pid_of_brick(s)/fd|grep deleted 

Actual results:
===================
fd leaks observed on brick process. 

root@king [Jul-31-2013-15:08:12] >gluster v status
Status of volume: vol_rep
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick king:/rhs/bricks/b0				49152	Y	28380
Brick hicks:/rhs/bricks/b1				49152	Y	27937
NFS Server on localhost					2049	Y	28392
Self-heal Daemon on localhost				N/A	Y	28400
NFS Server on hicks					2049	Y	27996
Self-heal Daemon on hicks				N/A	Y	28009
 
Storage_node1 "/proc/<brick_pid>/fd" directory listing.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
root@king [Jul-31-2013-15:10:35] >ls -l /proc/28380/fd
total 0
lr-x------ 1 root root 64 Jul 30 18:00 0 -> /dev/null
l-wx------ 1 root root 64 Jul 30 18:00 1 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:00 10 -> socket:[2364581]
lrwx------ 1 root root 64 Jul 31 14:41 101 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:00 11 -> /dev/urandom
lr-x------ 1 root root 64 Jul 30 18:00 12 -> /rhs/bricks/b0
lrwx------ 1 root root 64 Jul 30 18:00 13 -> socket:[2364729]
lrwx------ 1 root root 64 Jul 30 18:00 14 -> socket:[2364939]
lrwx------ 1 root root 64 Jul 30 18:00 15 -> socket:[2364733]
lrwx------ 1 root root 64 Jul 30 18:00 16 -> socket:[2364921]
lrwx------ 1 root root 64 Jul 30 18:00 17 -> socket:[2364941]
lrwx------ 1 root root 64 Jul 30 18:06 18 -> socket:[2407574]
lrwx------ 1 root root 64 Jul 30 18:06 19 -> socket:[2407575]
l-wx------ 1 root root 64 Jul 30 18:00 2 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:06 20 -> socket:[2410718]
lrwx------ 1 root root 64 Jul 30 18:06 21 -> socket:[2410719]
lr-x------ 1 root root 64 Jul 30 18:06 22 -> socket:[2410972]
lr-x------ 1 root root 64 Jul 30 18:06 23 -> socket:[2410973]
lrwx------ 1 root root 64 Jul 30 18:06 24 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 25 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 26 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 27 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 28 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__43212.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:06 29 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 3 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jul 30 18:06 30 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 31 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 32 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 33 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 34 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 35 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 36 -> /rhs/bricks/b0/clients/client6/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 37 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 38 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 39 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__3F2C4.DB (deleted)
l-wx------ 1 root root 64 Jul 30 18:00 4 -> /var/log/glusterfs/bricks/rhs-bricks-b0.log
lrwx------ 1 root root 64 Jul 30 18:06 40 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 41 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 42 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 43 -> /rhs/bricks/b0/clients/client9/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 44 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 45 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 46 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 47 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 48 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 49 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 5 -> /var/lib/glusterd/vols/vol_rep/run/king-rhs-bricks-b0.pid
lrwx------ 1 root root 64 Jul 30 18:06 51 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 52 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 54 -> /rhs/bricks/b0/clients/client0/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 55 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 56 -> /rhs/bricks/b0/clients/client2/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 57 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__52E2B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 58 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 59 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 6 -> socket:[2364565]
lrwx------ 1 root root 64 Jul 31 14:41 60 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 61 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 62 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 63 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 65 -> /rhs/bricks/b0/clients/client1/~dmtmp/PARADOX/__414F2.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 67 -> /rhs/bricks/b0/clients/client0/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 69 -> /rhs/bricks/b0/clients/client6/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 7 -> socket:[2364656]
lrwx------ 1 root root 64 Jul 31 14:41 70 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 71 -> /rhs/bricks/b0/clients/client5/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 73 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 75 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 76 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 78 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 79 -> /rhs/bricks/b0/clients/client3/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 8 -> socket:[2364574]
lrwx------ 1 root root 64 Jul 31 14:41 80 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 84 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 85 -> /rhs/bricks/b0/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 9 -> socket:[2364796]
lrwx------ 1 root root 64 Jul 31 14:41 91 -> /rhs/bricks/b0/clients/client7/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 92 -> /rhs/bricks/b0/clients/client7/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 31 14:41 99 -> /rhs/bricks/b0/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)

Storage_node2 "/proc/<brick_pid>/fd" directory listing.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
oot@hicks [Jul-31-2013-15:10:41] >ls -l /proc/27937/fd
total 0
lr-x------ 1 root root 64 Jul 30 18:00 0 -> /dev/null
l-wx------ 1 root root 64 Jul 30 18:00 1 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:00 10 -> socket:[2426084]
lrwx------ 1 root root 64 Jul 31 14:42 101 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:00 11 -> /dev/urandom
lr-x------ 1 root root 64 Jul 30 18:00 12 -> /rhs/bricks/b1
lrwx------ 1 root root 64 Jul 30 18:00 13 -> socket:[2427374]
lrwx------ 1 root root 64 Jul 30 18:00 14 -> socket:[2427427]
lrwx------ 1 root root 64 Jul 30 18:00 15 -> socket:[2426324]
lrwx------ 1 root root 64 Jul 30 18:00 16 -> socket:[2426332]
lrwx------ 1 root root 64 Jul 30 18:00 17 -> socket:[2426334]
lrwx------ 1 root root 64 Jul 30 18:06 18 -> socket:[2469921]
lrwx------ 1 root root 64 Jul 30 18:06 19 -> socket:[2469922]
l-wx------ 1 root root 64 Jul 30 18:00 2 -> /dev/null
lrwx------ 1 root root 64 Jul 30 18:06 20 -> socket:[2472781]
lrwx------ 1 root root 64 Jul 30 18:06 21 -> socket:[2472782]
lr-x------ 1 root root 64 Jul 30 18:06 22 -> socket:[2472942]
lr-x------ 1 root root 64 Jul 30 18:06 23 -> socket:[2472943]
lrwx------ 1 root root 64 Jul 30 18:06 24 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 25 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 26 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 27 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/__42B96.DB (deleted)
lr-x------ 1 root root 64 Jul 30 18:06 28 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 29 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 3 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Jul 30 18:06 30 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 31 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 32 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 33 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 34 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 35 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 36 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 37 -> /rhs/bricks/b1/clients/client6/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 38 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__3F2C4.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 39 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__S31.VAL (deleted)
l-wx------ 1 root root 64 Jul 30 18:00 4 -> /var/log/glusterfs/bricks/rhs-bricks-b1.log
lrwx------ 1 root root 64 Jul 30 18:06 40 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 41 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 42 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 43 -> /rhs/bricks/b1/clients/client9/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 44 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 45 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 46 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 47 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 48 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 5 -> /var/lib/glusterd/vols/vol_rep/run/hicks-rhs-bricks-b1.pid
lrwx------ 1 root root 64 Jul 30 18:06 50 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 51 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 52 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 53 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 54 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 55 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__52E2B.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:06 56 -> /rhs/bricks/b1/clients/client0/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 57 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 59 -> /rhs/bricks/b1/clients/client2/~dmtmp/PARADOX/__S31.VAL (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 6 -> socket:[2426068]
lrwx------ 1 root root 64 Jul 31 14:42 60 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 61 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 62 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 63 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 64 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 65 -> /rhs/bricks/b1/clients/client6/~dmtmp/PARADOX/__40D6B.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 66 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 67 -> /rhs/bricks/b1/clients/client0/~dmtmp/PARADOX/__43904.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 69 -> /rhs/bricks/b1/clients/client1/~dmtmp/PARADOX/__414F2.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 7 -> socket:[2427233]
lrwx------ 1 root root 64 Jul 31 14:42 70 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 72 -> /rhs/bricks/b1/clients/client5/~dmtmp/PARADOX/__QB4.MB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 73 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__52EF7.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 74 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 76 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__S31.DB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 8 -> socket:[2426077]
lrwx------ 1 root root 64 Jul 31 14:42 80 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__42B96.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 81 -> /rhs/bricks/b1/clients/client3/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 85 -> /rhs/bricks/b1/clients/client8/~dmtmp/PARADOX/ERRORCHG.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 87 -> /rhs/bricks/b1/clients/client7/~dmtmp/ACCESS/FASTENER.LDB (deleted)
lrwx------ 1 root root 64 Jul 30 18:00 9 -> socket:[2427234]
lrwx------ 1 root root 64 Jul 31 14:42 94 -> /rhs/bricks/b1/clients/client7/~dmtmp/PARADOX/__43212.DB (deleted)
lrwx------ 1 root root 64 Jul 31 14:42 98 -> /rhs/bricks/b1/clients/client4/~dmtmp/PARADOX/__40D6B.DB (deleted)

Expected results:
=================
Once dbench is killed, the brick processes must not have open/unlinked fds 

Additional info:
=================

root@king [Jul-31-2013-14:30:03] >gluster v info
 
Volume Name: vol_rep
Type: Replicate
Volume ID: 0a19be1c-e6ad-44c9-880d-e33774ce4311
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: king:/rhs/bricks/b0
Brick2: hicks:/rhs/bricks/b1
Options Reconfigured:
performance.open-behind: on

Comment 2 Amar Tumballi 2013-08-05 01:52:39 UTC

https://code.engineering.redhat.com/gerrit/#/c/11077/

Comment 3 Pranith Kumar K 2013-08-05 07:01:44 UTC

The bug is resulted whenever application does open on a file and does either unlink of the file or rename on to this file, replacing this file. This fix does necessary unrefs on the fd objects so that close happens on this opened files even in those cases. Please look at http://review.gluster.org/#/c/5493/3/tests/bugs/bug-991622.t for detailed test cases.

Comment 4 spandura 2013-08-07 12:21:03 UTC

Verified the fix on the build:
===============================
glusterfs 3.4.0.17rhs built on Aug  6 2013 13:47:26

Case 2:-
===========
1. Create a replicate volume (1 x 2). Set the open-behind to on. 

2. Create a fuse mount

3. Execute the following from fuse mount

exec 5>>test_file1
exec 6>>test_file2

echo "In file1" >&5
echo "In file2" >&6

cat ./test_file1
cat ./test_file2

touch ./test_file3

ls -l 
ls -l /proc/self/fd

rm -f ./test_file1
mv test_file3 test_file2

ls -l
ls -l /proc/self/fd

// This will list the fd's for files test_file1 and test_file2 (deleted) 

// Close the fd's.
exec 5>&-
exec 6>&-

Check the /proc/<brick_pid>/fd on both the bricks. 

There should not be any fd's to file test_file1 and test_file2 . 


Bug is fixed.

Comment 5 Scott Haines 2013-09-23 22:35:57 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.