Created attachment 934366 [details]
Logs of the three gluster servers

Description of problem:
 
Disconnections from glusterfs through libgfapi.
 
Version-Release number of selected component (if applicable):
 
glusterfs-3.6.0.22-1.el6rhs.x86_64
 
How reproducible:
 
Open some connections to the glusterfs through libgfapi and write 50K files and read 100K files os 1 MB
 
Steps to Reproduce:
1. Open multiple connections (i.e 64) to a gluster volume using libgfapi.
2. open-write-close files using different threads on the multiple connections
3. Client get disconnected from the gluster volume (open fails)
 
Actual results:
 
I am having an issue with glusterfs disconnections while creating and writing files.
 
The environment is 3 RHEV hypervisors running each one 2 virtual machines, one client and one server, both of them with RHEL 6.5
 
I am running a script that open n threads from the clients to write or read files from the glusterfs volume who shares the three servers, using libgfapi. Before try to do nothing, the script removes the destination folders of the writers if it exists. It only makes a mount using nfs to launch a rm -rf of the destination folder.
Here i have the first strange log:
 
nfs.log:[2014-09-04 07:46:16.260087] W [dht-layout.c:180:dht_layout_search] 0-vol02-dht: no subvolume for hash (value) = 3004083931
 
It only occurs in one of the three servers, and to daze me it only occurs with exactly 1000 files, so as i have to remove 3 folders, i have exactly 3000 events like this in the logs.
 
After that, the script begins to open connections to the gluster volume, it connects to the gluster:
 
[2014-09-04 07:56:59.609786] I [server-handshake.c:578:server_setvolume] 0-vol02-server: accepted client from vxoa01.vna-2273-2014/09/04-07:56:53:865815-vol02-client-22-0-0 (version: 3.6.0.22)
 
but after one or two minutes it begins:
 
[2014-09-04 07:58:17.290542] I [server-helpers.c:291:do_fd_cleanup] 0-vol02-server: fd cleanup on /rtest_3/dir_026/dptest.26008
[2014-09-04 07:58:17.290796] E [client_t.c:384:gf_client_unref] (-->/usr/lib64/glusterfs/3.6.0.22/xlator/features/locks.so(pl_flush_cbk+0xb9) [0x7f0b1a546399] (-->/usr/lib64/libglusterfs.so.0(default_flush_cbk+0xb9) [0x3be322d919] (-->/usr/lib64/glusterfs/3.6.0.22/xlator/debug/io-stats.so(io_stats_flush_cbk+0xed) [0x7f0b198d952d]))) 0-client_t: client is NULL
[2014-09-04 07:58:17.353584] I [server-resolve.c:519:server_resolve_fd] 0-: fd not found in context
[2014-09-04 07:58:17.353679] E [server-rpc-fops.c:1336:server_flush_cbk] 0-vol02-server: 831: FLUSH 4 (d35cf248-fa0b-4b9d-b4ca-733c4d457ea2) ==> (Wrong file descriptor)
 
In the next 2 seconds i have the same message for a lot of different files in differnt volumes and the sessions are disconnected.
 
The gluster volume is online before and after, all in online state and no messages in /var/log except the glusterfs ones. The limit of number of files in the kernel is far (cat /proc/sys/fs/file-nr don't go further of 3500 files and ulimit -n show 32000).
 
Expected results:
 
Avoid the disconnections.
 
Additional info:

Logs of the three servers are attached to this bug.