Bug 959486

Description of problem:
When using the RDMA transport type on a distributed replicated volume there is a chance that a bad file descriptor error is returned. Due note, I have not tested any other volume type. Changing the volume transport type to tcp according to these directions: 
http://community.gluster.org/q/how-to-change-transport-type-on-active-volume---glusterfs-3-3/
and retesting using IPoIB yields 100% passing results; no failure of any kind. This leads me to believe that the OS, my hardware and infiniband fabric are stable and that the issue lies in the RDMA libraries provided by and used by gluster.


Version-Release number of selected component (if applicable):
OFED 1.5.4.1, GLusterFS 3.4.0 alpha3, debian squeeze OS.

How reproducible:
This issue appears transient as not all test iterations will cause it. I have been able to produce this problem using dd, iozone and our own java sequence writer. During all tests, the issue started showing up when the bytesize was >= 64k. I have seen this issue when the underlying disks are formatted as ext4, XFS and ZFS mirror. 


Steps to Reproduce:
1. Created a distributed replicated volume utilizing only the RDMA transport type.
2. Mount
3. Run dd/iozone operations against the mount
    a. I run dd in a loop with 3 iterations of each bytesize. The dd operation I perform: dd if=/dev/zero of=<mnt>/<file> bs=<16k|32k|64k|128k|256K> count=10000
    b. I run iozone in a loop with 1 iteration of each bytesize. The iozone operation I perform: iozone -R -l 5 -u 5 -r <16k|32k|64k|128k|256K> -s 100m -F <mnt>/<file1> <mnt>/<file2> <mnt>/<file3> <mnt>/<file4> <mnt>/<file5>

NOTE: naturally, replace <*> components in the above cmd's with actual information.

  
Actual results:
Bad File Descriptor:
[2013-05-02 15:46:23.156343] W [fuse-bridge.c:2127:fuse_writev_cbk] 0-glusterfs-fuse: 1175: WRITE => -1 (Bad file descriptor)
[2013-05-02 15:46:23.156820] W [fuse-bridge.c:1132:fuse_err_cbk] 0-glusterfs-fuse: 1176: FLUSH() ERR => -1 (Bad file descriptor)

In the failure case with dd, you have to killall glusterfs/glusterd processes before the dd operation will die as it is stuck in kernel io code. Interestingly, iozone will exit gracefully when this error is encountered.


Expected results:
No error


Additional info:
My infiniband fabric is entirely QDR Mellanox.
Given the nature of my environment and the lack of availability of a recent OFED version for debian, I can provide the packages that I have built.
I can also provide the perl programs that I have written to perform the tests.
Connected mode was used during all IPoIB testing.