Bug 849137

+++ This bug was initially created as a clone of Bug #831205 +++

Description of problem:
With rdma transport type emptyfiles_delete, directory_crawl and directory_recrawl performance is very poor compared to tcp transport type.

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa45

How reproducible:
Consistent

Steps to Reproduce:
1. Create a 2*2 dist-rep volume with tcp transport.
2. Run perf test on the mountpoint 3 times and take average.
3. Repeat above steps for rdma transport.
  
Actual results:

Testname                        tcp-numbers         rdma-numbers
emptyfiles_create               763.2867            884.3033
emptyfiles_delete               874.9833            1551.387
smallfiles_create               1200.597            1507.357
smallfiles_rewrite              1013.730            1293.710
smallfiles_read                 669.6133            893.9067
smallfiles_reread               707.1167            898.2900
smallfiles_delete               1061.247            1957.257
largefile_create                26.51667            17.93000
largefile_rewrite               27.60333            12.27334
largefile_read                  15.36000            12.30000
largefile_reread                15.34000            12.19333
largefile_delete                0.61                0.530000
directory_crawl_create          1193.777            1512.123
directory_crawl                 162.9500            657.6867
directory_recrawl               92.33667            442.9267
metadata_modify                 271.2900            190.6067
directory_crawl_delete          422.7900            371.3733

--- Additional comment from amarts on 2012-06-12 09:51:59 EDT ---

possible reason for this to happen is because we are doing a RDMA write for every readdirp() call. Just analyzing, and not having any solid proof to backup with the data other than a code review:

------------
client3_1_ops.c:5640:

	readdirp_rsp_size = xdr_sizeof ((xdrproc_t) xdr_gfs3_readdirp_rsp, &rsp)
                + args->size;

        if ((readdirp_rsp_size + GLUSTERFS_RPC_REPLY_SIZE
             + GLUSTERFS_RDMA_MAX_HEADER_SIZE)
            > (GLUSTERFS_RDMA_INLINE_THRESHOLD)) {

------------

This can be an issue because we are calculating the size of an empty response structure, and when we get some data filled requests we end up getting more XDR bytes than what we asked for in request (ie, args->size). This would lead to disconnections on readdirp() cbk getting more data. This could have caused the issue of performance degradation in this case.

Notice that even for small file delete (and empty file delete), you need to crawl the directory to get all the file names, which is consuming more time IMO.

Raghavendra, please evaluate the possibilities and let me know what you find out.