+++ This bug was initially created as a clone of Bug #831205 +++ Description of problem: With rdma transport type emptyfiles_delete, directory_crawl and directory_recrawl performance is very poor compared to tcp transport type. Version-Release number of selected component (if applicable): glusterfs-3.3.0qa45 How reproducible: Consistent Steps to Reproduce: 1. Create a 2*2 dist-rep volume with tcp transport. 2. Run perf test on the mountpoint 3 times and take average. 3. Repeat above steps for rdma transport. Actual results: Testname tcp-numbers rdma-numbers emptyfiles_create 763.2867 884.3033 emptyfiles_delete 874.9833 1551.387 smallfiles_create 1200.597 1507.357 smallfiles_rewrite 1013.730 1293.710 smallfiles_read 669.6133 893.9067 smallfiles_reread 707.1167 898.2900 smallfiles_delete 1061.247 1957.257 largefile_create 26.51667 17.93000 largefile_rewrite 27.60333 12.27334 largefile_read 15.36000 12.30000 largefile_reread 15.34000 12.19333 largefile_delete 0.61 0.530000 directory_crawl_create 1193.777 1512.123 directory_crawl 162.9500 657.6867 directory_recrawl 92.33667 442.9267 metadata_modify 271.2900 190.6067 directory_crawl_delete 422.7900 371.3733 --- Additional comment from amarts on 2012-06-12 09:51:59 EDT --- possible reason for this to happen is because we are doing a RDMA write for every readdirp() call. Just analyzing, and not having any solid proof to backup with the data other than a code review: ------------ client3_1_ops.c:5640: readdirp_rsp_size = xdr_sizeof ((xdrproc_t) xdr_gfs3_readdirp_rsp, &rsp) + args->size; if ((readdirp_rsp_size + GLUSTERFS_RPC_REPLY_SIZE + GLUSTERFS_RDMA_MAX_HEADER_SIZE) > (GLUSTERFS_RDMA_INLINE_THRESHOLD)) { ------------ This can be an issue because we are calculating the size of an empty response structure, and when we get some data filled requests we end up getting more XDR bytes than what we asked for in request (ie, args->size). This would lead to disconnections on readdirp() cbk getting more data. This could have caused the issue of performance degradation in this case. Notice that even for small file delete (and empty file delete), you need to crawl the directory to get all the file names, which is consuming more time IMO. Raghavendra, please evaluate the possibilities and let me know what you find out.
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version. [1] https://rhn.redhat.com/errata/RHSA-2014-0821.html