Bug 849137 - poor 'emptyfiles_delete', directory_crawl' and 'directory_recrawl' performance with rdma transport
poor 'emptyfiles_delete', directory_crawl' and 'directory_recrawl' performanc...
Status: CLOSED WONTFIX
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rdma (Show other bugs)
2.0
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
storage-qa-internal@redhat.com
: Triaged, ZStream
Depends On: 831205 1186127
Blocks: 858457
  Show dependency treegraph
 
Reported: 2012-08-17 07:59 EDT by Vidya Sakar
Modified: 2015-03-23 03:40 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 831205
: 858457 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vidya Sakar 2012-08-17 07:59:23 EDT
+++ This bug was initially created as a clone of Bug #831205 +++

Description of problem:
With rdma transport type emptyfiles_delete, directory_crawl and directory_recrawl performance is very poor compared to tcp transport type.

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa45

How reproducible:
Consistent

Steps to Reproduce:
1. Create a 2*2 dist-rep volume with tcp transport.
2. Run perf test on the mountpoint 3 times and take average.
3. Repeat above steps for rdma transport.
  
Actual results:

Testname                        tcp-numbers         rdma-numbers
emptyfiles_create               763.2867            884.3033
emptyfiles_delete               874.9833            1551.387
smallfiles_create               1200.597            1507.357
smallfiles_rewrite              1013.730            1293.710
smallfiles_read                 669.6133            893.9067
smallfiles_reread               707.1167            898.2900
smallfiles_delete               1061.247            1957.257
largefile_create                26.51667            17.93000
largefile_rewrite               27.60333            12.27334
largefile_read                  15.36000            12.30000
largefile_reread                15.34000            12.19333
largefile_delete                0.61                0.530000
directory_crawl_create          1193.777            1512.123
directory_crawl                 162.9500            657.6867
directory_recrawl               92.33667            442.9267
metadata_modify                 271.2900            190.6067
directory_crawl_delete          422.7900            371.3733

--- Additional comment from amarts@redhat.com on 2012-06-12 09:51:59 EDT ---

possible reason for this to happen is because we are doing a RDMA write for every readdirp() call. Just analyzing, and not having any solid proof to backup with the data other than a code review:

------------
client3_1_ops.c:5640:

	readdirp_rsp_size = xdr_sizeof ((xdrproc_t) xdr_gfs3_readdirp_rsp, &rsp)
                + args->size;

        if ((readdirp_rsp_size + GLUSTERFS_RPC_REPLY_SIZE
             + GLUSTERFS_RDMA_MAX_HEADER_SIZE)
            > (GLUSTERFS_RDMA_INLINE_THRESHOLD)) {

------------

This can be an issue because we are calculating the size of an empty response structure, and when we get some data filled requests we end up getting more XDR bytes than what we asked for in request (ie, args->size). This would lead to disconnections on readdirp() cbk getting more data. This could have caused the issue of performance degradation in this case.

Notice that even for small file delete (and empty file delete), you need to crawl the directory to get all the file names, which is consuming more time IMO.

Raghavendra, please evaluate the possibilities and let me know what you find out.
Comment 5 Vivek Agarwal 2015-03-23 03:39:11 EDT
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html
Comment 6 Vivek Agarwal 2015-03-23 03:40:09 EDT
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html

Note You need to log in before you can comment on or make changes to this bug.