Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: ======================= Tracker bug for the performance issue observed in glusterfs-3.4.0.15rhs-1.el6rhs.x86_64 Version-Release number of selected component (if applicable): ============================================================== glusterfs-geo-replication-3.4.0.15rhs-1.el6rhs.x86_64 glusterfs-3.4.0.15rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.15rhs-1.el6rhs.x86_64 glusterfs-devel-3.4.0.15rhs-1.el6rhs.x86_64 glusterfs-debuginfo-3.4.0.15rhs-1.el6rhs.x86_64 glusterfs-fuse-3.4.0.15rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.15rhs-1.el6rhs.x86_64 Steps to Reproduce: ==================== 1. Create and start 6*2 volume 2. Mount it on Fuse client 3. untar the linux kernel from mount Actual results: =============== performance is degraded, it is extracting ~10 in 6 seconds.
Problem: In 3.3, inode locks of both metadata and data are competing in same domain called data domain (old style). This coupled with eager-lock, delayed post-ops introduce delays for metadata operations like chmod, chown etc. To avoid this problem, inode locks for metadata ops are moved to different domain called metadata domain in 3.4 (new style). But when both 3.3 clients and 3.4 clients are present, 3.4 clients for metadata operations still need to take locks in "old style" so that proper synchronization happens across 3.3 and 3.4 clients. Only when all clients are >= 3.4 locks will be taken in "new style" for metadata locks. Because of this behavior as long as at least one 3.3 client is present, delays will be perceived for doing metadata operations on all 3.4 clients while data operations are in progress (Ex: Untar will untar one file per sec). Fix: Make locks xlators translate old-style metadata locks to new-style metadata locks. Since upgrade process suggests upgrading servers first and then clients, this approach gives good results. Tests: 1) Tested that old style metadata locks are converted to new style by locks xlator using gdb 2) Tested that disconnects purge locks in meta-data domain as well using gdb and statedumps. 3) Tested that untar performance is not hampered by meta-data and data operations. 4) Had two mounts one with orthogonal-meta-data on and other with orthogonal-meta-data off ran chmod 777 <file> on one mount and chmod 555 <file> on the other mount in while loops when I took statedumps I saw that both the transports are taking lock on same domain with same range. 18:49:30 :) ⚡ sudo grep -B1 "ACTIVE" /usr/local/var/run/gluster/home-gfs-r2_0.324.dump.* home-gfs-r2_0.324.dump.1375794971-lock-dump.domain.domain=r2-replicate-0:metadata home-gfs-r2_0.324.dump.1375794971:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=9223372036854775806, len=0, pid = 7525, owner=78f9e652497f0000, transport=0x15ac9e0, , granted at Tue Aug 6 18:46:11 2013 home-gfs-r2_0.324.dump.1375795051-lock-dump.domain.domain=r2-replicate-0:metadata home-gfs-r2_0.324.dump.1375795051:inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=9223372036854775806, len=0, pid = 8879, owner=0019cc3cad7f0000, transport=0x158f580, , granted at Tue Aug 6 18:47:31 2013
https://code.engineering.redhat.com/gerrit/#/c/11195 & https://code.engineering.redhat.com/gerrit/#/c/11191
With the 3.4.0.18rhs-1 version I am seeing normal performance for kernel compiles over gigabit: executing compile_kernel real 37m17.459s user 8m44.282s sys 2m10.512s end:13:09:19 removed kernel 1 Total 1 tests were successful Marking as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html