Bug 1449971 - use extra spinlock and hashtable to relieve lock contention in rpc_clnt->conn->saved_frams
Summary: use extra spinlock and hashtable to relieve lock contention in rpc_clnt->conn...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: GlusterFS
Classification: Community
Component: rpc
Version: mainline
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
Assignee: Zhou Zhengping
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-11 09:30 UTC by Zhou Zhengping
Modified: 2019-04-17 12:25 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-17 12:25:00 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)

Description Zhou Zhengping 2017-05-11 09:30:19 UTC 
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Worker Ant 2017-05-11 09:56:21 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#1) for review on master by Zhou Zhengping (johnzzpcrystal)
Comment 2 Worker Ant 2017-05-11 14:45:35 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#2) for review on master by Zhou Zhengping (johnzzpcrystal)
Comment 3 Worker Ant 2017-05-11 16:30:45 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#3) for review on master by Zhou Zhengping (johnzzpcrystal)
Comment 4 Worker Ant 2017-05-11 21:00:00 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#4) for review on master by Zhou Zhengping (johnzzpcrystal)
Comment 5 Worker Ant 2017-07-06 11:18:05 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#5) for review on master by Raghavendra G (rgowdapp)
Comment 6 Worker Ant 2017-07-06 13:58:04 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#6) for review on master by Raghavendra G (rgowdapp)
Comment 7 Worker Ant 2017-07-07 07:49:29 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#7) for review on master by Raghavendra G (rgowdapp)
Comment 8 Mohit Agrawal 2017-07-24 09:24:32 UTC 
Hi,

  I have executed fio after create environment where every brick represent a ramdisk(16G)

>>>>>>>>>>>>>>>>>>>>>>>>>> 

gluster v info
 
Volume Name: dist-repl
Type: Distributed-Replicate
Volume ID: 52073bfc-e847-4529-bbf5-fa4d8ed854cd
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: <host-name-1>:/mnt/dist1/brick1
Brick2: <host-name-2>:/mnt/dist2/brick1
Brick3: <host-name-3>:/mnt/dist3/brick1
Brick4: <host-name-4>:/mnt/dist4/brick2
Brick5: <host-name-5>:/mnt/dist5/brick2
Brick6: <host-name-6>:/mnt/dist6/brick2
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
performance.strict-o-direct: on
performance.open-behind: off

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Below is the job file that i have used

>>>>>>>>>>>>>>>>>>>
[global]
ioengine=libaio
#unified_rw_reporting=1
randrepeat=1
norandommap=1
group_reporting
runtime=60
thread
size=16g
[workload]
bs=4k
rw=randread
iodepth=8
numjobs=1
file_service_type=random
filename=/mnt/glusterfs/perf5/iotest/fio_5
filename=/mnt/glusterfs/perf6/iotest/fio_6
filename=/mnt/glusterfs/perf7/iotest/fio_7
filename=/mnt/glusterfs/perf8/iotest/fio_8

>>>>>>>>>>>>>>>>>>>>>>>



After apply the patch below is fio output

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

fio ./fio.jb 
workload: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio-2.2.8
Starting 1 thread
workload: Laying out IO file(s) (4 file(s) / 16384MB)
Jobs: 1 (f=4): [r(1)] [100.0% done] [1820KB/0KB/0KB /s] [455/0/0 iops] [eta 00m:00s]
workload: (groupid=0, jobs=1): err= 0: pid=26447: Mon Jul 24 04:42:18 2017
  read : io=108708KB, bw=1811.8KB/s, iops=452, runt= 60001msec
    slat (usec): min=329, max=3481, avg=2203.33, stdev=173.25
    clat (usec): min=4, max=19749, avg=15452.31, stdev=573.62
     lat (usec): min=2355, max=22829, avg=17656.33, stdev=623.97
    clat percentiles (usec):
     |  1.00th=[13376],  5.00th=[14656], 10.00th=[14784], 20.00th=[15040],
     | 30.00th=[15296], 40.00th=[15424], 50.00th=[15552], 60.00th=[15680],
     | 70.00th=[15680], 80.00th=[15808], 90.00th=[16064], 95.00th=[16192],
     | 99.00th=[16512], 99.50th=[16768], 99.90th=[17536], 99.95th=[17792],
     | 99.99th=[19072]
    bw (KB  /s): min= 1600, max= 1908, per=100.00%, avg=1812.81, stdev=38.76
    lat (usec) : 10=0.01%
    lat (msec) : 4=0.01%, 10=0.01%, 20=99.98%
  cpu          : usr=0.27%, sys=1.00%, ctx=54193, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=27177/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=108708KB, aggrb=1811KB/s, minb=1811KB/s, maxb=1811KB/s, mint=60001msec, maxt=60001msec



fio ./fio.jb
workload: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio-2.2.8
Starting 1 thread
workload: Laying out IO file(s) (4 file(s) / 16384MB)
Jobs: 1 (f=4): [r(1)] [100.0% done] [1672KB/0KB/0KB /s] [418/0/0 iops] [eta 00m:00s]
workload: (groupid=0, jobs=1): err= 0: pid=26474: Mon Jul 24 04:51:33 2017
  read : io=100616KB, bw=1676.9KB/s, iops=419, runt= 60002msec
    slat (usec): min=394, max=4182, avg=2376.34, stdev=169.48
    clat (usec): min=9, max=27934, avg=16695.50, stdev=616.79
     lat (usec): min=2338, max=31929, avg=19073.37, stdev=674.15
    clat percentiles (usec):
     |  1.00th=[14528],  5.00th=[15808], 10.00th=[16064], 20.00th=[16320],
     | 30.00th=[16512], 40.00th=[16512], 50.00th=[16768], 60.00th=[16768],
     | 70.00th=[17024], 80.00th=[17280], 90.00th=[17280], 95.00th=[17536],
     | 99.00th=[17792], 99.50th=[17792], 99.90th=[18816], 99.95th=[19840],
     | 99.99th=[25216]
    bw (KB  /s): min= 1498, max= 1740, per=100.00%, avg=1677.76, stdev=36.07
    lat (usec) : 10=0.01%
    lat (msec) : 4=0.01%, 10=0.01%, 20=99.94%, 50=0.04%
  cpu          : usr=0.43%, sys=1.63%, ctx=50178, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=25154/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=100616KB, aggrb=1676KB/s, minb=1676KB/s, maxb=1676KB/s, mint=60002msec, maxt=60002msec



>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Without patch below is the fio output

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

fio ./fio.jb
workload: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio-2.2.8
Starting 1 thread
workload: Laying out IO file(s) (4 file(s) / 16384MB)
Jobs: 1 (f=0): [r(1)] [0.6% done] [1752KB/0KB/0KB /s] [438/0/0 iops] [eta 02h:44m:47s]
workload: (groupid=0, jobs=1): err= 0: pid=26820: Mon Jul 24 05:08:01 2017
  read : io=102872KB, bw=1714.6KB/s, iops=428, runt= 60001msec
    slat (usec): min=413, max=4271, avg=2327.81, stdev=169.79
    clat (usec): min=4, max=24648, avg=16328.31, stdev=551.21
     lat (usec): min=2234, max=28929, avg=18656.90, stdev=596.20
    clat percentiles (usec):
     |  1.00th=[14272],  5.00th=[15552], 10.00th=[15808], 20.00th=[16064],
     | 30.00th=[16192], 40.00th=[16320], 50.00th=[16320], 60.00th=[16512],
     | 70.00th=[16512], 80.00th=[16768], 90.00th=[16768], 95.00th=[17024],
     | 99.00th=[17280], 99.50th=[17536], 99.90th=[18304], 99.95th=[19840],
     | 99.99th=[22656]
    bw (KB  /s): min= 1516, max= 1768, per=100.00%, avg=1715.31, stdev=26.93
    lat (usec) : 10=0.01%
    lat (msec) : 4=0.01%, 10=0.01%, 20=99.94%, 50=0.04%
  cpu          : usr=0.31%, sys=0.97%, ctx=51294, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=25718/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=102872KB, aggrb=1714KB/s, minb=1714KB/s, maxb=1714KB/s, mint=60001msec, maxt=60001msec



fio ./fio.jb
workload: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio-2.2.8
Starting 1 thread
workload: Laying out IO file(s) (4 file(s) / 16384MB)
Jobs: 1 (f=4): [r(1)] [100.0% done] [1796KB/0KB/0KB /s] [449/0/0 iops] [eta 00m:00s]
workload: (groupid=0, jobs=1): err= 0: pid=26846: Mon Jul 24 05:17:04 2017
  read : io=107204KB, bw=1786.8KB/s, iops=446, runt= 60001msec
    slat (usec): min=363, max=4204, avg=2234.24, stdev=161.73
    clat (usec): min=4, max=21290, avg=15669.09, stdev=493.39
     lat (usec): min=2368, max=23822, avg=17904.02, stdev=528.43
    clat percentiles (usec):
     |  1.00th=[13632],  5.00th=[15040], 10.00th=[15296], 20.00th=[15424],
     | 30.00th=[15552], 40.00th=[15680], 50.00th=[15680], 60.00th=[15808],
     | 70.00th=[15936], 80.00th=[15936], 90.00th=[16192], 95.00th=[16320],
     | 99.00th=[16512], 99.50th=[16768], 99.90th=[17280], 99.95th=[17536],
     | 99.99th=[18560]
    bw (KB  /s): min= 1712, max= 1828, per=100.00%, avg=1787.66, stdev=15.46
    lat (usec) : 10=0.01%
    lat (msec) : 4=0.01%, 10=0.01%, 20=99.97%, 50=0.01%
  cpu          : usr=0.25%, sys=0.92%, ctx=53445, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=26801/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: io=107204KB, aggrb=1786KB/s, minb=1786KB/s, maxb=1786KB/s, mint=60001msec, maxt=60001msec

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Regards
Mohit Agrawal
Comment 9 Worker Ant 2017-07-30 16:13:24 UTC 
REVIEW: https://review.gluster.org/17251 (rpc: use extra spinlock and hash table to relieve lock contention) posted (#8) for review on master by Raghavendra G (rgowdapp)
Comment 10 Amar Tumballi 2018-10-08 17:40:23 UTC 
the above patch is now Abandon'd.
Note You need to log in before you can comment on or make changes to this bug.