Bug 1283988 - [RFE] introducing unix domain socket for I/O
Summary: [RFE] introducing unix domain socket for I/O
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: GlusterFS
Classification: Community
Component: transport
Version: mainline
Hardware: All
OS: Unspecified
medium
medium
Target Milestone: ---
Assignee: Prasanna Kumar Kalever
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-11-20 13:19 UTC by Prasanna Kumar Kalever
Modified: 2020-06-19 11:53 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2019-05-11 01:56:59 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)
fio stats with loopback and uds connection mode on latest master (110.00 KB, application/x-tar)
2020-06-19 11:53 UTC, Vinayak Hariharmath 		no flags Details
View All

Description Prasanna Kumar Kalever 2015-11-20 13:19:19 UTC 
Description of problem:
                              
In any hyperconverged setup, client and brick process may run on same node,  in such cases it is always better to use unix domain socket (UDS).
                                      
Currently we use TCP loopback connection for communicating between gluster
client and brick process.      
                         
Version-Release number of selected component (if applicable):
mainline

Expected Result:
Better Performance
Comment 1 Vijay Bellur 2015-11-20 13:50:34 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#1) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 2 Vijay Bellur 2015-11-20 15:00:18 UTC 
REVIEW: http://review.gluster.org/12709 ([WIP]transport: introducing unix domain socket for I/O) posted (#2) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 3 Vijay Bellur 2015-11-21 08:43:41 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#3) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 4 Vijay Bellur 2015-12-08 06:31:20 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#4) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 5 Vijay Bellur 2015-12-08 21:47:59 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#5) for review on master by Vijay Bellur (vbellur)
Comment 6 Vijay Bellur 2016-02-09 14:36:42 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#6) for review on master by Jeff Darcy (jdarcy)
Comment 7 Vijay Bellur 2016-03-17 15:11:13 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#7) for review on master by Prasanna Kumar Kalever (prasanna.kalever)
Comment 8 Vijay Bellur 2016-03-17 16:44:03 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#8) for review on master by Prasanna Kumar Kalever (prasanna.kalever)
Comment 9 Vijay Bellur 2016-03-22 12:02:19 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#9) for review on master by Prasanna Kumar Kalever (prasanna.kalever)
Comment 10 Mike McCune 2016-03-28 23:22:56 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 11 Vijay Bellur 2016-03-29 06:42:58 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#10) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 12 Vijay Bellur 2016-03-30 08:27:57 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#11) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 13 Vijay Bellur 2016-04-01 17:47:16 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#12) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 14 Vijay Bellur 2016-04-02 15:18:51 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#13) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 15 Vijay Bellur 2016-04-11 05:59:41 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#14) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 16 Vijay Bellur 2016-05-11 10:18:57 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#15) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 17 Vijay Bellur 2016-05-24 08:03:32 UTC 
REVIEW: http://review.gluster.org/12709 (transport: introducing unix domain socket for I/O) posted (#16) for review on master by Prasanna Kumar Kalever (pkalever)
Comment 18 Amar Tumballi 2019-05-11 01:56:59 UTC 
We will revisit it after couple of releases, and hence DEFERRED.
Comment 19 Vinayak Hariharmath 2020-06-19 11:53:17 UTC 
Created attachment 1698078 [details]
fio stats with loopback and uds connection mode on latest master

Considerations:
randwrite and randread:
no_jobs: 16 (twice num CPUs)
loops: 2

seqwrite and seqread:
no_jobs: 1
loops: 2

Observations:
We see some improvement in reads and writes that too in few MBs but not in so proving qty.

About network load using nload:

We see an equal amount of traffic flowing (in MBPS) across loopback address in loopback connection mode.
When we switch to uds, the traffic across loopback will be almost nil (some noise found which is expected)

List of commands:
randwrite:
# fio --name=randwrite --name=randwrite --ioengine=sync --rw=randwrite --bs=4k --direct=0 --size=512M --nr_files=1 --numjobs=16 --fsync_on_close=1 --end_fsync=1 --fallocate=none --sync=1 --randrepeat=0 --overwrite=0 --directory=/mnt/lustre --loops=2

seq_write:
# fio --name=seqwrite --ioengine=sync --rw=write --bs=1m --direct=0 --size=10G --nr_files=1 --numjobs=1 --fsync_on_close=1 --end_fsync=1 --fallocate=none --sync=1 --randrepeat=0 --overwrite=0 --directory=/mnt/lustre --loops=2

randread:
# fio --name=randread --ioengine=sync --rw=randread --nr_files=1 --bs=4k --direct=0 --size=512m --numjobs=16 --fsync_on_close=1 --sync=1 --end_fsync=1 --fallocate=none --randrepeat=0 --invalidate=1 --directory=/mnt/lustre --loops=2

seq_read:
# fio --name=seqread --ioengine=sync --rw=read --bs=1m --direct=0 --size=50G --numjobs=1 --fsync_on_close=1 --end_fsync=1 --fallocate=none --randrepeat=0 --invalidate=1 --directory=/mnt/lustre --loops=2
Note You need to log in before you can comment on or make changes to this bug.