Bug 1164079
Summary: | [Tracker] RDMA support in glusterfs | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Mohammed Rafi KC <rkavunga> | |
Component: | rdma | Assignee: | Mohammed Rafi KC <rkavunga> | |
Status: | CLOSED EOL | QA Contact: | ||
Severity: | medium | Docs Contact: | ||
Priority: | medium | |||
Version: | mainline | CC: | bturner, bugs, chrisw, jthottan, lmohanty, nlevinki, rtalur, rwheeler, sankarshan, storage-qa-internal | |
Target Milestone: | --- | Keywords: | FutureFeature, Tracking, Triaged | |
Target Release: | --- | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Enhancement | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | 1134839 | |||
: | 1166140 (view as bug list) | Environment: | ||
Last Closed: | 2018-11-19 04:07:25 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 831205, 839810, 852370, 1131502, 1134839, 1154617 | |||
Bug Blocks: | 1166140, 1166515 |
Comment 1
Anand Avati
2014-11-14 05:59:07 UTC
COMMIT: http://review.gluster.org/8925 committed in master by Raghavendra G (rgowdapp) ------ commit f645655c65d1f9787a82a5dcbb0e24e2d32bed87 Author: Mohammed Rafi KC <rkavunga> Date: Mon Oct 13 11:12:14 2014 +0530 rdma: glusterd crash if rdma_disconnect is called as soon as connect a request. we are initializing connection in server side immediately after rdma_accept is called. But we are delaying adding the transport to listener list until getting RDMA_CM_EVENT_ESTABLISHED event. Before getting this event if disconnect is called glusterd will try to remove the transport from list which is not added. So if the list is empty it causes a glusterd crash . In this patch we will call the function to initialize the connection as soon as rdma_accept is called. Change-Id: I019480297a85349ede3101ee9c7c1596dc5c73e2 Signed-off-by: Mohammed Rafi KC <rkavunga> BUG: 1164079 Reviewed-on: http://review.gluster.org/8925 Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp> REVIEW: http://review.gluster.org/9141 (rdma:Removing RDMA tech preview cli message) posted (#3) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9141 (rdma:Removing RDMA tech preview cli message) posted (#4) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9093 (rdma:uss support for rdma and tcp,rdma type volumes.) posted (#3) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9141 (rdma:Removing RDMA tech preview cli message.) posted (#5) for review on master by Humble Devassy Chirammal (humble.devassy) REVIEW: http://review.gluster.org/9093 (rdma:uss support for rdma and tcp,rdma type volumes.) posted (#4) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9141 (rdma: Removing RDMA tech preview cli message.) posted (#6) for review on master by Humble Devassy Chirammal (humble.devassy) REVIEW: http://review.gluster.org/9093 (rdma:uss support for rdma and tcp,rdma type volumes.) posted (#5) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9146 (rdma: Client volfile name change for supporting rdma) posted (#1) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9147 (mount: Append -o transport option in mount.glusterfs) posted (#1) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9147 (mount: Handle -o transport option in mount.glusterfs) posted (#2) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9146 (rdma: Client volfile name change for supporting rdma) posted (#2) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9147 (mount: Handle -o transport option in mount.glusterfs) posted (#3) for review on master by Humble Devassy Chirammal (humble.devassy) REVIEW: http://review.gluster.org/9147 (mount: Handle -o transport option in mount.glusterfs) posted (#4) for review on master by Humble Devassy Chirammal (humble.devassy) REVIEW: http://review.gluster.org/9146 (rdma: Client volfile name change for supporting rdma) posted (#3) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9147 (mount: Handle -o transport option in mount.glusterfs) posted (#5) for review on master by mohammed rafi kc (rkavunga) COMMIT: http://review.gluster.org/9146 committed in master by Kaushal M (kaushal) ------ commit 720e2c0df7bbf2d3daed20cadd5f3f710576a05c Author: Anoop C S <achiraya> Date: Wed Nov 19 16:27:05 2014 +0530 rdma: Client volfile name change for supporting rdma For rdma only volumes, daemons like snapd, glustershd etc make use of tcp transport for their operations. This patch will introduce the support of rdma by default for those daemons in rdma only volumes. In order to accomodate this change we rename the tcp client volfile labels from <volname>-fuse.vol to <volname>.tcp-fuse.vol Change-Id: Id9727b97d00e62a4a1556b9c0c56653d45c8fe1d BUG: 1164079 Signed-off-by: Anoop C S <achiraya> Reviewed-on: http://review.gluster.org/9146 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Kaushal M <kaushal> REVIEW: http://review.gluster.org/9147 (mount:Handle -o transport option in mount.glusterfs) posted (#6) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9141 (rdma:Removing RDMA tech preview cli message.) posted (#7) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9141 (rdma:Removing RDMA tech preview cli message.) posted (#8) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9147 (mount:Handle -o transport option in mount.glusterfs) posted (#7) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9141 (rdma:Removing RDMA tech preview cli message.) posted (#9) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9147 (mount:Handle -o transport option in mount.glusterfs) posted (#8) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9147 (mount:Handle -o transport option in mount.glusterfs) posted (#9) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9141 (rdma:Removing RDMA tech preview cli message.) posted (#10) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9147 (mount:Handle -o transport option in mount.glusterfs) posted (#10) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9184 (fuse: Prevent fuse mount from adding transport name to volfile_id) posted (#1) for review on master by Anoop C S (achiraya) COMMIT: http://review.gluster.org/9147 committed in master by Raghavendra G (rgowdapp) ------ commit 0dd772d13853ad862270dea74ad671118ee51813 Author: Mohammed Rafi KC <rkavunga> Date: Wed Nov 19 17:22:21 2014 +0530 mount:Handle -o transport option in mount.glusterfs In current scenario ,when tcp transport type(default) specified for mounting,glusterfs mount script won't append '.tcp' to volume name.But to accommodate the change in http://review.gluster.org/#/c/9146/, we need to append ".tcp" with volfile-id if '-o transport=tcp' is given. Change-Id: I506edeb339de062087f820f10e17c7d67b8b63fe BUG: 1164079 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/9147 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: jiffin tony Thottan <jthottan> Reviewed-by: Raghavendra Talur <rtalur> Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp> COMMIT: http://review.gluster.org/9141 committed in master by Raghavendra G (rgowdapp) ------ commit eb7a2f47bdee5048cfdaf2af149af5cfd7de4b10 Author: Mohammed Rafi KC <rkavunga> Date: Tue Nov 18 14:58:20 2014 +0530 rdma:Removing RDMA tech preview cli message. Creation of rdma and tcp,rdma volume will display a warning message since it was in tech preview. This patch will remove the warning message during the volume creation. Change-Id: I54f46097073bcb5a757ab047f540cd9b92553897 BUG: 1164079 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/9141 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: jiffin tony Thottan <jthottan> Reviewed-by: Raghavendra Talur <rtalur> Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp> REVIEW: http://review.gluster.org/9191 (cli:volume status for tcp,rdma type volume display only tcp port) posted (#2) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9191 (cli:volume status for tcp,rdma type volume display only tcp port) posted (#3) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma:post multiple work request in a single call.) posted (#1) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9321 (rdma : agregate a vectored read as one) posted (#3) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma:post multiple work request in a single call.) posted (#2) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9329 (rdma:read multiple wr from cq and ack them in one call) posted (#1) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9329 (rdma:read multiple wr from cq and ack them in one call) posted (#2) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9321 (rdma : agregate a vectored read as one) posted (#4) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9191 (cli:volume status for tcp,rdma type volume display only tcp port) posted (#4) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9191 (cli:volume status for tcp,rdma type volume display only tcp port) posted (#5) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9191 (cli: volume status for tcp,rdma type volume display only tcp port) posted (#6) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma:post multiple work request in a single call.) posted (#3) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma:post multiple work request in a single call.) posted (#4) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma: post multiple work request in a single call.) posted (#5) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9191 (cli: volume status for tcp,rdma type volume display only tcp port) posted (#7) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma:post multiple work request in a single call.) posted (#6) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma: post multiple work request in a single call.) posted (#7) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9327 (rdma: post multiple work request in a single call.) posted (#8) for review on master by mohammed rafi kc (rkavunga) COMMIT: http://review.gluster.org/9321 committed in master by Raghavendra G (rgowdapp) ------ commit 60361d84a03048b0882935fb92c19ac3b68f9325 Author: Mohammed Rafi KC <rkavunga> Date: Mon Dec 22 11:04:50 2014 +0530 rdma : agregate a vectored read as one For a vectored read with payload count>1 will make two read requests and to hold that a single contiguous memory is allocated. So after completing the read request, instead of sending as vector we will aggregate all the reads one. Change-Id: I15e7d7bddc1a62d5097a39392575f47cfff3d3a8 BUG: 1164079 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/9321 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp> REVIEW: http://review.gluster.org/9327 (rdma: post multiple work request in a single call.) posted (#9) for review on master by Raghavendra G (rgowdapp) REVIEW: http://review.gluster.org/9327 (rdma: post multiple work request in a single call.) posted (#10) for review on master by mohammed rafi kc (rkavunga) COMMIT: http://review.gluster.org/9327 committed in master by Raghavendra G (rgowdapp) ------ commit b8ed8da853ac60a19a2099f0c1de21f50fe34fe4 Author: Mohammed Rafi KC <rkavunga> Date: Mon Dec 22 15:16:43 2014 +0530 rdma: post multiple work request in a single call. ibv_post-send will allow to send multiple work request in a single call posting as linked list. So if the payload count > 1, we can perform the data operation in a single call to ibv_post_send. Change-Id: Ib2e485cbbe6887919109e73e17d4fab595d5e65e BUG: 1164079 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/9327 Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp> REVIEW: http://review.gluster.org/9191 (cli: volume status for tcp,rdma type volume display only tcp port) posted (#8) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9329 (rdma:read multiple wr from cq and ack them in one call) posted (#3) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/9329 (rdma:read multiple wr from cq and ack them in one call) posted (#4) for review on master by mohammed rafi kc (rkavunga) COMMIT: http://review.gluster.org/9329 committed in master by Raghavendra G (rgowdapp) ------ commit 55ce0ef667de7995c4197314153877719a7de539 Author: Mohammed Rafi KC <rkavunga> Date: Tue Dec 23 11:31:37 2014 +0530 rdma:read multiple wr from cq and ack them in one call we are reading one work completion request at a time from cq, though we can read multiple work completion requests from cq. Also we can acknowledge them in one call itself. Both will give a better performance because of less mutual exclusion locks are being performed. Change-Id: Ib5664cab25c87db7f575d482eee4dcd2b5005c04 BUG: 1164079 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/9329 Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp> REVIEW: http://review.gluster.org/9191 (cli: volume status for tcp,rdma type volume display only tcp port) posted (#9) for review on master by mohammed rafi kc (rkavunga) COMMIT: http://review.gluster.org/9191 committed in master by Kaushal M (kaushal) ------ commit 692dd7e83cc8a1c85581dda3f752d5ea3b184f8c Author: Mohammed Rafi KC <rkavunga> Date: Mon Nov 24 17:07:02 2014 +0530 cli: volume status for tcp,rdma type volume display only tcp port For tcp,rdma type voumes, there will be two ports, one for tcp and one for rdma. But volume status command only display tcp port. By this change, adding an extra column for rdma port and changing the port to tcp port. Eg: >gluster volume status pathy >For tcp,rdma type volume Status of volume: patchy Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick brickname 49152 49153 Y 14158 >For rdma type volume Status of volume: patchy Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick brickname 0 49153 Y 14158 For tcp type volume Status of volume: patchy Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick brickname 49152 0 Y 14158 >gluster volume status patchy detail Status of volume: xcube2 ------------------------------------------------------------------------------ Brick : Brick brickname TCP Port : 49152 RDMA Port : 49153 Online : Y Pid : 14158 File System : ext4 Device : /dev/mapper/luks-2099dd4a-0050-4cae-ad7b-c6a0498c4e88 Mount Options : rw,seclabel,relatime,data=ordered Inode Size : 256 Disk Space Free : 31.1GB Total Disk Space : 47.9GB Inode Count : 3203072 Free Inodes : 2926789 >gluster volume status xcube --xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput> <opRet>0</opRet> <opErrno>0</opErrno> <opErrstr>(null)</opErrstr> <volStatus> <volumes> <volume> <volName>xcube</volName> <nodeCount>2</nodeCount> <node> <hostname>hostname</hostname> <path>/home/brick1</path> <peerid>2d7bcb95-3d26-4d4f-b3c6-e2ee01b71662</peerid> <status>1</status> <port>49152</port> <ports> <tcp>49152</tcp> <rdma>N/A</rdma> </ports> <pid>5657</pid> </node> <node> <hostname>NFS Server</hostname> <path>localhost</path> <peerid>2d7bcb95-3d26-4d4f-b3c6-e2ee01b71662</peerid> <status>1</status> <port>2049</port> <ports> <tcp>2049</tcp> <rdma>N/A</rdma> </ports> <pid>5665</pid> </node> <tasks/> </volume> </volumes> </volStatus> </cliOutput> Change-Id: I81aab226edbd400d29cd3f510af4f344dd99ba51 BUG: 1164079 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/9191 Reviewed-by: Atin Mukherjee <amukherj> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Kaushal M <kaushal> REVIEW: http://review.gluster.org/9687 (rdma: free rdma priv data if init fails) posted (#1) for review on master by Atin Mukherjee (amukherj) COMMIT: http://review.gluster.org/9687 committed in master by Raghavendra G (rgowdapp) ------ commit bdb5ca2339edc76b81d1ff996d695eecd849760f Author: Atin Mukherjee <amukherj> Date: Wed Feb 18 22:50:34 2015 +0530 rdma: free rdma priv data if init fails Change-Id: I57b38c8783666e806836dacf3f74cf9f6876070a BUG: 1164079 Signed-off-by: Atin Mukherjee <amukherj> Reviewed-on: http://review.gluster.org/9687 Reviewed-by: mohammed rafi kc <rkavunga> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp> This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions Closing this bug as we have stopped the active development in upstream for the component. |