Bug 631012
| Summary: | Initscripts changes needed to support NFS over RDMA | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Steve Dickson <steved> |
| Component: | nfs-utils | Assignee: | Steve Dickson <steved> |
| Status: | CLOSED ERRATA | QA Contact: | yanfu,wang <yanwang> |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | 6.1 | CC: | lwang, tao, yanwang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | nfs-utils-1.2.3-1.el6 | Doc Type: | Bug Fix |
| Doc Text: |
Previously, mounting NFS over RDMA (remote direct memory access) failed due to missing code for such mounting in the NFS initiation script and the sysconfig file. This update adds the missing code and mounting of NFS over RDMA works correctly.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-05-19 14:16:46 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 453032 | ||
| Bug Blocks: | |||
|
Description
Steve Dickson
2010-09-07 15:33:17 UTC
This is the diff needed to enable RDMA support on the server
--- nfs.init 2010-05-26 09:32:00.327407000 -0400
+++ /tmp/nfs.init 2010-09-07 11:44:47.421450923 -0400
@@ -81,8 +81,9 @@ case "$1" in
fi
# Load the nfsd module so /proc/fs/nfsd will exist
- [ "$NFSD_MODULE" != "noload" ] && {
- [ -x /sbin/modprobe ] && /sbin/modprobe nfsd
+ [ "$NFSD_MODULE" != "noload" -a -x /sbin/modprobe ] && {
+ /sbin/modprobe nfsd
+ [ -n "$RDMA_PORT" ] && /sbin/modprobe svcrdma
}
action $"Starting NFS services: " /usr/sbin/exportfs -r
@@ -95,12 +96,15 @@ case "$1" in
echo
fi
+
echo -n $"Starting NFS daemon: "
daemon rpc.nfsd $RPCNFSDARGS $RPCNFSDCOUNT
RETVAL=$?
echo
[ $RETVAL -ne 0 ] && exit $RETVAL
+ [ -n "$RDMA_PORT" ] && echo "rdma $RDMA_PORT" > /proc/fs/nfsd/portlist
+
[ -n "$MOUNTD_PORT" ] \
&& RPCMOUNTDOPTS="$RPCMOUNTDOPTS -p $MOUNTD_PORT"
*** Bug 629724 has been marked as a duplicate of this bug. *** hi Gurhan, I change qe owner to you since you help test NFS over RDMA in RHEL6, thanks. hi, any update? hi Steve, Could you tell me how to test the nfs over RDMA function? thanks. ------- Comment From 2010-06-07 06:41 EDT------- Hello Redhat, I can see changes in nfs and rdma packages in RHEL6 Snap6. nfs-utils-1.2.2-2.el6.x86_64 -> nfs-utils-1.2.2-3.el6.x86_64 rdma-1.0-6.el6.noarch -> rdma-1.0-7.el6.noarch So, will update the latest results from it. Thanks Muni ------- Comment From 2010-06-09 01:01 EDT------- (In reply to comment #11) > Hello, > Please let me know the results of testing the new packages. > > Thank You > Joe Kachuck Same issue is happening on RHEL6 Snap6 as well. ~]# rpm -qa | grep nfs nfs-utils-1.2.2-3.el6.x86_64 nfs4-acl-tools-0.3.3-5.el6.x86_64 nfs-utils-lib-1.1.5-1.el6.x86_64 ~]# rpm -qa | grep rdma librdmacm-1.0.10-2.el6.x86_64 rdma-1.0-7.el6.noarch librdmacm-utils-1.0.10-2.el6.x86_64 ~]# echo "rdma 20049" > /proc/fs/nfsd/portlist -bash: echo: write error: Cannot assign requested address ~]# echo "rdma 20049" >> /proc/fs/nfsd/portlist -bash: echo: write error: Cannot assign requested address Thanks Muni ------- Comment From iranna.ankad.com 2010-06-09 01:51 EDT------- This bug is blocking us to verify NFS over RDMA feature in RHEL6 i.e Bug 48461 - [LTC 6.0 FEAT] 201227:NFS over RDMA support . Hence raising the severity to "Block" There's limited infiniband/rdma environment for us to test, I just have x86_64 arch server to test. My test result shown as below: nfs server: [root@rdma2 ~]# rpm -qa|grep nfs-utils nfs-utils-1.2.3-6.el6.x86_64 nfs-utils-lib-1.1.5-3.el6.x86_64 [root@rdma2 ~]# uname -a Linux rdma2.rhts.eng.bos.redhat.com 2.6.32-122.el6.x86_64 #1 SMP Wed Mar 9 23:54:34 EST 2011 x86_64 x86_64 x86_64 GNU/Linux [root@rdma2 ~]# showmount -e Export list for rdma2.rhts.eng.bos.redhat.com: /mnt/nfs 172.31.0.4 nfs client: [root@rdma4 ~]# mount -t nfs -o rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv mount: fstab path: "/etc/fstab" mount: mtab path: "/etc/mtab" mount: lock path: "/etc/mtab~" mount: temp path: "/etc/mtab.tmp" mount: UID: 0 mount: eUID: 0 mount: spec: "172.31.0.2:/mnt/nfs" mount: node: "/mnt" mount: types: "nfs" mount: opts: "rdma,port=20049" mount: external mount: argv[0] = "/sbin/mount.nfs" mount: external mount: argv[1] = "172.31.0.2:/mnt/nfs" mount: external mount: argv[2] = "/mnt" mount: external mount: argv[3] = "-v" mount: external mount: argv[4] = "-o" mount: external mount: argv[5] = "rw,rdma,port=20049" mount.nfs: timeout set for Sat Apr 2 01:55:16 2011 mount.nfs: trying text-based options 'rdma,port=20049,vers=4,addr=172.31.0.2,clientaddr=172.31.0.4' mount.nfs: mount(2): No such file or directory mount.nfs: trying text-based options 'rdma,port=20049,addr=172.31.0.2' 172.31.0.2:/mnt/nfs on /mnt type nfs (rw,rdma,port=20049) [root@rdma4 ~]# cat /proc/mounts ... 172.31.0.2:/mnt/nfs /mnt nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,mountaddr=172.31.0.2,mountvers=3,mountproto=tcp,local_lock=none,addr=172.31.0.2 0 0 [root@rdma4 ~]# ls -l /mnt 总用量 0 -rw-r--r--. 1 root root 0 4月 1 06:10 a [root@rdma4 ~]# touch /mnt/b [root@rdma4 ~]# ls -l /mnt 总用量 0 -rw-r--r--. 1 root root 0 4月 1 06:10 a -rw-r--r--. 1 root root 0 4月 2 01:53 b note: It shown that client use NFS/RDMA successfully, but when I use mount.nfs4 then got failed mount: [root@rdma4 ~]# mount -t nfs4 -o rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv mount: fstab path: "/etc/fstab" mount: mtab path: "/etc/mtab" mount: lock path: "/etc/mtab~" mount: temp path: "/etc/mtab.tmp" mount: UID: 0 mount: eUID: 0 mount: spec: "172.31.0.2:/mnt/nfs" mount: node: "/mnt" mount: types: "nfs4" mount: opts: "rdma,port=20049" mount: external mount: argv[0] = "/sbin/mount.nfs4" mount: external mount: argv[1] = "172.31.0.2:/mnt/nfs" mount: external mount: argv[2] = "/mnt" mount: external mount: argv[3] = "-v" mount: external mount: argv[4] = "-o" mount: external mount: argv[5] = "rw,rdma,port=20049" mount.nfs4: timeout set for Sat Apr 2 01:54:27 2011 mount.nfs4: trying text-based options 'rdma,port=20049,addr=172.31.0.2,clientaddr=172.31.0.4' mount.nfs4: mount(2): No such file or directory mount.nfs4: mounting 172.31.0.2:/mnt/nfs failed, reason given by server: No such file or directory [root@rdma4 ~]# dmesg|tail rpcrdma: connection to 172.31.0.2:20049 on qib0, memreg 6 slots 32 ird 4 SELinux: initialized (dev 0:15, type nfs4), uses genfs_contexts rpcrdma: connection to 172.31.0.2:20049 closed (-103) rpcrdma: connection to 172.31.0.2:20049 on qib0, memreg 6 slots 32 ird 4 SELinux: initialized (dev 0:15, type nfs4), uses genfs_contexts rpcrdma: connection to 172.31.0.2:20049 closed (-103) I known mount.nfs4 is a soft linked to mount.nfs, Steve, do you know why it failed? So to be clear, the mount works with v3 but not v4, correct? I wonder if it has something to do with how the server is exporting its exports.... (In reply to comment #12) > So to be clear, the mount works with v3 but not v4, correct? > > I wonder if it has something to do with how the server > is exporting its exports.... hi Steve, The export shown as below: /mnt/nfs 172.31.0.4(fsid=0,rw,async,insecure,no_root_squash) These are my test server and you can log in with root/redhat if you need to check, thanks. nfs server: rdma2.rhts.eng.bos.redhat.com nfs client: rdma4.rhts.eng.bos.redhat.com (In reply to comment #13) > (In reply to comment #12) > > So to be clear, the mount works with v3 but not v4, correct? > > > > I wonder if it has something to do with how the server > > is exporting its exports.... > > hi Steve, > The export shown as below: > /mnt/nfs 172.31.0.4(fsid=0,rw,async,insecure,no_root_squash) I removed the 'fsid=0' since its no longer needed and on the client side I took off the "proto=udp" since rdma is the protocol. The meant I got the following mounts to work: mount -t nfs -o v2,rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv mount -t nfs -o v3,rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv mount -t nfs -o v4,rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv I also updated rdma4's hosts file so a simple mount like this now works: mount -o rdma,port=20049 ib0rdma2:/mnt/nfs /mnt -vvv ------- Comment From iranna.ankad.com 2011-04-20 07:15 EDT------- Well..I could verify NFS over RDMA on RHEL6.1 snap3 and confirm that everything is fine. I could successfully nfs mount (both v4 & v3) over RDMA and run some file system I/O stress tests for nearly 24 hours, without any issues. FYI, Here is the setup I used: Server: [root@elm3a191 /]# uname -a Linux elm3a191.beaverton.ibm.com 2.6.32-130.el6.x86_64 #1 SMP Tue Apr 5 19:58:31 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux [root@elm3a191 /]# [root@elm3a191 /]# cat /etc/exports /NFS3 *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check) /NFS4 *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check) [root@elm3a191 /]# [root@elm3a191 /]# /etc/init.d/nfs restart Shutting down NFS mountd: [ OK ] Shutting down NFS daemon: [ OK ] Shutting down NFS quotas: [ OK ] Starting NFS services: [ OK ] Starting NFS quotas: [ OK ] Starting NFS daemon: [ OK ] Starting NFS mountd: [ OK ] [root@elm3a191 /]# exportfs /NFS3 <world> /NFS4 <world> [root@elm3a191 /]# /etc/init.d/nfs-rdma status [root@elm3a191 /]# /etc/init.d/nfs-rdma start Enabling NFSoRDMA support: [ OK ] [root@elm3a191 /]# Client: [root@elm3a216 ~]# uname -a Linux elm3a216.beaverton.ibm.com 2.6.32-130.el6.x86_64 #1 SMP Tue Apr 5 19:58:31 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux [root@elm3a216 ~]# [root@elm3a216 ~]# mount -t nfs 10.0.0.2:/NFS3 /NFS3 [root@elm3a216 ~]# mount -t nfs4 10.0.0.4:/NFS4 /NFS4 [root@elm3a216 fsstress]# mount /dev/sda7 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) 10.0.0.2:/NFS3 on /NFS3 type nfs (rw,vers=4,addr=10.0.0.2,clientaddr=10.0.0.3) 10.0.0.4:/NFS4 on /NFS4 type nfs4 (rw,addr=10.0.0.4,clientaddr=10.0.0.3) [root@elm3a216 fsstress]# [root@elm3a216 fsstress]# jobs [1]- Running nohup ./fsstress -d /NFS3 -l 0 -n 100 -p 100 -r & [2]+ Running nohup ./fsstress -d /NFS4 -l 0 -n 100 -p 100 -r & [root@elm3a216 fsstress]# Result Analysis: Monitored netstat, nfsiostat, df, top, dmesg & /var/log/messages. No issues are found! [root@elm3a216 ~]# netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 999233 0 0 0 226927 0 0 0 BMRU ib0 2044 0 37 0 0 0 0 0 7 0 BMRU ib1 2044 0 124256550 0 0 0 118938632 0 7 0 BMRU lo 16436 0 9668 0 0 0 9668 0 0 0 LRU [root@elm3a216 ~]# nfsiostat 10.0.0.4:/NFS4/ mounted on /NFS4: op/s rpc bklog 476.41 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.483 27.639 57.236 0 (0.0%) 6.008 6.109 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 6.908 464.651 67.263 0 (0.0%) 7.546 7.642 10.0.0.2:/NFS3/ mounted on /NFS3: op/s rpc bklog 489.80 0.00 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 0.483 41.278 85.475 0 (0.0%) 3.901 3.997 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 8.461 558.072 65.956 6 (0.0%) 7.152 7.246 [root@elm3a216 ~]# (In reply to comment #16) > ------- Comment From iranna.ankad.com 2011-04-20 07:15 EDT------- > Well..I could verify NFS over RDMA on RHEL6.1 snap3 and confirm that everything > is fine. > I could successfully nfs mount (both v4 & v3) over RDMA and run some file > system I/O stress tests for nearly 24 hours, without any issues. > > FYI, Here is the setup I used: > > Server: > > [root@elm3a191 /]# uname -a > Linux elm3a191.beaverton.ibm.com 2.6.32-130.el6.x86_64 #1 SMP Tue Apr 5 > 19:58:31 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux > [root@elm3a191 /]# > > [root@elm3a191 /]# cat /etc/exports > /NFS3 *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check) > /NFS4 *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check) > [root@elm3a191 /]# > [root@elm3a191 /]# /etc/init.d/nfs restart > Shutting down NFS mountd: [ OK ] > Shutting down NFS daemon: [ OK ] > Shutting down NFS quotas: [ OK ] > Starting NFS services: [ OK ] > Starting NFS quotas: [ OK ] > Starting NFS daemon: [ OK ] > Starting NFS mountd: [ OK ] > [root@elm3a191 /]# exportfs > /NFS3 <world> > /NFS4 <world> > [root@elm3a191 /]# /etc/init.d/nfs-rdma status > [root@elm3a191 /]# /etc/init.d/nfs-rdma start > Enabling NFSoRDMA support: [ OK ] > [root@elm3a191 /]# Just curious... where did the 'nfs-rdma' initscript come from? The changes I made were to the 'nfs' initscript and the /etc/sysconf/nfs config file.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Previously, mounting NFS over RDMA (remote direct memory access) failed due to missing code for such mounting in the NFS initiation script and the sysconfig file. This update adds the missing code and mounting of NFS over RDMA works correctly.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-0738.html |