RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 631012 - Initscripts changes needed to support NFS over RDMA
Summary: Initscripts changes needed to support NFS over RDMA
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: nfs-utils
Version: 6.1
Hardware: All
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: Steve Dickson
QA Contact: yanfu,wang
URL:
Whiteboard:
: 629724 (view as bug list)
Depends On: 453032
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-09-07 15:33 UTC by Steve Dickson
Modified: 2018-11-14 15:27 UTC (History)
3 users (show)

Fixed In Version: nfs-utils-1.2.3-1.el6
Doc Type: Bug Fix
Doc Text:
Previously, mounting NFS over RDMA (remote direct memory access) failed due to missing code for such mounting in the NFS initiation script and the sysconfig file. This update adds the missing code and mounting of NFS over RDMA works correctly.
Clone Of:
Environment:
Last Closed: 2011-05-19 14:16:46 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0738 0 normal SHIPPED_LIVE nfs-utils bug fix and enhancement update 2011-05-18 18:09:16 UTC

Description Steve Dickson 2010-09-07 15:33:17 UTC 
Description of problem:
A few changes to the nfs initscript and the nfs sysconfig
file are needed to enable RDMA support on both the client
and server.
Comment 2 Steve Dickson 2010-09-07 15:45:27 UTC 
This is the diff needed to enable RDMA support on the server

--- nfs.init	2010-05-26 09:32:00.327407000 -0400
+++ /tmp/nfs.init	2010-09-07 11:44:47.421450923 -0400
@@ -81,8 +81,9 @@ case "$1" in
 	fi
 
 	# Load the nfsd module so /proc/fs/nfsd will exist
-	[ "$NFSD_MODULE" != "noload" ] && {
-		[ -x /sbin/modprobe ] && /sbin/modprobe nfsd
+	[ "$NFSD_MODULE" != "noload" -a -x /sbin/modprobe ] && {
+		/sbin/modprobe nfsd
+		[ -n "$RDMA_PORT" ] && /sbin/modprobe svcrdma
 	}
 	action $"Starting NFS services: " /usr/sbin/exportfs -r
 
@@ -95,12 +96,15 @@ case "$1" in
 	    echo
 	fi
 
+
 	echo -n $"Starting NFS daemon: "
 	daemon rpc.nfsd $RPCNFSDARGS $RPCNFSDCOUNT
 	RETVAL=$?
 	echo
 	[ $RETVAL -ne 0 ] && exit $RETVAL
 
+	[ -n "$RDMA_PORT" ] && echo "rdma $RDMA_PORT" > /proc/fs/nfsd/portlist
+
 	[ -n "$MOUNTD_PORT" ] \
 	&& RPCMOUNTDOPTS="$RPCMOUNTDOPTS -p $MOUNTD_PORT"
Comment 3 Steve Dickson 2011-01-13 21:45:24 UTC 
*** Bug 629724 has been marked as a duplicate of this bug. ***
Comment 7 yanfu,wang 2011-02-16 09:04:57 UTC 
hi Gurhan,
I change qe owner to you since you help test NFS over RDMA in RHEL6, thanks.
Comment 8 yanfu,wang 2011-03-14 05:44:55 UTC 
hi,
any update?
Comment 9 yanfu,wang 2011-03-23 05:28:55 UTC 
hi Steve,
Could you tell me how to test the nfs over RDMA function? thanks.
Comment 10 IBM Bug Proxy 2011-03-26 14:34:01 UTC 
------- Comment From  2010-06-07 06:41 EDT-------
Hello Redhat,

I can see changes in nfs and rdma packages in RHEL6 Snap6.

nfs-utils-1.2.2-2.el6.x86_64 -> nfs-utils-1.2.2-3.el6.x86_64
rdma-1.0-6.el6.noarch -> rdma-1.0-7.el6.noarch

So, will update the latest results from it.

Thanks
Muni

------- Comment From  2010-06-09 01:01 EDT-------
(In reply to comment #11)
> Hello,
> Please let me know the results of testing the new packages.
>
> Thank You
> Joe Kachuck

Same issue is happening on RHEL6 Snap6 as well.

~]# rpm -qa | grep nfs
nfs-utils-1.2.2-3.el6.x86_64
nfs4-acl-tools-0.3.3-5.el6.x86_64
nfs-utils-lib-1.1.5-1.el6.x86_64

~]# rpm -qa | grep rdma
librdmacm-1.0.10-2.el6.x86_64
rdma-1.0-7.el6.noarch
librdmacm-utils-1.0.10-2.el6.x86_64

~]# echo "rdma 20049" > /proc/fs/nfsd/portlist
-bash: echo: write error: Cannot assign requested address
~]# echo "rdma 20049" >> /proc/fs/nfsd/portlist
-bash: echo: write error: Cannot assign requested address

Thanks
Muni

------- Comment From iranna.ankad.com 2010-06-09 01:51 EDT-------
This bug is blocking us to verify NFS over RDMA feature in RHEL6
i.e Bug 48461   -  [LTC 6.0 FEAT] 201227:NFS over RDMA support .

Hence raising the severity to "Block"
Comment 11 yanfu,wang 2011-04-02 06:17:52 UTC 
There's limited infiniband/rdma environment for us to test, I just have x86_64 arch server to test. My test result shown as below:
nfs server:
[root@rdma2 ~]# rpm -qa|grep nfs-utils
nfs-utils-1.2.3-6.el6.x86_64
nfs-utils-lib-1.1.5-3.el6.x86_64
[root@rdma2 ~]# uname -a 
Linux rdma2.rhts.eng.bos.redhat.com 2.6.32-122.el6.x86_64 #1 SMP Wed Mar 9 23:54:34 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@rdma2 ~]# showmount -e 
Export list for rdma2.rhts.eng.bos.redhat.com:
/mnt/nfs 172.31.0.4

nfs client:
[root@rdma4 ~]# mount -t nfs -o rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "172.31.0.2:/mnt/nfs"
mount: node:  "/mnt"
mount: types: "nfs"
mount: opts:  "rdma,port=20049"
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = "172.31.0.2:/mnt/nfs"
mount: external mount: argv[2] = "/mnt"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,rdma,port=20049"
mount.nfs: timeout set for Sat Apr  2 01:55:16 2011
mount.nfs: trying text-based options 'rdma,port=20049,vers=4,addr=172.31.0.2,clientaddr=172.31.0.4'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'rdma,port=20049,addr=172.31.0.2'
172.31.0.2:/mnt/nfs on /mnt type nfs (rw,rdma,port=20049)
[root@rdma4 ~]# cat /proc/mounts 
...
172.31.0.2:/mnt/nfs /mnt nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,mountaddr=172.31.0.2,mountvers=3,mountproto=tcp,local_lock=none,addr=172.31.0.2 0 0
[root@rdma4 ~]# ls -l /mnt
总用量 0
-rw-r--r--. 1 root root 0  4月  1 06:10 a
[root@rdma4 ~]# touch /mnt/b
[root@rdma4 ~]# ls -l /mnt
总用量 0
-rw-r--r--. 1 root root 0  4月  1 06:10 a
-rw-r--r--. 1 root root 0  4月  2 01:53 b


note: It shown that client use NFS/RDMA successfully, but when I use mount.nfs4 then got failed mount:
[root@rdma4 ~]# mount -t nfs4 -o rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "172.31.0.2:/mnt/nfs"
mount: node:  "/mnt"
mount: types: "nfs4"
mount: opts:  "rdma,port=20049"
mount: external mount: argv[0] = "/sbin/mount.nfs4"
mount: external mount: argv[1] = "172.31.0.2:/mnt/nfs"
mount: external mount: argv[2] = "/mnt"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,rdma,port=20049"
mount.nfs4: timeout set for Sat Apr  2 01:54:27 2011
mount.nfs4: trying text-based options 'rdma,port=20049,addr=172.31.0.2,clientaddr=172.31.0.4'
mount.nfs4: mount(2): No such file or directory
mount.nfs4: mounting 172.31.0.2:/mnt/nfs failed, reason given by server:
  No such file or directory

[root@rdma4 ~]# dmesg|tail
rpcrdma: connection to 172.31.0.2:20049 on qib0, memreg 6 slots 32 ird 4
SELinux: initialized (dev 0:15, type nfs4), uses genfs_contexts
rpcrdma: connection to 172.31.0.2:20049 closed (-103)
rpcrdma: connection to 172.31.0.2:20049 on qib0, memreg 6 slots 32 ird 4
SELinux: initialized (dev 0:15, type nfs4), uses genfs_contexts
rpcrdma: connection to 172.31.0.2:20049 closed (-103)

I known mount.nfs4 is a soft linked to mount.nfs, Steve, do you know why it failed?
Comment 12 Steve Dickson 2011-04-05 11:20:29 UTC 
So to be clear, the mount works with v3 but not v4, correct?

I wonder if it has something to do with how the server
is exporting its exports....
Comment 13 yanfu,wang 2011-04-06 06:48:24 UTC 
(In reply to comment #12)
> So to be clear, the mount works with v3 but not v4, correct?
> 
> I wonder if it has something to do with how the server
> is exporting its exports....

hi Steve,
The export shown as below:
/mnt/nfs 172.31.0.4(fsid=0,rw,async,insecure,no_root_squash)

These are my test server and you can log in with root/redhat if you need to check, thanks.
nfs server: rdma2.rhts.eng.bos.redhat.com
nfs client: rdma4.rhts.eng.bos.redhat.com
Comment 14 Steve Dickson 2011-04-07 18:56:10 UTC 
(In reply to comment #13)
> (In reply to comment #12)
> > So to be clear, the mount works with v3 but not v4, correct?
> > 
> > I wonder if it has something to do with how the server
> > is exporting its exports....
> 
> hi Steve,
> The export shown as below:
> /mnt/nfs 172.31.0.4(fsid=0,rw,async,insecure,no_root_squash)
I removed the 'fsid=0' since its no longer needed 

and on the client side I took off the "proto=udp" since 
rdma is the protocol. The meant I got the following mounts
to work:

mount -t nfs -o v2,rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv
mount -t nfs -o v3,rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv
mount -t nfs -o v4,rdma,port=20049 172.31.0.2:/mnt/nfs /mnt -vvv

I also updated rdma4's hosts file so a simple mount like this 
now works:
     mount -o rdma,port=20049 ib0rdma2:/mnt/nfs /mnt -vvv
Comment 16 IBM Bug Proxy 2011-04-20 11:21:01 UTC 
------- Comment From iranna.ankad.com 2011-04-20 07:15 EDT-------
Well..I could verify NFS over RDMA on RHEL6.1 snap3 and confirm that everything is fine.
I could successfully nfs mount (both v4 & v3) over RDMA and run some file system I/O stress tests for nearly 24 hours, without any issues.

FYI, Here is the setup I used:

Server:

[root@elm3a191 /]# uname -a
Linux elm3a191.beaverton.ibm.com 2.6.32-130.el6.x86_64 #1 SMP Tue Apr 5 19:58:31 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@elm3a191 /]#

[root@elm3a191 /]# cat /etc/exports
/NFS3   *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check)
/NFS4   *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check)
[root@elm3a191 /]#
[root@elm3a191 /]# /etc/init.d/nfs restart
Shutting down NFS mountd:                                  [  OK  ]
Shutting down NFS daemon:                                  [  OK  ]
Shutting down NFS quotas:                                  [  OK  ]
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
[root@elm3a191 /]# exportfs
/NFS3           <world>
/NFS4           <world>
[root@elm3a191 /]# /etc/init.d/nfs-rdma status
[root@elm3a191 /]# /etc/init.d/nfs-rdma start
Enabling NFSoRDMA support:                                 [  OK  ]
[root@elm3a191 /]#

Client:

[root@elm3a216 ~]# uname -a
Linux elm3a216.beaverton.ibm.com 2.6.32-130.el6.x86_64 #1 SMP Tue Apr 5 19:58:31 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@elm3a216 ~]#

[root@elm3a216 ~]# mount -t nfs 10.0.0.2:/NFS3 /NFS3
[root@elm3a216 ~]# mount -t nfs4 10.0.0.4:/NFS4 /NFS4

[root@elm3a216 fsstress]# mount
/dev/sda7 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
10.0.0.2:/NFS3 on /NFS3 type nfs (rw,vers=4,addr=10.0.0.2,clientaddr=10.0.0.3)
10.0.0.4:/NFS4 on /NFS4 type nfs4 (rw,addr=10.0.0.4,clientaddr=10.0.0.3)
[root@elm3a216 fsstress]#

[root@elm3a216 fsstress]# jobs
[1]-  Running                 nohup ./fsstress -d /NFS3 -l 0 -n 100 -p 100 -r &
[2]+  Running                 nohup ./fsstress -d /NFS4 -l 0 -n 100 -p 100 -r &
[root@elm3a216 fsstress]#

Result Analysis:
Monitored netstat, nfsiostat, df, top, dmesg & /var/log/messages. No issues are found!

[root@elm3a216 ~]# netstat -i
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0   999233      0      0      0   226927      0      0      0 BMRU
ib0        2044   0       37      0      0      0        0      0      7      0 BMRU
ib1        2044   0 124256550      0      0      0 118938632      0      7      0 BMRU
lo        16436   0     9668      0      0      0     9668      0      0      0 LRU
[root@elm3a216 ~]# nfsiostat

10.0.0.4:/NFS4/ mounted on /NFS4:

op/s		rpc bklog
476.41 	   0.00
read:             ops/s		   kB/s		  kB/op		retrans		avg RTT (ms)	avg exe (ms)
0.483 	 27.639 	 57.236        0 (0.0%) 	  6.008 	  6.109
write:            ops/s		   kB/s		  kB/op		retrans		avg RTT (ms)	avg exe (ms)
6.908 	464.651 	 67.263        0 (0.0%) 	  7.546 	  7.642

10.0.0.2:/NFS3/ mounted on /NFS3:

op/s		rpc bklog
489.80 	   0.00
read:             ops/s		   kB/s		  kB/op		retrans		avg RTT (ms)	avg exe (ms)
0.483 	 41.278 	 85.475        0 (0.0%) 	  3.901 	  3.997
write:            ops/s		   kB/s		  kB/op		retrans		avg RTT (ms)	avg exe (ms)
8.461 	558.072 	 65.956        6 (0.0%) 	  7.152 	  7.246
[root@elm3a216 ~]#
Comment 17 Steve Dickson 2011-05-03 13:01:09 UTC 
(In reply to comment #16)
> ------- Comment From iranna.ankad.com 2011-04-20 07:15 EDT-------
> Well..I could verify NFS over RDMA on RHEL6.1 snap3 and confirm that everything
> is fine.
> I could successfully nfs mount (both v4 & v3) over RDMA and run some file
> system I/O stress tests for nearly 24 hours, without any issues.
> 
> FYI, Here is the setup I used:
> 
> Server:
> 
> [root@elm3a191 /]# uname -a
> Linux elm3a191.beaverton.ibm.com 2.6.32-130.el6.x86_64 #1 SMP Tue Apr 5
> 19:58:31 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> [root@elm3a191 /]#
> 
> [root@elm3a191 /]# cat /etc/exports
> /NFS3   *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check)
> /NFS4   *(rw,insecure,no_root_squash,sync,anonuid=0,no_subtree_check)
> [root@elm3a191 /]#
> [root@elm3a191 /]# /etc/init.d/nfs restart
> Shutting down NFS mountd:                                  [  OK  ]
> Shutting down NFS daemon:                                  [  OK  ]
> Shutting down NFS quotas:                                  [  OK  ]
> Starting NFS services:                                     [  OK  ]
> Starting NFS quotas:                                       [  OK  ]
> Starting NFS daemon:                                       [  OK  ]
> Starting NFS mountd:                                       [  OK  ]
> [root@elm3a191 /]# exportfs
> /NFS3           <world>
> /NFS4           <world>
> [root@elm3a191 /]# /etc/init.d/nfs-rdma status
> [root@elm3a191 /]# /etc/init.d/nfs-rdma start
> Enabling NFSoRDMA support:                                 [  OK  ]
> [root@elm3a191 /]#

Just curious...  where did the 'nfs-rdma' initscript come from? The 
changes I made were to the 'nfs' initscript and the /etc/sysconf/nfs
config file.
Comment 18 Eva Kopalova 2011-05-12 15:43:17 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, mounting NFS over RDMA (remote direct memory access) failed due to missing code for such mounting in the NFS initiation script and the sysconfig file. This update adds the missing code and mounting of NFS over RDMA works correctly.
Comment 19 errata-xmlrpc 2011-05-19 14:16:46 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0738.html
Note You need to log in before you can comment on or make changes to this bug.