Bug 1046011

Summary: NFS mount over RDMA cannot read more than 812 bytes
Product: [Fedora] Fedora Reporter: Markus Stockhausen <mst>
Component: kernelAssignee: Jeff Layton <jlayton>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: bfields, chuck.lever, gansalmon, itamar, jlayton, jonathan, kernel-maint, madhu.chinakonda, nfs-maint, steved
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1048477 (view as bug list) Environment:
Last Closed: 2014-04-24 14:29:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1048477    

Description Markus Stockhausen 2013-12-23 08:00:06 UTC
Description of problem:

When mount NFS shares using RDMA protocol the client cannot read blocks that are larger than 812 bytes. 

Version-Release number of selected component (if applicable):
Kernel 3.11.10-200.fc19.x86_64

How reproducible:

Steps to Reproduce:
1) Create NFS RDMA server (as explained in the kernel doc)
# modprobe svcrdma
# service nfs start
# echo rdma 20049 > /proc/fs/nfsd/portlist

2) Create some export
# exportfs -a

3) Configure NFS RDMA client
# modprobe xprtrdma

4) mount filesystem on client
# mount -o rdma,port=20049,nfsvers=4 /mnt

5) check for rdma mount on client
# mount
... on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none,addr=

6) create file on nfs server with 812 bytes
dd if=/dev/urandom of=bla.txt bs=812 count=1

7) read file on client (works fine)
cat bla.txt > /dev/null

8) create file on nfs server with 813 bytes
dd if=/dev/urandom of=bla.txt bs=813 count=1

9) read file on client (gives error - german text for I/O error)
# cat bla.txt > /dev/null
cat: bla.txt: Eingabe-/Ausgabefehler

Actual results:
file not readable

Expected results:
no error should occur

Additional info:
Similar problem discussed on the NFS mailing list http://www.spinics.net/lists/linux-nfs/msg40253.html

Comment 1 Markus Stockhausen 2013-12-23 08:03:01 UTC
I'm using Mellanox ConnectX Infiniband cards with driver mlx4_ib. Fedora 19 is fully patched on both nodes.

Comment 2 Markus Stockhausen 2013-12-23 17:05:03 UTC
If I get the call stack the right way in net/sunrpc/xprtrdma/rpc_rdma.c:

1) rpcrdma_reply_handler() calls rpcrdma_inline_fixup()

2) rpcrdma_inline_fixup() executes rqst->rq_rcv_buf.page_len = 0;

activating rpc debugging gives the following log 
RPC: rpcrdma_inline_fixup: srcp 0xffff8805eb1c4094 len 60 hdrlen 60

Kernel commit http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a11a2bf4de5679fa0b63474c7d39bea2dac7d061 changes the logic that net/sunrpc/xdr.c works with page_len value. The datalen is reduced to page_len even if page_len is zero.

Comment 3 Markus Stockhausen 2013-12-23 17:43:14 UTC
I cannot confirm it but this might be an issue for RHEL 7 Beta too.

Comment 4 Jeff Layton 2014-01-20 17:34:58 UTC
Looks like Chuck has a patch for this. I'll plan to test it out soon.

Comment 5 Jeff Layton 2014-01-21 20:28:19 UTC
...and Chuck's patch works. Looks like he ended up fixing the same spot that Markus noticed. See:


I imagine this'll get merged for 3.14 and marked for stable.

Comment 6 Jeff Layton 2014-04-24 14:29:43 UTC
Oof, didn't make 3.14, but did get merged for v3.15. I'll close with a resolution of RAWHIDE.