Bug 1329094

Summary: Default NFS settings kernel cause high load and low throughput on random write loads
Product: [Fedora] Fedora Reporter: D.S. Ljungmark <spider>
Component: kernelAssignee: nfs-maint
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 23CC: bfields, gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda, mchehab, spider
Target Milestone: ---Flags: spider: needinfo-
spider: needinfo-
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-26 16:51:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
fio result, 8k wsize
none
fio result, 1M wsize
none
nfs server dstat, 8k wsize
none
nfs server dstat, 1M wsize
none
nfs client dstat, 8k wsize
none
nfs client dstat, 1M wsize none

Description D.S. Ljungmark 2016-04-21 08:10:11 UTC 
Description of problem:
The default NFS settings expose wsize and rsize as 1MiB on both client and server.

This setting, on both TCP and UDP,  and protocol version 3 and 4, causes _severe_ throughput and latency on loads with small writes.





Version-Release number of selected component (if applicable):


How reproducible:

Export a filesystem over NFS
Run a small `fio` load on first the server (on the raw disk) and note the iowait, throughput (to disk) and similar stats with your favourite monitoring tool. I used `dstat`

The simplest `fio` load I've found was this:
fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=256M --numjobs=8 --runtime=60 --group_reporting


This appears to be related to some buffer management, as when you force down rsize/wsize to 8k, performance and interactivity goes up again.

Testing on our environment has showed the same behaviour towards older kernels, (CentOs 7 NFS host), as well as in a pure Fedora environments.


The main issues here are:
- Completely _abysmal_ throughput on both network and disk ( Writes in kilobytes/second, not even megabytes/second)
- Horridly large IOWait on both client and Server
- Awful interactivity on both clients and servers


Host Filesystem has been tested with both Ext4 and BTRFS and both show the same tendency. It's difficult to get a proper measure of how badly the load affects the server in order to compare if one filesystem is better than others.

Host devices tested has been both SSD raid10, and spinning metal, and while SSD's appear slightly better, it's still very noticable.


NFS settings:
( That work) 
172.27.1.8:/export/software /media/store/software nfs4 rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.27.1.3,local_lock=none,addr=172.27.1.8 0 0

Not forcinge rsize/Wsize down will cause it to be 1MiB and provoke the issue. 



Steps to Reproduce:
1. Mount NFS with default settings
2. Run dstat on client & server
3. run fio command from above on client


Actual results:
Extremely high loads, high IOWait and appearantly throughput stalls


Expected results:
Performance that saturates the network and disk, with higher latency than on the host server local load, but with comparative iowait/load numbers.


Additional info:
Reproduced on CentOS 7 host, Fedora 22 host (4.4.5 kernel) 
Fedora 23 clients,  Fedora 22 clients, CentOS6 and 7 clients
Comment 1 Josh Boyer 2016-04-21 11:11:16 UTC 
Perhaps the NFS maintainers will have more insight, but I suspect the answer will be "it's tunable from userspace so tune it to your workload because no default will ever be 100% correct for everyone."
Comment 2 J. Bruce Fields 2016-04-21 15:02:08 UTC 
Josh Boyer said:
> I suspect the answer will be "it's tunable from userspace so tune it to your
> workload because no default will ever be 100% correct for everyone."

I wonder why rsize/wsize makes a difference at all in this case, since the client's likely not able to make >4k writes anyway.  I'm more inclined to accept this as a bug.  Given that it's easily reproduceable on a variety of versions, I bet it's been noticed before, but I don't recall any discussion off the top of my head.

> "Reproduced on CentOS 7 host, Fedora 22 host (4.4.5 kernel) 
> Fedora 23 clients,  Fedora 22 clients, CentOS6 and 7 clients"

Sorry, by "host" here, do you mean "NFS server"?
Comment 3 D.S. Ljungmark 2016-04-21 23:59:24 UTC 
Yes, Sorry about the confusion of terms there.

Attaching some logs and results from two CentOS machines, ( I don't have the results of the Fedora ones on hand, sadly)  which would probably help.

In the logs, Acorn is the client, Omap is the server.
Comment 4 D.S. Ljungmark 2016-04-22 00:01:01 UTC 
Created attachment 1149609 [details]
fio result, 8k wsize
Comment 5 D.S. Ljungmark 2016-04-22 00:01:25 UTC 
Created attachment 1149610 [details]
fio result, 1M wsize
Comment 6 D.S. Ljungmark 2016-04-22 00:01:53 UTC 
Created attachment 1149611 [details]
nfs server dstat, 8k wsize
Comment 7 D.S. Ljungmark 2016-04-22 00:02:22 UTC 
Created attachment 1149612 [details]
nfs server dstat, 1M wsize
Comment 8 D.S. Ljungmark 2016-04-22 00:02:46 UTC 
Created attachment 1149613 [details]
nfs client dstat, 8k wsize
Comment 9 D.S. Ljungmark 2016-04-22 00:03:13 UTC 
Created attachment 1149616 [details]
nfs client dstat, 1M wsize
Comment 10 D.S. Ljungmark 2016-04-22 00:04:03 UTC 
Oh, both wsize and rsize were the same, sorry for not mentioning that in the attachments, my bad.
Comment 11 J. Bruce Fields 2016-04-29 19:39:43 UTC 
I don't really have time to troubleshoot this right now, just trying to think of suggestions for things you might try.

It might be interesting to look at /proc/self/mountstats (see also the "mountstats" command), and compare the rpc statistics for the "good" (8K w/rsize) vs "bad" (1M w/rsize) cases.
Comment 12 Laura Abbott 2016-09-23 19:39:28 UTC 
*********** MASS BUG UPDATE **************
 
We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 23 kernel bugs.
 
Fedora 23 has now been rebased to 4.7.4-100.fc23.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.
 
If you have moved on to Fedora 24 or 25, and are still experiencing this issue, please change the version to Fedora 24 or 25.
 
If you experience different issues, please open a new bug report for those.
Comment 13 Laura Abbott 2016-10-26 16:51:15 UTC 
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 4 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.