Bug 241600

Created attachment 155555 [details]
Kernel 2.6.9-42.0.10 performance on nfs

Created attachment 155556 [details]
Kernel 2.6.9-55 performance on nfs

There were some substantial changes made to fix problems with cache coherency
between multiple clients in 4.5. Earlier kernels often didnt invalidate the
cache when they should have, so that may account for some of the difference
you're seeing here. If you're able, could you test against the kernels on my
people page?

http://people.redhat.com/jlayton

and post the results here? They have some patches that may help NFS performance.

Same thing. Another patch or test?

# uname -r
2.6.9-55.4.EL.jtltest.9smp
# iozone -i 0 -i 1 -r 32 -s 256m -t 2 -F 1 2
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.279 $
		Compiled for 32 bit mode.
		Build: linux 
[...]
	Record Size 32 KB
	File size set to 262144 KB
	Command line used: iozone -i 0 -i 1 -r 32 -s 256m -t 2 -F 1 2
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 2 processes
	Each process writes a 262144 Kbyte file in 32 Kbyte records
[...]
	Children see throughput for  2 readers 		=   44300.36 KB/sec
	Parent sees throughput for  2 readers 		=   44294.51 KB/sec
	Min throughput per process 			=   22140.38 KB/sec 
	Max throughput per process 			=   22159.97 KB/sec
	Avg throughput per process 			=   22150.18 KB/sec
	Min xfer 					=  261920.00 KB
[...]
#

I've tried reproducing this here and am not seeing the same degree of
performance degradation that you are. The first 3 passes of that test show
roughly equivalent performance. The last test *does* show a more significant
difference, but I'd sort of expect that since earlier kernels didn't invalidate
the cache when it should have (I'll look closer at that to see if I can verify
what's happening there).

That said, my test rig doesn't have the throughput that yours evidently does. I
can probably get time on one, but it may take me a little while before I can get it.

There's a lot going on in this test, so I think we need to narrow down where the
problem is. I'd be interested to see if you can reproduce this with a simpler
run. Maybe one that just does the initial writing of the file from a single
thread. Do you see the performance differential with that?

Also, when doing these tests, it would be good to also collect "nfsstat -a"
output afterward so we can see whether we have a spike in the number of a
particular call. Unfortunately there's not an easy way to zero out those stats,
so you'll want to reboot, do the test (with with no other traffic to the mount
from the client) and then collect nfsstat afterward as soon as possible.

For the record, on my test mounting a Linux host with default options. Only
posting the results of the last test since the first tests were already pretty
close:

-42.0.10.EL:

        Children see throughput for 2 re-readers        =   21416.05 KB/sec
        Parent sees throughput for 2 re-readers         =   21327.33 KB/sec
        Min throughput per process                      =    4756.18 KB/sec 
        Max throughput per process                      =   16659.87 KB/sec
        Avg throughput per process                      =   10708.02 KB/sec
        Min xfer                                        =   75200.00 KB

Client nfs v3:
null         getattr      setattr      lookup       access       readlink     
0         0% 22        0% 0         0% 9         0% 16        0% 0         0% 
read         write        create       mkdir        symlink      mknod        
22360    40% 32073    58% 2         0% 0         0% 0         0% 0         0% 
remove       rmdir        rename       link         readdir      readdirplus  
2         0% 0         0% 0         0% 0         0% 0         0% 9         0% 
fsstat       fsinfo       pathconf     commit       
0         0% 1         0% 0         0% 436       0% 

-55.EL:

        Children see throughput for 2 re-readers        =   12078.67 KB/sec
        Parent sees throughput for 2 re-readers         =   12065.85 KB/sec
        Min throughput per process                      =    5420.79 KB/sec 
        Max throughput per process                      =    6657.89 KB/sec
        Avg throughput per process                      =    6039.34 KB/sec
        Min xfer                                        =  213440.00 KB

Client nfs v3:
null         getattr      setattr      lookup       access       readlink     
0         0% 23        0% 0         0% 7         0% 17        0% 0         0% 
read         write        create       mkdir        symlink      mknod        
30152    48% 31385    50% 2         0% 0         0% 0         0% 0         0% 
remove       rmdir        rename       link         readdir      readdirplus  
2         0% 0         0% 0         0% 0         0% 0         0% 9         0% 
fsstat       fsinfo       pathconf     commit       
0         0% 1         0% 0         0% 64        0% 

...so we see a rather large jump in read calls on the newer kernel with this
test, but that doesn't really surprise me since the -55 kernel will invalidate
the cache in more situations than the -42.0.10 kernel did.

Comment on attachment 155555 [details]
Kernel 2.6.9-42.0.10 performance on nfs

# nfsstat -a
Client packet stats:
packets    udp	      tcp	 tcpconn
0	   0	      0 	 0	 
Client rpc stats:
calls	   retrans    authrefrsh
6	   0	      0       
Client nfs v2:
null	   getattr    setattr	 root	    lookup     readlink   
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
read	   wrcache    write	 create     remove     rename	  
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
link	   symlink    mkdir	 rmdir	    readdir    fsstat	  
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 

Client nfs v3:
null	   getattr    setattr	 lookup     access     readlink   
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
read	   write      create	 mkdir	    symlink    mknod	  
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
remove	   rmdir      rename	 link	    readdir    readdirplus
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
fsstat	   fsinfo     pathconf	 commit     
0	0% 6	  100% 0       0% 0	  0% 

# uname -r
2.6.9-42.0.10.ELsmp
# cd /nfs/mail01ns07 ; iozone -i 0 -i 1 -r 32 -s 256m -t 1 -F 1
	Iozone: Performance Test of File I/O
		Version $Revision: 3.279 $
		Compiled for 32 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
		     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
		     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
		     Randy Dunlap, Mark Montague, Dan Million, 
		     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
		     Erik Habbinga, Kris Strecker, Walter Wong.

	Run began: Fri Jun  1 10:35:07 2007

	Record Size 32 KB
	File size set to 262144 KB
	Command line used: iozone -i 0 -i 1 -r 32 -s 256m -t 1 -F 1
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 1 process
	Each process writes a 262144 Kbyte file in 32 Kbyte records

	Children see throughput for  1 initial writers	=  575368.06 KB/sec
	Parent sees throughput for  1 initial writers	=   56815.93 KB/sec
	Min throughput per process			=  575368.06 KB/sec 
	Max throughput per process			=  575368.06 KB/sec
	Avg throughput per process			=  575368.06 KB/sec
	Min xfer					=  262144.00 KB

	Children see throughput for  1 rewriters	=  607398.19 KB/sec
	Parent sees throughput for  1 rewriters 	=   63355.92 KB/sec
	Min throughput per process			=  607398.19 KB/sec 
	Max throughput per process			=  607398.19 KB/sec
	Avg throughput per process			=  607398.19 KB/sec
	Min xfer					=  262144.00 KB

	Children see throughput for  1 readers		= 1184595.88 KB/sec
	Parent sees throughput for  1 readers		= 1182239.54 KB/sec
	Min throughput per process			= 1184595.88 KB/sec 
	Max throughput per process			= 1184595.88 KB/sec
	Avg throughput per process			= 1184595.88 KB/sec
	Min xfer					=  262144.00 KB

	Children see throughput for 1 re-readers	= 1198585.88 KB/sec
	Parent sees throughput for 1 re-readers 	= 1186440.93 KB/sec
	Min throughput per process			= 1198585.88 KB/sec 
	Max throughput per process			= 1198585.88 KB/sec
	Avg throughput per process			= 1198585.88 KB/sec
	Min xfer					=  262144.00 KB



iozone test complete.
# nfsstat -a
Client packet stats:
packets    udp	      tcp	 tcpconn
0	   0	      0 	 0	 
Client rpc stats:
calls	   retrans    authrefrsh
16411	   0	      0       
Client nfs v2:
null	   getattr    setattr	 root	    lookup     readlink   
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
read	   wrcache    write	 create     remove     rename	  
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
link	   symlink    mkdir	 rmdir	    readdir    fsstat	  
0	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 

Client nfs v3:
null	   getattr    setattr	 lookup     access     readlink   
0	0% 9	   0% 0       0% 4	 0% 4	    0% 0       0% 
read	   write      create	 mkdir	    symlink    mknod	  
0	0% 16384  99% 1       0% 0	 0% 0	    0% 0       0% 
remove	   rmdir      rename	 link	    readdir    readdirplus
1	0% 0	   0% 0       0% 0	 0% 0	    0% 0       0% 
fsstat	   fsinfo     pathconf	 commit     
0	0% 6	   0% 0       0% 2	 0%

Created attachment 155877 [details]
Nfs tests with 2.6.9-42.0.10.EL (v2)

Created attachment 155878 [details]
Nfs tests with 2.6.9-55.4.EL.jtltest.9smp (v2)

New tests attached, now i test with 1 thread only. One think that i see is:

* New kernel
read       write
8194   33% 16390  66%

* Old kernel
read       write
0       0% 16384  99%

Again, that test makes it very difficult to see what's happening. We see some
extra reads, but in which phase of the test were they incurred? We can't tell
from this. I'd like to focus for now on the first phase of the test (initial write).

One thing that would help is specifying -i 0 only (without -i 1) so we get rid
of the read/reread test for now. Since the "initial writers" test shows a
performance delta in your setup, I'd be interested to see if we're incurring
extra read calls with the new kernel with just an -i 0 test.

Unfortunately it looks like -i0 still does the "rewrite" test. It would be ideal
to just do the "initial write" test, but I'm not sure how to do that with iozone.

Also, could you let me know exactly what mount options you're using and some
info about what sort of server this is?

Well, after you say about more invalidade caches i test performance with another
tool, a personal tool that simulate email IO in nfs server. My nfs server is one
EMC NS700 and i can see same performance with the new kernel and the old kernel.
I test only read yet and i'll test write too later.

My mount options are:
rw,noexec,nfsvers=3,proto=tcp,intr,nolock,rsize=32768,wsize=32768

Now i suspect that's not a bug, and what i see are old performance because
caches in iozone tests.

I'll test more...

Thanks, let me know what you find...

While I'd expect a bit of performance degradation in some situations with the
new kernel, it's certainly possible that there are performance impacting bugs in
there as well. The trick is to narrow down the reproducer as much as possible so
that we can definitively figure out why one is slower than the other for a
particular test.

Setting to NEEDINFO reporter to await Marcus' testing...

No response to this for some time. Closing as NOTABUG. Please reopen if you have
further info.