Bug 504647

Summary: Low UDP throughput from Linux KVM guest
Product: Red Hat Enterprise Linux 5 Reporter: Mark Wagner <mwagner>
Component: kvmAssignee: Mark McLoughlin <markmc>
Status: CLOSED ERRATA QA Contact: Lawrence Lim <llim>
Severity: medium Docs Contact:
Priority: low    
Version: 5.4CC: cpelland, dshaks, herbert.xu, markmc, mwagner, shuang, tburke, tools-bugs, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: kvm-83-94.el5 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 09:35:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 513501    
Attachments:
Description Flags
Output of oprofile run with 581 byte message
none
oprofile output for a message size of 582 bytes
none
virtio_net-remove-the-tx-mitigation-timer.patch none

Description Mark Wagner 2009-06-08 16:48:58 UTC
Description of problem:
When running a RHEL5.3 or RHEL54 stream (-151) guest, the UDP performance as measured via netperf is very low with message sizes greater than 581 bytes.

Version-Release number of selected component (if applicable):
I have seen this behavior with multiple combinations of RHEL5.3 and RHEL5.4 stream kernels on both host and guest. The earliest version of kvm that I currently have loaded in the lab is kvm-83-13 and this exhibits the problem.

I know from previous testing that this was not a problem, I'm just not sure how back in the kvm history I need to go. 

Note that as the data below will show, the latest UDP backpressure patches are applied and thus not involved  in this issue. 

How reproducible:

Every time

Steps to Reproduce:
1.run netperf from guest to external box or host
2.
3.
  
Actual results:

[root@dhcp47-69 np2.4]# ./netperf -l 5 -H 192.168.1.15 -t UDP_STREAM 
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.15 (192.168.1.15) port 0 AF_INET : spin interval : demo
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024   65507   5.00         3353      0     351.41
129024           5.00         3353            351.41


Expected results:

This should ideally be at wire speed (961 for 1Gb/sec link). 
Additional info:

a message size of 581 bytes seems to be the magical breaking point. 
[root@dhcp47-69 np2.4]# ./netperf -l 20 -H 192.168.1.15 -t UDP_STREAM -- -m 581
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.15 (192.168.1.15) port 0 AF_INET : spin interval : demo
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024     581   20.00     5981253      0    1389.98
129024           20.00     3865413            898.28


Increase the message size by 1 byte causes a dramatic drop in throughput

[root@dhcp47-69 np2.4]# ./netperf -l 20 -H 192.168.1.15 -t UDP_STREAM -- -m 582
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.15 (192.168.1.15) port 0 AF_INET : spin interval : demo
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024     582   20.00      940036      0     218.83
129024           20.00      940036            218.83

Comment 1 Mark Wagner 2009-06-09 02:10:31 UTC
Created attachment 346956 [details]
Output of oprofile run with 581 byte message

This is the output of an oprofile run of the "good" case.  I was using a 581 byte message.

Comment 2 Mark Wagner 2009-06-09 02:11:32 UTC
Created attachment 346957 [details]
oprofile output for a message size of 582 bytes

Comment 3 Eduardo Habkost 2009-06-09 17:15:30 UTC
Proposing exception flag.

Comment 5 Mark McLoughlin 2009-06-15 09:16:38 UTC
To clarify:

  1) There's no patch yet

  2) This is a regression since the last time it was tested; we don't know
     what the previous version was

  3) This is not related to UDP flow control feature request (bug #495863)
     since packets aren't being dropped here, it's just low throughput

Comment 7 Mark McLoughlin 2009-07-02 16:28:47 UTC
(In reply to comment #0)

> Increase the message size by 1 byte causes a dramatic drop in throughput

Okay, this seemed to be the best clue we had to go on here, so I spent a good deal of time looking into it.

I can reproduce here - a simple x86_64 guest->host UDP_STREAM test drops ~30% if you increase the message size from 581 to 582.

With 581 bytes of data, we get a skb size of 640 bytes (5 x 128b cache lines) and a truesize of 888 bytes, allowing us 146 packets before we hit the (guest) sndbuf limit of 129024.

With 582 bytes of data, we jump to an skb size of 768 bytes (6 x128b cache lines) and a truesize of 1016 bytes, allowing us 127 packets before we hit the sndbuf limit.

Now, our virtio queue size is 256, and each packet needs two ring entries (because of the virtio_net header) so the ring can actually only hold 128 packets.

So, in reality - with 581, we actually can only queue up 128 packets, whereas with 582, we can queue up 127.

The difference appears to be between the limits we are hitting; with 581 bytes, we're hitting the queue size limit and with 582 bytes, we're hitting the sndbuf limit. Hitting the queue size limit appears to be much more efficient than hitting the sndbuf limit.

Sure enough, if I set net.core.wmem_default to 258048, then the results for 581/582 are much more similar.

(That's not a final conclusion, just adding my findings so far)

Comment 8 Herbert Xu 2009-07-03 00:24:55 UTC
Mark, have you tried turning TX mitigation off?

Comment 9 Mark Wagner 2009-07-03 02:26:23 UTC
Herbert, I had done some tests with tx mitigation off, the results and some high level "analysis" are posted here: 
http://perfweb.usersys.redhat.com/?q=node/198

Comment 10 Mark Wagner 2009-07-03 02:29:43 UTC
To continue comment #9, the ping latency also comes way down when the stuff is removed. I have not tried any MRG style latency tests on the box w/o the tx timers, however, I would expect that those latency times would be way down as well.

 Latency is reduced with the timers removed.  When there is a timer the minimum ping time is over 1 msec. 

--- 172.17.10.15 ping statistics ---
12 packets transmitted, 12 received, 0% packet loss, time 10998ms
rtt min/avg/max/mdev = 1.058/1.213/2.689/0.445 ms
 

Without the timer things are an order of magnitude better, the max time was 0.114  msec. 

 --- 172.17.10.15 ping statistics ---
13 packets transmitted, 13 received, 0% packet loss, time 12000ms
rtt min/avg/max/mdev = 0.080/0.088/0.114/0.011 ms

Without the timer we are still slightly more than bare metal though.. bare-metal is:

 --- 172.17.10.15 ping statistics ---
13 packets transmitted, 13 received, 0% packet loss, time 12000ms
rtt min/avg/max/mdev = 0.034/0.039/0.066/0.010 ms

Comment 12 Herbert Xu 2009-07-03 02:37:39 UTC
Actually I think he has :)

It's no mystery why the TX mitigation would hurt one-way UDP performance.  The whole point of TX mitigation is to withhold the notification from the guest.  This means it simply becomes an arms race between the guest and the host with regard to who has the longer queue.

Since the host queue is not a single entity (the packets can be scattered around into different queues), there is no consistent strategy for the guest to win the race.

Therefore the only sane solution is to get rid of TX mitigation.

So the conclusion is find out how we can do this without hurting other benchmarks, in particular, TCP bulk transfers as identified by Mark W.

Comment 13 Mark McLoughlin 2009-07-03 06:41:29 UTC
(In reply to comment #7)
> Hitting the queue size limit appears to be much more efficient than hitting the 
> sndbuf limit.

Any thoughts on this? This is in the guest

Comment 14 Mark McLoughlin 2009-07-03 06:51:27 UTC
(In reply to comment #12)

> It's no mystery why the TX mitigation would hurt one-way UDP performance.

Yes, TX mitigation is making the guest stop sending while it waits for the host to wake up and flush the packets.

I had thought that when the ring fills up a queue flush is forced, however we discovered lately that this is not the case.

> The whole point of TX mitigation is to withhold the notification from the 
> guest.

"withhold the notification"? I don't follow?

The whole point of TX mitigation is to have the guest avoid exiting every time it sends a packet.

> This means it simply becomes an arms race between the guest and the host with
> regard to who has the longer queue.
>
> Since the host queue is not a single entity (the packets can be scattered
> around into different queues), there is no consistent strategy for the guest 
> to win the race.

Nice story, but I don't understand. What two queues are you referring to?

> Therefore the only sane solution is to get rid of TX mitigation.

Well, yes - I know that's your conclusion :-)

> So the conclusion is find out how we can do this without hurting other
> benchmarks, in particular, TCP bulk transfers as identified by Mark W.

Um, yes - if we could remove TX mitigation without hurting TCP bulk transfers, we'd do it in an instant.

Comment 16 Mark McLoughlin 2009-07-03 06:59:30 UTC
Also, tx mitigation isn't an answer for why this has regressed - we haven't changed the tx timer. The regression is what we should be first getting to the bottom of here, IMHO

Comment 17 Herbert Xu 2009-07-03 07:36:24 UTC
Not everything is a regression.  Sometimes things work by accident, and then they break.

In this case, the guest requires a timely notification to make forward progress.  Because the sender's socket's TX limit has been reached.  You can hack around this by increasing the TX buffer size as you noticed, or you can lie by calling skb_orphan early in the driver.  The latter is definitely not a good idea.  Look at cxgb3 which does exactly this and how it manages to break Xen.  This is also not specific to Xen.  The same thing can occur for anything that requires page releases to make forward progress, e.g., iSCSI.

The fundamental issue is that timely notifications are needed in many cases to make forward progress.  It simply makes no sense for both the host and the guest to sit idle when a notification is pending from the host that can allow the guest start working.

Comment 18 Mark McLoughlin 2009-07-03 08:48:30 UTC
(In reply to comment #17)
> Sometimes things work by accident, and then they break.

And then we do something else to make them work again by accident, and then they break again.

I'd prefer to understand what changed, if possible.

> The fundamental issue is that timely notifications are needed in many cases to
> make forward progress.

TX mitigation doesn't delay sending the notification back to the guest, it delays the processing of the packets in the host.

> It simply makes no sense for both the host and the guest to sit idle when a 
> notification is pending from the host that can allow the guest start 
> working.  

Right, it makes no sense for both the host and guest to sit idle - we should make sure the host flushes packets before the tx queue fills up.

i.e. if we fixed tx mitigation to avoid this happening, would you still object to it?

Or do you have any better ideas for reducing guest exits?

> Because the sender's socket's TX limit has been reached.

There are two limits. With smaller packets, we hit the limit of the virtio_net queue, and with larger packets we hit the sndbuf limit as you say. The former appears to be substantially more efficient. Any ideas why?

Comment 19 Herbert Xu 2009-07-03 09:19:39 UTC
(In reply to comment #18)
> > The fundamental issue is that timely notifications are needed in many cases to
> > make forward progress.
> 
> TX mitigation doesn't delay sending the notification back to the guest, it
> delays the processing of the packets in the host.

It's the same thing from the perspective of the guest.  The notification is not sent as early as it can be.
 
> > It simply makes no sense for both the host and the guest to sit idle when a 
> > notification is pending from the host that can allow the guest start 
> > working.  
> 
> Right, it makes no sense for both the host and guest to sit idle - we should
> make sure the host flushes packets before the tx queue fills up.
> 
> i.e. if we fixed tx mitigation to avoid this happening, would you still object
> to it?

If you can ensure that we never enter a state where both the host and the guest are idle while there is work to do, then yes I won't object.

> Or do you have any better ideas for reducing guest exits?

We shouldn't need TX mitigation if the ring is managed properly.  In particular, when the guest first notifies the host that there is data via the HV, the HV should not schedule the host straight away if it's on the same CPU as the guest.  This gives a natural mitigation without resorting to artificial timers, the scheduler's timer is the only one needed.

> There are two limits. With smaller packets, we hit the limit of the virtio_net
> queue, and with larger packets we hit the sndbuf limit as you say. The former
> appears to be substantially more efficient. Any ideas why?  

Because the virtio net queue fills up we still have more packets to tickle the other side.  When the socket buffer is reached traffic will stop altogether until the mitigation timer expires.

Comment 22 Mark McLoughlin 2009-07-21 16:33:51 UTC
Created attachment 354518 [details]
virtio_net-remove-the-tx-mitigation-timer.patch

Comment 30 Suqin Huang 2009-07-23 07:00:16 UTC
1. test on kvm-83-94.el5

[root@dhcp-66-83-81 ~]# netperf -l 20 -H 10.66.82.249 -t UDP_STREAM -- -m 581
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.82.249 (10.66.82.249) port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024     581   20.00      306984      0      71.34
129024           20.00      306984             71.34

[root@dhcp-66-83-81 ~]# netperf -l 20 -H 10.66.82.249 -t UDP_STREAM -- -m 582
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.82.249 (10.66.82.249) port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024     582   20.00      313141      0      72.89
129024           20.00      313141             72.89


2. test on kvm-83-87.el5

[root@dhcp-66-83-95 ~]# netperf -l 20 -H 10.66.82.249 -t UDP_STREAM -- -m 581
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.82.249 (10.66.82.249) port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024     581   20.00      318016      0      73.90
129024           20.00      318016             73.90


[root@dhcp-66-83-95 ~]# netperf -l 20 -H 10.66.82.249 -t UDP_STREAM -- -m 582
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.82.249 (10.66.82.249) port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024     582   20.00      688977      0     160.39
129024           20.00      688977            160.39

Comment 31 Mark McLoughlin 2009-07-23 07:56:51 UTC
That's not good

shuang: what kind of network is the host connected to 10.66.82.249 over? What results do you get if you run the netperf test to 10.66.82.249 on the host?

mwagner: could you see if you can reproduce those results?

Comment 32 Suqin Huang 2009-07-23 09:30:34 UTC
this is the result connect from host.
[root@intel-q9400-4-2 ~]# !netperf 
netperf -l 20 -H 10.66.82.249 -t UDP_STREAM -- -m 582 
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.82.249 (10.66.82.249) port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

129024     582   20.00     3846256      0     895.37
129024           20.00     3837750            893.39



10.66.82.249 is another host.

Comment 33 Mark Wagner 2009-07-23 15:05:44 UTC
If you have more than one network interface in the host can you make sure that you add 
net.ipv4.conf.default.arp_filter = 1 

in /etc/sysctl.conf in both the host and guests? (Well actually all systems involved in the network tests)

Also disable the firewalls and repeat

Comment 35 Mark McLoughlin 2009-07-24 16:14:06 UTC
I've just done some very simple testing of guest->host ...

with kvm-83-92.el5:

for b in $(seq 580 1 584); do netperf -t UDP_STREAM -f m -H 192.168.1.1 -P 0 -l 5 -- -m $b; done
124928     580   5.00       492375      0     456.90
124928     581   5.00       455793      0     423.69
124928     582   5.00       312625      0     291.02
124928     583   5.00       312625      0     291.52
124928     584   5.00       312589      0     291.98

with kvm-83-94.el5:
for b in $(seq 580 1 584); do netperf -t UDP_STREAM -f m -H 192.168.1.1 -P 0 -l 1 -- -m $b; done
124928     580   1.00       276566      0    1283.10
124928     581   1.00       276805      0    1285.83
124928     582   1.00       275455      0    1280.10
124928     583   1.00       275038      0    1281.50
124928     584   1.00       274497      0    1281.18

shuang: please answer mwagner's questions/try his suggestion

Comment 36 Mark Wagner 2009-07-24 20:10:07 UTC
*** Bug 501607 has been marked as a duplicate of this bug. ***

Comment 37 Suqin Huang 2009-07-27 03:02:07 UTC
already add net.ipv4.conf.default.arp_filter = 1 and stop iptables

on kvm-83-94.el5:

rhel4.8-64 with virtio:

135168     580   5.00       601020      0     557.68
129024           5.00       601020            557.68

135168     581   5.00       641627      0     596.27
129024           5.00       641627            596.27

135168     582   5.00       652495      0     607.62
129024           5.00       652495            607.62

135168     583   5.00       662810      0     618.13
129024           5.00       662810            618.13

135168     584   5.00       629807      0     588.43
129024           5.00       629807            588.43


rhel4.8-64 with e1000:

135168     580   5.00        87491      0      81.19
129024           5.00        87491             81.19

135168     581   5.00        87579      0      81.42
129024           5.00        87579             81.42

135168     582   5.00        86554      0      80.60
129024           5.00        86554             80.60

135168     583   5.00        87270      0      81.41
129024           5.00        87270             81.41

135168     584   5.00        87369      0      81.64
129024           5.00        87369             81.64


rhel4.8-64 with rtl8139:

135168     580   5.00       118701      0     110.16
129024           5.00       118701            110.16

135168     581   5.00       119877      0     111.44
129024           5.00       119877            111.44

135168     582   5.00       119825      0     111.58
129024           5.00       119825            111.58

135168     583   5.01       118868      0     110.73
129024           5.01       118868            110.73

135168     584   5.00       119870      0     112.02
129024           5.00       119870            112.02


similar result when test on other guest.

Comment 38 Suqin Huang 2009-07-27 03:23:31 UTC
on kvm-83-87.el5

rhel4.8-64 with virtio:

for b in $(seq 580 1 584); do netperf -t UDP_STREAM -f m -H 10.66.70.31 -P 0 -l 5 -- -m $b; done
135168     580   5.00       746852      0     693.09
129024           5.00       746233            692.52

135168     581   5.00       794769      0     738.84
129024           5.00       781561            726.57

135168     582   5.00       718060      0     668.66
129024           5.00       718060            668.66

135168     583   5.01       785544      0     730.60
129024           5.01       770838            716.93

135168     584   5.00       732211      0     684.20
129024           5.00       732211            684.20


rhel4.8-64 with rtl8139:

for b in $(seq 580 1 584); do netperf -t UDP_STREAM -f m -H 10.66.70.31 -P 0 -l 5 -- -m $b; done
135168     580   5.00       119625      0     111.02
129024           5.00       119625            111.02

135168     581   5.00       260162      0     241.82
129024           5.00       260162            241.82

135168     582   5.00       120398      0     112.13
129024           5.00       120398            112.13

135168     583   5.00       120044      0     111.98
129024           5.00       120044            111.98

135168     584   5.00       120678      0     112.76
129024           5.00       120678            112.76


e1000:
135168     580   5.00        84856      0      78.75
129024           5.00        84856             78.75

135168     581   5.00        85488      0      79.47
129024           5.00        85488             79.47

135168     582   5.00        85770      0      79.87
129024           5.00        85770             79.87

135168     583   5.00        85717      0      79.95
129024           5.00        85717             79.95

135168     584   5.00        84658      0      79.11
129024           5.00        84658             79.11

Comment 39 Suqin Huang 2009-07-27 05:24:00 UTC
retest on rhev-hypervisor-5.4-2.0.99 (11) with kvm-83-90.el5
for comment #38 is test on rhev-hypervisor-5.4-2.0.99.12.1 and downgrade kvm-83-94.el5 to kvm-83-87.el5.

virtio:
[root@dhcp-66-70-51 ~]# for b in $(seq 580 1 584); do netperf -t UDP_STREAM -f m -H 10.66.70.31 -P 0 -l 5 -- -m $b; done
129024     580   5.00       667131      0     618.87
129024           5.00       667131            618.87

129024     581   5.00       667950      0     620.82
129024           5.00       667950            620.82

129024     582   5.00       312959      0     291.25
129024           5.00       312959            291.25

129024     583   5.00       294102      0     274.26
129024           5.00       294102            274.26

129024     584   5.00       291166      0     272.02
129024           5.00       291166            272.02


e1000:
129024     580   5.00        92604      0      85.91
129024           5.00        92604             85.91

129024     581   5.00        93497      0      86.91
129024           5.00        93497             86.91

129024     582   5.00        94902      0      88.34
129024           5.00        94902             88.34

129024     583   5.00        94771      0      88.38
129024           5.00        94771             88.38

129024     584   5.00        94472      0      88.27
129024           5.00        94472             88.27

Comment 40 Mark McLoughlin 2009-07-28 13:32:56 UTC
Okay, I think we've got a little side-tracked by this 581/582 byte issue - that's not the bug we're trying to fix here, but more generally that UDP performance is poor

So, I just tried guest->host with:

 $> for b in 32 64 128 256 512 1024 2048 4096 8192 16834 32768; do netperf -t UDP_STREAM -f m -H 192.168.1.1 -P 0 -l 1 -- -m $b; done

with kvm-83-87.el5, kvm-83-90.el5 and kvm-83-94.el5 and the results look like:

  http://tinyurl.com/nllqhp

Which shows that -94 is significantly better except for 512/1024 byte messages

shuang: I think if you can produce similar results for guest->external, we can mark this issue as verified

Comment 41 Mark Wagner 2009-07-28 14:09:34 UTC
Mark  I modified and ran your script in my guest to an external box.  I ran on both the 1Gbit/sec link and the 10Gbit/sec link.  It appears that the original issue is corrected by this change.

For 1Gbit/sec
[root@dhcp47-64 np2.4]# for b in 32 64 128 256 512 1024 2048 4096 8192 16834 32768; do ./netperf -t UDP_STREAM -f m -H 192.168.1.15 -P 0 -l 10 -- -m $b; done
129024      32   10.00     4296814      0     109.99
129024           10.00     3842164             98.35

129024      64   10.00     4231166      0     216.63
129024           10.00     3736179            191.29

129024     128   10.00     4335454      0     443.91
129024           10.00     2449011            250.75

129024     256   10.00     4487681      0     918.98
129024           10.00     3679605            753.51

129024     512   10.00     4875082      0    1996.64
129024           10.00     2164447            886.47

129024    1024   10.00     1147764      0     940.22
129024           10.00     1147764            940.22

129024    2048   10.00      575888      0     943.46
129024           10.00      575888            943.46

129024    4096   10.00      292410      0     958.02
129024           10.00      292410            958.02

129024    8192   10.00      146405      0     959.42
129024           10.00      146405            959.42

129024   16834   10.00       71343      0     960.73
129024           10.00       71343            960.73

129024   32768   10.00       36691      0     961.71
129024           10.00       36691            961.71

For 10Gbit/sec

[root@dhcp47-64 np2.4]# for b in 32 64 128 256 512 1024 2048 4096 8192 16834 32768; do ./netperf -t UDP_STREAM -f m -H 172.17.10.15 -P 0 -l 10 -- -m $b; done
129024      32   10.00     2773236      0      70.98
129024           10.00     2773236             70.98

129024      64   10.00     2610841      0     133.65
129024           10.00     2610841            133.65

129024     128   10.00     2876831      0     294.57
129024           10.00     2876831            294.57

129024     256   10.00     2846514      0     582.92
129024           10.00     2846514            582.92

129024     512   10.00     2915333      0    1194.07
129024           10.00     2914717           1193.81

129024    1024   10.00     2921396      0    2392.84
129024           10.00     2919378           2391.19

129024    2048   10.00     1568752      0    2569.97
129024           10.00     1541255           2524.92

129024    4096   10.00     1207473      0    3956.31
129024           10.00     1151074           3771.52

129024    8192   10.00      610241      0    3998.81
129024           10.00      594470           3895.46

129024   16834   10.00      279059      0    3757.94
129024           10.00      259639           3496.42

129024   32768   10.00      140095      0    3672.15
129024           10.00      114791           3008.88

Comment 42 Mark McLoughlin 2009-07-28 14:15:30 UTC
(In reply to comment #41)
> Mark  I modified and ran your script in my guest to an external box.  I ran on
> both the 1Gbit/sec link and the 10Gbit/sec link.  It appears that the original
> issue is corrected by this change.

thanks, just need shuang to reproduce in order to move this to VERIFIED

Comment 43 Suqin Huang 2009-07-29 09:50:13 UTC
on kvm-83-94.el5

start vm with virtio network interface

#for b in 32 64 128 256 512 1024 2048 4096 8192 16834 32768; do netperf -t UDP_STREAM -f m -H 10.66.70.31 -P 0 -l 10 -- -m $b; done
129024      32   10.00     2096937      0      53.68
129024           10.00     2096937             53.68

129024      64   10.00     2241090      0     114.73
129024           10.00     2241090            114.73

129024     128   10.00     2217548      0     227.05
129024           10.00     2217548            227.05

129024     256   10.00     2194591      0     449.39
129024           10.00     2194382            449.35

129024     512   10.00     2143656      0     877.97
129024           10.00     2140185            876.55

129024    1024   10.00     1963698      0    1608.43
129024           10.00     1962477           1607.43

129024    2048   10.00     1185332      0    1941.77
129024           10.00     1184391           1940.23

129024    4096   10.00      843659      0    2764.07
129024           10.00      842967           2761.80

129024    8192   10.00      466460      0    3056.64
129024           10.00      466109           3054.34

129024   16834   10.00      252689      0    3402.58
129024           10.00      252652           3402.08

129024   32768   10.00      128217      0    3360.47
129024           10.00      127977           3354.18


on kvm-83-90.el5:

129024      32   10.00     1396409      0      35.75
129024           10.00     1396409             35.75

129024      64   10.00     1356632      0      69.45
129024           10.00     1356632             69.45

129024     128   10.00     1281657      0     131.21
129024           10.00     1281657            131.21

129024     256   10.00     1261600      0     258.36
129024           10.00     1261600            258.36

129024     512   10.00     1194475      0     489.25
129024           10.00     1194475            489.25

129024    1024   10.00      458113      0     375.25
129024           10.00      458113            375.25

129024    2048   10.00      250802      0     410.87
129024           10.00      250801            410.87

129024    4096   10.00      259159      0     849.05
129024           10.00      259159            849.05

129024    8192   10.00      128421      0     841.53
129024           10.00      128421            841.53

129024   16834   10.00       69501      0     935.88
129024           10.00       69501            935.88

129024   32768   10.00       39903      0    1045.92
129024           10.00       39903           1045.92

Comment 44 Suqin Huang 2009-07-29 10:01:40 UTC
markmc:
Can I VERIFIED the bug according to the test result.

Comment 45 Suqin Huang 2009-07-29 10:33:54 UTC
start vm with e1000 network interface

on kvm-83-90.el5:


for b in 32 64 128 256 512 1024 2048 4096 8192 16834 32768; do netperf -t UDP_STREAM -f m -H 10.66.70.31 -P 0 -l 10 -- -m $b; done
129024      32   10.00     1396409      0      35.75
129024           10.00     1396409             35.75

129024      64   10.00     1356632      0      69.45
129024           10.00     1356632             69.45

129024     128   10.00     1281657      0     131.21
129024           10.00     1281657            131.21

129024     256   10.00     1261600      0     258.36
129024           10.00     1261600            258.36

129024     512   10.00     1194475      0     489.25
129024           10.00     1194475            489.25

129024    1024   10.00      458113      0     375.25
129024           10.00      458113            375.25

129024    2048   10.00      250802      0     410.87
129024           10.00      250801            410.87

129024    4096   10.00      259159      0     849.05
129024           10.00      259159            849.05

129024    8192   10.00      128421      0     841.53
129024           10.00      128421            841.53

129024   16834   10.00       69501      0     935.88
129024           10.00       69501            935.88

129024   32768   10.00       39903      0    1045.92
129024           10.00       39903           1045.92



on kvm-83-94.el5:

129024      32   10.00      195384      0       5.00
129024           10.00      195384              5.00

129024      64   10.00      191712      0       9.81
129024           10.00      191712              9.81

129024     128   10.00      190155      0      19.47
129024           10.00      190155             19.47

129024     256   10.00      190526      0      39.01
129024           10.00      190526             39.01

129024     512   10.00      191912      0      78.60
129024           10.00      191912             78.60

129024    1024   10.00      192820      0     157.93
129024           10.00      115343             94.47

129024    2048   10.00       99738      0     163.38
129024           10.00       29670             48.60

129024    4096   10.00       69036      0     226.20
129024           10.00       22532             73.83

129024    8192   10.00       36650      0     240.14
129024           10.00        8477             55.54

129024   16834   10.00       18995      0     255.78
129024           10.00         153              2.06

129024   32768   10.00       10052      0     263.49
129024           10.00          79              2.07


speed of e1000 is slow.

Comment 46 Mark McLoughlin 2009-07-29 12:03:55 UTC
(In reply to comment #44)
> markmc:
> Can I VERIFIED the bug according to the test result.  

Yes, thanks.

(In reply to comment #45)
> on kvm-83-94.el5:
> 
> 129024      32   10.00      195384      0       5.00
> 129024           10.00      195384              5.00
> 
> 129024      64   10.00      191712      0       9.81
> 129024           10.00      191712              9.81
> 
> 129024     128   10.00      190155      0      19.47
> 129024           10.00      190155             19.47
> 
> 129024     256   10.00      190526      0      39.01
> 129024           10.00      190526             39.01
> 
> 129024     512   10.00      191912      0      78.60
> 129024           10.00      191912             78.60
> 
> 129024    1024   10.00      192820      0     157.93
> 129024           10.00      115343             94.47
> 
> 129024    2048   10.00       99738      0     163.38
> 129024           10.00       29670             48.60
> 
> 129024    4096   10.00       69036      0     226.20
> 129024           10.00       22532             73.83
> 
> 129024    8192   10.00       36650      0     240.14
> 129024           10.00        8477             55.54
> 
> 129024   16834   10.00       18995      0     255.78
> 129024           10.00         153              2.06
> 
> 129024   32768   10.00       10052      0     263.49
> 129024           10.00          79              2.07 
> 
> speed of e1000 is slow.

Wow. Please file a new bug for this - the fix for this bug can not affect e1000, so it's something else. Indeed, I don't see anything obvious between kvm-83-90.el5 and kvm-83-94.el5

Comment 47 Suqin Huang 2009-07-30 03:55:35 UTC
(In reply to comment #45)

> on kvm-83-94.el5:
> 
> 129024      32   10.00      195384      0       5.00
> 129024           10.00      195384              5.00
> 
> 129024      64   10.00      191712      0       9.81
> 129024           10.00      191712              9.81
> 
> 129024     128   10.00      190155      0      19.47
> 129024           10.00      190155             19.47
> 
> 129024     256   10.00      190526      0      39.01
> 129024           10.00      190526             39.01
> 
> 129024     512   10.00      191912      0      78.60
> 129024           10.00      191912             78.60
> 
> 129024    1024   10.00      192820      0     157.93
> 129024           10.00      115343             94.47
> 
> 129024    2048   10.00       99738      0     163.38
> 129024           10.00       29670             48.60
> 
> 129024    4096   10.00       69036      0     226.20
> 129024           10.00       22532             73.83
> 
> 129024    8192   10.00       36650      0     240.14
> 129024           10.00        8477             55.54
> 
> 129024   16834   10.00       18995      0     255.78
> 129024           10.00         153              2.06
> 
> 129024   32768   10.00       10052      0     263.49
> 129024           10.00          79              2.07
> 
> 
> speed of e1000 is slow.  


also lose packets when start vm with virtio network interface on kvm-83-94.el5, already report it in bug 508861.

Comment 48 Suqin Huang 2009-07-30 03:56:08 UTC
Change the status to *VERIFIED* according to the test result, please reopen it if you can produce it.

Comment 50 errata-xmlrpc 2009-09-02 09:35:35 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1272.html