Description of problem:

virtio_net performance is very poor with RHEL 5.4 KVM

Version-Release number of selected component (if applicable):

RHEL-5.4Beta

kernel-2.6.18-155.el5
kvm-83-80.el5
kvm-qemu-img-83-80.el5
kvm-tools-83-80.el5
kmod-kvm-83-80.el5
libvirt-0.6.3-11.el5
libvirt-python-0.6.3-11.el5
python-virtinst-0.400.3-4.el5


How reproducible:

100%

Steps to Reproduce:
1.

Install RHEL-5.4beta as guest under RHEL-5.4beta host. the conf file is here:

<domain type='kvm'>
  <name>vmtest</name>
  <uuid>abd2ee53-bef9-60ad-253a-a9f7768896f9</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <source dev='/dev/vgtest/lvtest'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <disk type='block' device='disk'>
      <source dev='/dev/vgtest/lvtest2'/>
      <target dev='vdb' bus='virtio'/>
    </disk>
    <interface type='bridge'>
      <mac address='54:52:00:73:e5:f8'/>
      <source bridge='xenbr0'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <mac address='54:52:00:73:e5:f9'/>
      <source bridge='xenbr1'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target port='0'/>
    </console>
  </devices>
</domain>

2. run netserver of netperf on another machine

3. run netperf -t UDP_RR -H remote
   run ping -fc 1000 remote
  
Actual results:

Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

129024 129024 1        1       10.00     993.18
129024 129024

1000 packets transmitted, 1000 received, 0% packet loss, time 1035ms
rtt min/avg/max/mdev = 0.596/0.949/4.260/0.194 ms, ipg/ewma 1.036/0.921 ms

Expected results:

reduce rtt to less than 0.2ms. 

Additional info:

Xen HVM domu can do it well at the same machine.
[root@localhost ~]# netperf -t TCP_RR -H 10.12.12.111
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

16384  87380  1        1       10.00    7355.47   
16384  87380 
[root@localhost ~]# ping -fc 1000 10.12.12.111
PING 10.12.12.111 (10.12.12.111) 56(84) bytes of data.
 
--- 10.12.12.111 ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 173ms
rtt min/avg/max/mdev = 0.096/0.120/0.175/0.016 ms, ipg/ewma 0.173/0.118 ms
[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.18-155.el5 #1 SMP Fri Jun 19 17:06:31 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux