Bug 203669

Summary: packetloss on vifN.0 interfaces
Product: [Fedora] Fedora Reporter: Matt C <wago>
Component: xenAssignee: Herbert Xu <herbert.xu>
Status: CLOSED RAWHIDE QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 6CC: bstein, katzj
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-08-30 13:11:00 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 216441    

Description Matt C 2006-08-22 18:07:50 EDT
Description of problem:
Transmit packetloss on the vifN.0 interfaces in domain0

Version-Release number of selected component (if applicable):
kernel-xen-2.6.17-1.2517.fc6
xen-3.0.2-23

How reproducible:
High velocity network traffic where the destination is a nonprivileged virtual
host. Traffic can originate from any of Domain0, another DomainU or outside of
the host. Packetloss seems highest when the data is moving between two DomainU
hosts on the same hardware. Default bridge network setup, no NAT or iptables.

Steps to Reproduce:
1. build xen system with FC6t2
2. build two nonprivileged virtual hosts
3. install ttcp on both virtual hosts
   (http://dag.wieers.com/packages/ttcp/ or wherever)
4. in Domain0, start watching Tx drops in netstat
   'watch -n1 netstat -ni'
5. on one virtual host, start the receiver
   'ttcp -r -s'
6. on another virtual host (or anywhere else), start the sender
   'ttcp -t -s -n 100000 [ip-of-receiver]'
  
Actual results:
Bursty packetloss reported by netstat/ifconfig in domain0 in the range of
100-1000 packets/second.

Expected results:
No packetloss reported in domain0.

Additional info:
TCP throughput still seems to be good (106MB/sec on GBE), but the packet loss
reported by 'netstat -ni' concerns me.
Comment 1 Herbert Xu 2006-08-24 08:51:12 EDT
OK this behaviour is expected because until recently the dom0 vif interface
didn't have a transmit queue.  So as soon as the ring buffer filled up packets
were dropped on the floor, without sending the network stack an immediate
congestion notification.

The SG patches add a queue which should resolve this problem.

Just to confirm that this explanation is correct please do an ifconfig in dom0
and show me the result.  Thanks.
Comment 2 Matt C 2006-08-25 14:18:11 EDT
Here's the ifconfig output (IP addr removed):

$ /sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:45:5D:91:86
          inet addr:10.xx.xx.xx  Bcast:10.xx.xx.255  Mask:255.255.255.0
          inet6 addr: fe80::250:45ff:fe5d:9186/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1208630 errors:0 dropped:0 overruns:0 frame:0
          TX packets:225372 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:200175027 (190.9 MiB)  TX bytes:48826730 (46.5 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:786 (786.0 b)  TX bytes:786 (786.0 b)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:6216531 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2633304 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2003284415 (1.8 GiB)  TX bytes:511532266 (487.8 MiB)
          Interrupt:17

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:225374 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1208630 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:48827078 (46.5 MiB)  TX bytes:200175027 (190.9 MiB)

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:161436 errors:0 dropped:0 overruns:0 frame:0
          TX packets:793263 errors:0 dropped:322 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:41966974 (40.0 MiB)  TX bytes:158188705 (150.8 MiB)

vif2.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:181273 errors:0 dropped:0 overruns:0 frame:0
          TX packets:850254 errors:0 dropped:386 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:42929708 (40.9 MiB)  TX bytes:234592103 (223.7 MiB)

vif3.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:160703 errors:0 dropped:0 overruns:0 frame:0
          TX packets:792578 errors:0 dropped:365 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:41673446 (39.7 MiB)  TX bytes:158843029 (151.4 MiB)

vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:118610 errors:0 dropped:0 overruns:0 frame:0
          TX packets:747814 errors:0 dropped:430 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:40574479 (38.6 MiB)  TX bytes:157880775 (150.5 MiB)

vif5.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:165392 errors:0 dropped:0 overruns:0 frame:0
          TX packets:790442 errors:0 dropped:533 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:42070929 (40.1 MiB)  TX bytes:159598416 (152.2 MiB)

vif6.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:167543 errors:0 dropped:0 overruns:0 frame:0
          TX packets:795199 errors:0 dropped:380 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:42130124 (40.1 MiB)  TX bytes:162684719 (155.1 MiB)

vif7.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:161643 errors:0 dropped:0 overruns:0 frame:0
          TX packets:789296 errors:0 dropped:418 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:41619683 (39.6 MiB)  TX bytes:159980192 (152.5 MiB)

vif8.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:162413 errors:0 dropped:0 overruns:0 frame:0
          TX packets:787204 errors:0 dropped:504 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:41681501 (39.7 MiB)  TX bytes:159337088 (151.9 MiB)

vif18.0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:30935 errors:0 dropped:0 overruns:0 frame:0
          TX packets:584441 errors:0 dropped:296 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:16821920 (16.0 MiB)  TX bytes:89773840 (85.6 MiB)

xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:841226 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:90860227 (86.6 MiB)  TX bytes:0 (0.0 b)
Comment 3 Herbert Xu 2006-08-26 02:36:12 EDT
Great, this confirms that the txqueuelen is 0.  So once you upgrade your kernel
to 2583 or later (you have to do it in both dom0 and domU in order to get a TX
queue), the problem should go away.
Comment 4 Matt C 2006-08-30 13:11:00 EDT
Thanks. I've tested with kernel-xen-2.6.17-1.2586.fc6, and this does indeed fix
my problem. TCP throughput increased from 104MB/s to 113MB/s with this new
kernel, and zero packetloss is reported in Dom0. Closing this bug.