Bug 688125

Summary:	IP packet loss on dom0 while xen domU running high load
Product:	Red Hat Enterprise Linux 5	Reporter:	Kirby Zhou <kirbyzhou>
Component:	xen	Assignee:	Xen Maintainance List <xen-maint>
Status:	CLOSED NOTABUG	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	5.5	CC:	drjones, leiwang, mrezanin, xen-maint, yuzhang
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-03-17 08:25:17 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Kirby Zhou 2011-03-16 11:56:29 UTC

Description of problem:

IP packet loss on dom0 while xen domU running high load.


Version-Release number of selected component (if applicable):

kernel-xen-2.6.18-194.32.1.el5
xen-3.0.3-105.el5_5.5
libvirt-0.6.3-33.el5_5.3

How reproducible:

100%

Steps to Reproduce:
1. Create a domU with 14 vcpu on a E5520*2 box.
Note that: E5520*2 has 16 logical-cpu, we only take 14 in domU
dom0~]# cat /etc/xen/vm-test
name = "vm-test"
uuid = "0de437c1-b224-0bad-c21a-c66f1beb2a13"
maxmem = 16384
memory = 16384
vcpus = 14
cpus = "0-11"
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
disk = [ "phy:/dev/vgext/vm-test,xvda,w", "phy:/dev/vgext/vm-test-data,xvdb,w" ]
vif = [ "mac=00:16:36:37:80:94,bridge=xenbr0,script=vif-bridge", "mac=00:16:36:37:80:95,bridge=xenbr1,script=vif-bridge", "mac=00:16:36:37:80:96,bridge=xenbr2,script=vif-bridge" ]

2. Run something with high cpu usage inside domU, such as following:
Note that: domU has 14 vcpu, we only take 12 jobs here.
domU~]# for ((i=0;i<12;++i)); do bzip2 < /dev/zero > /dev/null & done

domU]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
12  0      0 4180384 293372 6193784    0    0    12     7    4   30  3  1 97  0  0
12  0      0 4180392 293372 6193784    0    0     0     0 1291  250 84  1 14  0  1
12  0      0 4180556 293372 6193816    0    0     0     0 1237  250 85  0 14  0  0
12  0      0 4180556 293380 6193808    0    0     0    16 1258  256 84  1 14  0  0
1

3. run iperf on dom0

dom0] iper -s -u -i 1

4. send udp to dom0:
OtherPC~] iperf -u -i 1 -c 10.11.200.4 -b 10M -l 100

 
Actual results:

with bzip2 on domU, 23% packet loss
[  3] Server Report:
[  3]  0.0-10.0 sec  9.14 MBytes  7.66 Mbits/sec   0.402 ms 29137/125000 (23%)

without bzip2 on domU, no packet loss.
[  3] Server Report:
[  3]  0.0-10.0 sec  11.9 MBytes  10.0 Mbits/sec   0.002 ms    0/125000 (0%)

Expected results:

something like 'without bzip2 on domU'

Additional info:

[@tc_200_4 ~]# cat /proc/cpuinfo | head
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 26
model name  : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
stepping    : 5
cpu MHz     : 2266.746
cache size  : 8192 KB
physical id : 0
siblings    : 1

If I do run 12 bzip2 instance on dom0, everything goes right.

Comment 1 Kirby Zhou 2011-03-16 12:03:53 UTC

More info:

in the file /etc/xen/vm-test, change the following lines
    vcpus = 14
    cpus = "0-11"
to 
    vcpus = 14
    cpus = "0-15"

The actual result would be:

[  3] Server Report:
[  3]  0.0-10.0 sec  58.9 MBytes  49.4 Mbits/sec   0.011 ms 7734/625000 (1.2%)

the packet drop rate is lower than prior, but still high.

Comment 2 Yufang Zhang 2011-03-17 07:44:44 UTC

I *guess* this is by design. All VCPUs including the ones belonging to Domain0 are given the same priority by default for schedule. So in case of high CPU load, all VCPUs belonging to DomainU and Domain0 have to share the limited computing resources. Thus the performance of Domain0 and DomainU is not isolated. 

Xen does provide some kind of scheme for performance isolation, such as CPU WEIGHT and CAP. But it really depends. For the scenario in Description, you could either

(1) Give lower weight to DomainU VCPUs via 'xm sched-credit'

or 

(2) Give a proper CAP(also via xm sched-credit) to DomainU to make sure it couldn't consume too much cpu time.

or 

(3) Simply reduce number of VCPUs of DomainU.

All these three methods could make sure Domain0 VCPUs get more CPU time, and of course would have some impact on application running within DomainU. It depends on how you trade off. 

BTW, it is not recommended to run applications within Domain0.

Comment 3 Andrew Jones 2011-03-17 08:25:17 UTC

I agree with Yufang that this isn't a bug.

When the hypervisor is scheduling vcpus it can't know that some of them are trying to receive udp packets, which, if not received immediately, will get dropped. When you move the bzip into dom0 then the dom0 kernel can schedule more intelligently, and thus keep the udp receiving task active, avoiding packet loss.

I would be a bit more concerned if you had packet loss while using tcp, but even then without the hypervisor being informed of vcpu priorities (using sched-credit as pointed out by Yufang), then the dom0 vcpus may not get the cycles they need to keep a tcp connection alive.