Hide Forgot
Description of problem: IP packet loss on dom0 while xen domU running high load. Version-Release number of selected component (if applicable): kernel-xen-2.6.18-194.32.1.el5 xen-3.0.3-105.el5_5.5 libvirt-0.6.3-33.el5_5.3 How reproducible: 100% Steps to Reproduce: 1. Create a domU with 14 vcpu on a E5520*2 box. Note that: E5520*2 has 16 logical-cpu, we only take 14 in domU dom0~]# cat /etc/xen/vm-test name = "vm-test" uuid = "0de437c1-b224-0bad-c21a-c66f1beb2a13" maxmem = 16384 memory = 16384 vcpus = 14 cpus = "0-11" bootloader = "/usr/bin/pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" disk = [ "phy:/dev/vgext/vm-test,xvda,w", "phy:/dev/vgext/vm-test-data,xvdb,w" ] vif = [ "mac=00:16:36:37:80:94,bridge=xenbr0,script=vif-bridge", "mac=00:16:36:37:80:95,bridge=xenbr1,script=vif-bridge", "mac=00:16:36:37:80:96,bridge=xenbr2,script=vif-bridge" ] 2. Run something with high cpu usage inside domU, such as following: Note that: domU has 14 vcpu, we only take 12 jobs here. domU~]# for ((i=0;i<12;++i)); do bzip2 < /dev/zero > /dev/null & done domU]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 12 0 0 4180384 293372 6193784 0 0 12 7 4 30 3 1 97 0 0 12 0 0 4180392 293372 6193784 0 0 0 0 1291 250 84 1 14 0 1 12 0 0 4180556 293372 6193816 0 0 0 0 1237 250 85 0 14 0 0 12 0 0 4180556 293380 6193808 0 0 0 16 1258 256 84 1 14 0 0 1 3. run iperf on dom0 dom0] iper -s -u -i 1 4. send udp to dom0: OtherPC~] iperf -u -i 1 -c 10.11.200.4 -b 10M -l 100 Actual results: with bzip2 on domU, 23% packet loss [ 3] Server Report: [ 3] 0.0-10.0 sec 9.14 MBytes 7.66 Mbits/sec 0.402 ms 29137/125000 (23%) without bzip2 on domU, no packet loss. [ 3] Server Report: [ 3] 0.0-10.0 sec 11.9 MBytes 10.0 Mbits/sec 0.002 ms 0/125000 (0%) Expected results: something like 'without bzip2 on domU' Additional info: [@tc_200_4 ~]# cat /proc/cpuinfo | head processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping : 5 cpu MHz : 2266.746 cache size : 8192 KB physical id : 0 siblings : 1 If I do run 12 bzip2 instance on dom0, everything goes right.
More info: in the file /etc/xen/vm-test, change the following lines vcpus = 14 cpus = "0-11" to vcpus = 14 cpus = "0-15" The actual result would be: [ 3] Server Report: [ 3] 0.0-10.0 sec 58.9 MBytes 49.4 Mbits/sec 0.011 ms 7734/625000 (1.2%) the packet drop rate is lower than prior, but still high.
I *guess* this is by design. All VCPUs including the ones belonging to Domain0 are given the same priority by default for schedule. So in case of high CPU load, all VCPUs belonging to DomainU and Domain0 have to share the limited computing resources. Thus the performance of Domain0 and DomainU is not isolated. Xen does provide some kind of scheme for performance isolation, such as CPU WEIGHT and CAP. But it really depends. For the scenario in Description, you could either (1) Give lower weight to DomainU VCPUs via 'xm sched-credit' or (2) Give a proper CAP(also via xm sched-credit) to DomainU to make sure it couldn't consume too much cpu time. or (3) Simply reduce number of VCPUs of DomainU. All these three methods could make sure Domain0 VCPUs get more CPU time, and of course would have some impact on application running within DomainU. It depends on how you trade off. BTW, it is not recommended to run applications within Domain0.
I agree with Yufang that this isn't a bug. When the hypervisor is scheduling vcpus it can't know that some of them are trying to receive udp packets, which, if not received immediately, will get dropped. When you move the bzip into dom0 then the dom0 kernel can schedule more intelligently, and thus keep the udp receiving task active, avoiding packet loss. I would be a bit more concerned if you had packet loss while using tcp, but even then without the hypervisor being informed of vcpu priorities (using sched-credit as pointed out by Yufang), then the dom0 vcpus may not get the cycles they need to keep a tcp connection alive.