Bug 1384374
| Summary: | L2 network latency has higher 'Max latency value' with vhotuser mq in KVM-RT environment | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Pei Zhang <pezhang> |
| Component: | kernel-rt | Assignee: | pagupta |
| kernel-rt sub component: | KVM | QA Contact: | Virtualization Bugs <virt-bugs> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | bhu, chayang, hhuang, juzhang, knoel, michen, pezhang, virt-maint, xfu |
| Version: | 7.3 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-12-06 04:47:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hello Pei Zhang, In the configuration I can see OVS PMD threads are configured to run on isolated physical cores with mask 15554(2,4,6,8,10,14,16). Could you please check if PMD threads are pinned to separate isolated cores and fifo priority is assigned as mentioned below: In the host: Identify OVS PMD threads, pin then to (different) isolated cores and assigned FIFO:95 to them. You can run top as shown below to identify the PMD threads. Those are going to be threads taking 100% of the CPU and having names like "pmd54". # top -d1 -H (to identify the PMD threads) # taskset -cp CORE-NR TID (to pin the PMD thread to core CORE-NR) # chrt -fp 95 TID (to assign priority FIFO:95) If above step does not help, please check physical NIC's are bound to which numa node with below command and also get command line parameters of OpenVSwitch: # hwloc-ls -v # ps -ef | grep -i ovs-vswitchd Best regards, Pankaj (In reply to pagupta from comment #7) > Hello Pei Zhang, > > In the configuration I can see OVS PMD threads are configured to run on > isolated physical cores with mask 15554(2,4,6,8,10,14,16). Could you please > check if PMD threads are pinned to separate isolated cores and fifo priority > is assigned as mentioned below: > > In the host: Identify OVS PMD threads, pin then to (different) isolated > cores and assigned FIFO:95 to them. You can run top as shown below to > identify the PMD threads. Those are going to be threads taking 100% of the > CPU and having names like "pmd54". > > # top -d1 -H (to identify the PMD threads) > # taskset -cp CORE-NR TID (to pin the PMD thread to core CORE-NR) > # chrt -fp 95 TID (to assign priority FIFO:95) Pankaj, thanks for your suggestions. These cores are pinned to separate isolated cores, however I didn't assign priority FIFO:95 to all those 8 cores, that's why the max latency is high. I re-tested with these steps, now the latency is lower much. 2q-rt: running 12 hours min=10.325, aver=12.448, max=75.781 Best Regards, Pei > If above step does not help, please check physical NIC's are bound to which > numa node with below command and also get command line parameters of > OpenVSwitch: > > # hwloc-ls -v > # ps -ef | grep -i ovs-vswitchd > > Best regards, > Pankaj Hello Pei Zhang, Thanks for the confirmation. So, it matches/better then non-RT results in comment 0: //snippet from original issue Min(us) Avg(us) Max(us) 2q-rt: 10.322 11.375 6185.621 2q-nonrt: 10.212 11.695 1276.902 Thanks, Pankaj Closing this BZ as per comment 8. If you feel issue persists please reopen it. Thanks, Pankaj (In reply to pagupta from comment #9) > Hello Pei Zhang, > > Thanks for the confirmation. > > So, it matches/better then non-RT results in comment 0: Pankaj, I finished the 24 hours testing, the max latency still keeps lower. 2q-rt: running 24 hours min=10.274, aver=13.149, max=74.046 So I agree with you, it's not a bug. Thank you. Best regards, Pei > //snippet from original issue > Min(us) Avg(us) Max(us) > 2q-rt: 10.322 11.375 6185.621 > 2q-nonrt: 10.212 11.695 1276.902 > > > > Thanks, > Pankaj |
Description of problem: Boot guest with vhostuser 2 queues in KVM-RT environment, the L2 network max latency value is higher than non-rt. With vhotuser 1q, KVM-RT has lower max latency. So seems vhotuser mq in KVM-RT environment has worse max latency value. Version-Release number of selected component (if applicable): Host: 3.10.0-510.rt56.415.el7.x86_64 libvirt-2.0.0-10.el7.x86_64 qemu-kvm-rhev-2.6.0-28.el7.x86_64 openvswitch-dpdk-2.5.0-4.git20160628.el7.x86_64 Guest: 3.10.0-510.rt56.415.el7.x86_64 dpdk-16.07.tar.xz Host info: Beaker machine: dell-per730-27.lab.eng.pek2.redhat.com Dell Inc. PowerEdge R730 Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz 65536 MB memory 2 XL710/i40e NICs. How reproducible: Always Steps to Reproduce: 1. Prepare KVM-RT environment in host # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-510.rt56.415.el7.x86_64 root=/dev/mapper/rhel_dell--per730--27-root ro crashkernel=auto rd.lvm.lv=rhel_dell-per730-27/root rd.lvm.lv=rhel_dell-per730-27/swap console=ttyS0,115200n81 default_hugepagesz=1G iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21 intel_pstate=disable nosoftlockup 2. Start OVS-DPDK and set 2 queues, using 8 cores(2,4,6,8,10,12,14,16) for 8 pmds. # ovs-vsctl show d890ab54-7512-4a16-b8e0-5dad7a54d9c0 Bridge "ovsbr0" Port "vhost-user1" Interface "vhost-user1" type: dpdkvhostuser Port "dpdk0" Interface "dpdk0" type: dpdk Port "ovsbr0" Interface "ovsbr0" type: internal Bridge "ovsbr1" Port "dpdk1" Interface "dpdk1" type: dpdk Port "vhost-user2" Interface "vhost-user2" type: dpdkvhostuser Port "ovsbr1" Interface "ovsbr1" type: internal ovs-vsctl set Open_vSwitch . other_config={} ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2 ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x1 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x15554 3. Boot guest with 2 queues. Using 7 cores, 5 for running testpmd and are rt cores, 2 are for housekeeping. <domain type='kvm'> <name>rhel7.3-2q-rt</name> <uuid>ff8b540c-9ef3-4441-9b0d-da7c97543f23</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB'/> </hugepages> </memoryBacking> <vcpu placement='static'>7</vcpu> <cputune> <vcpupin vcpu='0' cpuset='31'/> <vcpupin vcpu='1' cpuset='30'/> <vcpupin vcpu='2' cpuset='28'/> <vcpupin vcpu='3' cpuset='26'/> <vcpupin vcpu='4' cpuset='24'/> <vcpupin vcpu='5' cpuset='22'/> <vcpupin vcpu='6' cpuset='29'/> <emulatorpin cpuset='1,3,5,7,9'/> <vcpusched vcpus='0-6' scheduler='fifo' priority='1'/> </cputune> <numatune> <memory mode='strict' nodeset='0-1'/> </numatune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pmu state='off'/> <vmport state='off'/> </features> <cpu mode='host-passthrough'> <feature policy='require' name='tsc-deadline'/> <numa> <cell id='0' cpus='0-6' memory='4194304' unit='KiB' memAccess='shared'/> </numa> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/pezhang/rhel7.3-rt.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='usb' index='0' model='none'/> <interface type='bridge'> <mac address='14:18:77:48:02:01'/> <source bridge='switch'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='14:18:77:48:02:02'/> <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/> <model type='virtio'/> <driver name='vhost' queues='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='14:18:77:48:02:03'/> <source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/> <model type='virtio'/> <driver name='vhost' queues='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </memballoon> </devices> 4. Did cyclictest 1 hours in guests for 5 rt vCPUs, their latency value shows as expected. vCPU1 vCPU2 vCPU3 vCPU4 vCPU5 Min 00004 00006 00006 00006 00005 Avg 00006 00006 00006 00006 00006 Max 00015 00017 00017 00017 00017 5. Start testpmd in guest # cat testpmd-2q.sh queues=2 cores=4 /root/dpdk-16.07/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -l 1,2,3,4,5 -n 1 -d /root/dpdk-16.07/x86_64-native-linuxapp-gcc/lib/librte_pmd_virtio.so \ -w 00:02.0 -w 00:06.0 \ -- \ --disable-hw-vlan -i \ --crc-strip \ --nb-cores=${cores} \ --disable-rss \ --rxq=${queues} --txq=${queues} \ --auto-start \ --rxd=256 --txd=256 \ # sh testpmd-2q.sh 6. Start MoonGen in another host, run 12 hours. Get a higher max latency value than non-rt. # ./build/MoonGen examples/l2-load-latency.lua 0 1 150 Actual results: The max latency value in KVM-RT is higher than non-rt. Min(us) Avg(us) Max(us) 2q-rt: 10.322 11.375 6185.621 2q-nonrt: 10.212 11.695 1276.902 Expected results: The max latency value should be lower with KVM-RT. Additional info: 1. With vhotuser 1q, the KVM-RT has lower latency which is expected. Tested with 2 different host: (1) dell-per730-27.lab.eng.pek2.redhat.com Host info: Dell Inc. PowerEdge R730 Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz 65536 MB memory 2 XL710/i40e NICs. 1q results: Min(us) Avg(us) Max(us) 1q-rt: 10.907 17.308 1681.247 1q-nonrt: 13.058 17.488 17038.267.0 (2) dell-per730-12.lab.eng.pek2.redhat.com Host info: Dell Inc. PowerEdge R730 (SKU=NotProvided;ModelName=PowerEdge R730) Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz 32768 MB memory 2 X540/ixgbe NICs. 1q results: Min(us) Avg(us) Max(us) 1q-rt: 11.289 58.560 819.684 1q-nonrt: 11.257 58.512 7928.230