RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1384374 - L2 network latency has higher 'Max latency value' with vhotuser mq in KVM-RT environment
Summary: L2 network latency has higher 'Max latency value' with vhotuser mq in KVM-RT ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel-rt
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: pagupta
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-13 08:05 UTC by Pei Zhang
Modified: 2016-12-06 10:53 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-06 04:47:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Pei Zhang 2016-10-13 08:05:36 UTC 
Description of problem:
Boot guest with vhostuser 2 queues in KVM-RT environment, the L2 network max latency value is higher than non-rt.

With vhotuser 1q, KVM-RT has lower max latency.

So seems vhotuser mq in KVM-RT environment has worse max latency value.

Version-Release number of selected component (if applicable):
Host:
3.10.0-510.rt56.415.el7.x86_64
libvirt-2.0.0-10.el7.x86_64
qemu-kvm-rhev-2.6.0-28.el7.x86_64
openvswitch-dpdk-2.5.0-4.git20160628.el7.x86_64

Guest:
3.10.0-510.rt56.415.el7.x86_64
dpdk-16.07.tar.xz

Host info:
Beaker machine: dell-per730-27.lab.eng.pek2.redhat.com
Dell Inc. PowerEdge R730
Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz
65536 MB memory
2 XL710/i40e NICs.

How reproducible:
Always

Steps to Reproduce:
1. Prepare KVM-RT environment in host
# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-510.rt56.415.el7.x86_64 root=/dev/mapper/rhel_dell--per730--27-root ro crashkernel=auto rd.lvm.lv=rhel_dell-per730-27/root rd.lvm.lv=rhel_dell-per730-27/swap console=ttyS0,115200n81 default_hugepagesz=1G iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21 rcu_nocbs=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21 intel_pstate=disable nosoftlockup

2. Start OVS-DPDK and set 2 queues, using 8 cores(2,4,6,8,10,12,14,16) for 8 pmds.
# ovs-vsctl show
d890ab54-7512-4a16-b8e0-5dad7a54d9c0
    Bridge "ovsbr0"
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    Bridge "ovsbr1"
        Port "dpdk1"
            Interface "dpdk1"
                type: dpdk
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal

ovs-vsctl set Open_vSwitch . other_config={} 
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x1
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x15554


3. Boot guest with 2 queues. Using 7 cores, 5 for running testpmd and are rt cores, 2 are for housekeeping.

<domain type='kvm'>
  <name>rhel7.3-2q-rt</name>
  <uuid>ff8b540c-9ef3-4441-9b0d-da7c97543f23</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>7</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='31'/>
    <vcpupin vcpu='1' cpuset='30'/>
    <vcpupin vcpu='2' cpuset='28'/>
    <vcpupin vcpu='3' cpuset='26'/>
    <vcpupin vcpu='4' cpuset='24'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='29'/>
    <emulatorpin cpuset='1,3,5,7,9'/>
    <vcpusched vcpus='0-6' scheduler='fifo' priority='1'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-6' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/pezhang/rhel7.3-rt.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='none'/>
    <interface type='bridge'>
      <mac address='14:18:77:48:02:01'/>
      <source bridge='switch'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='14:18:77:48:02:02'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='14:18:77:48:02:03'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </memballoon>
  </devices>

4. Did cyclictest 1 hours in guests for 5 rt vCPUs, their latency value shows as expected.
    vCPU1  vCPU2  vCPU3  vCPU4  vCPU5
Min 00004  00006  00006  00006  00005 
Avg 00006  00006  00006  00006  00006
Max 00015  00017  00017  00017  00017

5. Start testpmd in guest
# cat testpmd-2q.sh 
queues=2
cores=4
/root/dpdk-16.07/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -l 1,2,3,4,5 -n 1 -d  /root/dpdk-16.07/x86_64-native-linuxapp-gcc/lib/librte_pmd_virtio.so \
-w 00:02.0 -w 00:06.0 \
-- \
--disable-hw-vlan -i \
--crc-strip \
--nb-cores=${cores} \
--disable-rss \
--rxq=${queues} --txq=${queues} \
--auto-start \
--rxd=256 --txd=256 \

# sh testpmd-2q.sh

6. Start MoonGen in another host, run 12 hours. Get a higher max latency value than non-rt.
# ./build/MoonGen examples/l2-load-latency.lua 0 1 150


Actual results:
The max latency value in KVM-RT is higher than non-rt.

              Min(us)    Avg(us)       Max(us)
2q-rt:        10.322      11.375       6185.621
2q-nonrt:     10.212      11.695       1276.902


Expected results:
The max latency value should be lower with KVM-RT.


Additional info:
1. With vhotuser 1q, the KVM-RT has lower latency which is expected. Tested with 2 different host:

(1) dell-per730-27.lab.eng.pek2.redhat.com
Host info:
Dell Inc. PowerEdge R730
Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz
65536 MB memory
2 XL710/i40e NICs.

1q results:
          Min(us)      Avg(us)         Max(us)
1q-rt:    10.907        17.308         1681.247
1q-nonrt: 13.058        17.488        17038.267.0

(2) dell-per730-12.lab.eng.pek2.redhat.com
Host info:
Dell Inc. PowerEdge R730 (SKU=NotProvided;ModelName=PowerEdge R730)
Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
32768 MB memory
2 X540/ixgbe NICs.

1q results:
          Min(us)      Avg(us)         Max(us)
1q-rt:    11.289        58.560         819.684
1q-nonrt: 11.257        58.512        7928.230
Comment 7 pagupta 2016-11-14 10:25:52 UTC 
Hello Pei Zhang,

In the configuration I can see OVS PMD threads are configured to run on isolated physical cores with mask 15554(2,4,6,8,10,14,16). Could you please check if PMD threads are pinned to separate isolated cores and fifo priority is assigned as mentioned below:

In the host: Identify OVS PMD threads, pin then to (different) isolated cores and assigned FIFO:95 to them. You can run top as shown below to identify the PMD threads. Those are going to be threads taking 100% of the CPU and having names like "pmd54".

# top -d1 -H              (to identify the PMD threads)
# taskset -cp CORE-NR TID (to pin the PMD thread to core CORE-NR) 
# chrt -fp 95 TID         (to assign priority FIFO:95)

If above step does not help, please check physical NIC's are bound to which numa node with below command and also get command line parameters of OpenVSwitch:

# hwloc-ls -v
# ps -ef | grep -i ovs-vswitchd

Best regards,
Pankaj
Comment 8 Pei Zhang 2016-12-02 08:01:04 UTC 
(In reply to pagupta from comment #7)
> Hello Pei Zhang,
> 
> In the configuration I can see OVS PMD threads are configured to run on
> isolated physical cores with mask 15554(2,4,6,8,10,14,16). Could you please
> check if PMD threads are pinned to separate isolated cores and fifo priority
> is assigned as mentioned below:
> 
> In the host: Identify OVS PMD threads, pin then to (different) isolated
> cores and assigned FIFO:95 to them. You can run top as shown below to
> identify the PMD threads. Those are going to be threads taking 100% of the
> CPU and having names like "pmd54".
> 
> # top -d1 -H              (to identify the PMD threads)
> # taskset -cp CORE-NR TID (to pin the PMD thread to core CORE-NR) 
> # chrt -fp 95 TID         (to assign priority FIFO:95)

Pankaj, thanks for your suggestions. These cores are pinned to separate isolated cores, however I didn't assign priority FIFO:95 to all those 8 cores, that's why the max latency is high. I re-tested with these steps, now the latency is lower much.

2q-rt: running 12 hours

min=10.325, aver=12.448, max=75.781


Best Regards,
Pei

> If above step does not help, please check physical NIC's are bound to which
> numa node with below command and also get command line parameters of
> OpenVSwitch:
> 
> # hwloc-ls -v
> # ps -ef | grep -i ovs-vswitchd
> 
> Best regards,
> Pankaj
Comment 9 pagupta 2016-12-05 06:19:07 UTC 
Hello Pei Zhang,

Thanks for the confirmation.

So, it matches/better then non-RT results in comment 0: 

//snippet from original issue
              Min(us)    Avg(us)       Max(us)
2q-rt:        10.322      11.375       6185.621
2q-nonrt:     10.212      11.695       1276.902



Thanks,
Pankaj
Comment 10 pagupta 2016-12-06 04:47:27 UTC 
Closing this BZ as per comment 8. If you feel issue persists please reopen it.

Thanks,
Pankaj
Comment 11 Pei Zhang 2016-12-06 10:53:26 UTC 
(In reply to pagupta from comment #9)
> Hello Pei Zhang,
> 
> Thanks for the confirmation.
> 
> So, it matches/better then non-RT results in comment 0: 

Pankaj, I finished the 24 hours testing, the max latency still keeps lower.

2q-rt: running 24 hours
min=10.274, aver=13.149, max=74.046

So I agree with you, it's not a bug. Thank you.


Best regards,
Pei


> //snippet from original issue
>               Min(us)    Avg(us)       Max(us)
> 2q-rt:        10.322      11.375       6185.621
> 2q-nonrt:     10.212      11.695       1276.902
> 
> 
> 
> Thanks,
> Pankaj
Note You need to log in before you can comment on or make changes to this bug.