Description of problem: OVS QOS linux htb netperf throughput below minimum queue rate Version-Release number of selected component (if applicable): RHEL-8.0.0 How reproducible: Always Steps to Reproduce: 1. Provision client and server with RHEL-8.0.0 2. Setup configuration using http://pkgs.devel.redhat.com/cgit/tests/kernel/tree/networking/openvswitch/ovs_qos/runtest.sh 3. Manually execute test steps and observe throughput below minimum queue rate that was set 4. See Additional info below for detailed test steps Actual results: Throughput falls below minimum queue rate that was set Expected results: Throughput meets or exceeds minimum queue rate that was set Additional info: Detailed test setup and steps: # provision systems (client and server) with RHEL-8.0.0 # set systems up for use with beaker: yum install -y http://download.eng.bos.redhat.com/beakerrepos/harness-testing/RedHatEnterpriseLinux8/beakerlib-1.17-19.el8bkr.noarch.rpm yum install -y http://download.eng.bos.redhat.com/beakerrepos/harness-testing/RedHatEnterpriseLinux8/restraint-0.1.37-1.el8bkr.x86_64.rpm yum install -y http://download.eng.bos.redhat.com/beakerrepos/harness-testing/RedHatEnterpriseLinux8/restraint-rhts-0.1.37-1.el8bkr.x86_64.rpm python_major_ver=$(rpm -q platform-python | awk -F "-" '{print $3}' | awk -F "." '{print $1}') python_pkg_ver=$(rpm -q platform-python | awk -F "-" '{print $3}' | awk -F "." '{print $1$2}') yum -y install python"$python_pkg_ver" yum -y install python"$python_major_ver"-tkinter git_install() { if rpm -q git 2>/dev/null; then echo "Git is already installed; doing a git pull"; cd /mnt/tests/kernel; git pull return 0 else yum -y install git mkdir /mnt/tests pushd /mnt/tests && git clone git://pkgs.devel.redhat.com/tests/kernel && popd fi } git_install alias python=/usr/libexec/platform-python . /mnt/tests/kernel/networking/common/include.sh # To set up the config, create VMs, etc, execute the following on both client and server: export OVS_TOPO=setup export NAY=yes export PVT=no export NIC_DRIVER=ixgbe export image_name=rhel8.0.qcow2 export RPM_OVS_SELINUX_EXTRA_POLICY="http://download-node-02.eng.bos.redhat.com/brewroot/packages/openvswitch-selinux-extra-policy/1.0/16.el8fdp/noarch/openvswitch-selinux-extra-policy-1.0-16.el8fdp.noarch.rpm" export RPM_OVS="http://download-node-02.eng.bos.redhat.com/brewroot/packages/openvswitch2.11/2.11.0/12.el8fdp/x86_64/openvswitch2.11-2.11.0-12.el8fdp.x86_64.rpm" export DISTRO_BUILD="RHEL-8.0.0" export CLIENTS=netqe5.knqe.lab.eng.bos.redhat.com export SERVERS=netqe6.knqe.lab.eng.bos.redhat.com export JOBID=12345 pushd /mnt/tests/kernel/networking/openvswitch/ovs_qos/ make run # wait for setup to complete on both systems, then proceed to below steps ON SERVER SYSTEM ONLY ########################################################################## execute on server only (netqe6): ovs-vsctl add-port ovsbr0 vnet1 -- set interface vnet1 ofport_request=5 ovs-vsctl add-port ovsbr0 vnet3 -- set interface vnet3 ofport_request=6 ovs-vsctl add-port ovsbr0 enp131s0f0 ovs-vsctl show total_max_egress_rate=200000000 q1_max_egress_rate=20000000 q2_max_egress_rate=40000000 q1_max_egress_rate=20000000 total_min_egress_rate=100000000 q1_min_egress_rate=10000000 q2_min_egress_rate=20000000 ovs-vsctl set port enp131s0f0 qos=@newqos -- --id=@newqos create qos type=linux-htb other-config:max-rate=200000000 other-config:min-rate=100000000 queues:123=@tap1queue queues:234=@tap2queue -- --id=@tap1queue create queue other-config:max-rate=20000000 other-config:min-rate=10000000 -- --id=@tap2queue create queue other-config:max-rate=40000000 other-config:min-rate=20000000 ovs-ofctl add-flow ovsbr0 in_port=5,actions=set_queue:123,normal ovs-ofctl add-flow ovsbr0 in_port=6,actions=set_queue:234,normal ovs-ofctl dump-flows ovsbr0 # connect to vm1, execute netperf commands: virsh console vm1 netperf -4 -t UDP_STREAM -H 172.31.115.1 -l 30 -- -m 10000 [root@localhost ~]# netperf -4 -t UDP_STREAM -H 172.31.115.1 -l 30 -- -m 10000 MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.31.115.1 () port 0 AF_INET Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 10000 30.00 3424366 0 9131.63 212992 30.00 143 0.38 root@localhost ~]# netperf -6 -t UDP_STREAM -H 2001:0db8:115::1 -l 30 -- -m 1000 MIGRATED UDP STREAM TEST from ::0 (::) port 0 AF_INET6 to 2001:0db8:115::1 () port 0 AF_INET6 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 10000 30.00 2861898 0 7631.72 212992 30.00 143 0.38 # connect to vm2, execute netperf commands: virsh console vm2 [root@localhost ~]# netperf -4 -t UDP_STREAM -H 172.31.115.1 -l 30 -- -m 10000 MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.31.115.1 () port 0 AF_INET Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 10000 30.00 3377949 0 9007.86 212992 30.00 143 0.38 00oot@localhost ~]# netperf -6 -t UDP_STREAM -H 2001:0db8:115::1 -l 30 -- -m 1000 MIGRATED UDP STREAM TEST from ::0 (::) port 0 AF_INET6 to 2001:0db8:115::1 () port 0 AF_INET6 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 10000 30.00 2856810 0 7618.15 212992 30.00 143 0.38 # Note that throughput is only 0.38 for the above netperf runs and falls below min queue rate. The other OVS QOS tests (ingress policing rate and ingress policing burst) PASSED. ########## Re-test using using compose RHEL-7.7-20190612.0 (kernel 3.10.0-1055.el7.x86_64) and openvswitch2.11-2.11.0-12.el7fdp.x86_64 # The same setup and test using compose RHEL-7.7-20190612.0 (kernel 3.10.0-1055.el7.x86_64) and openvswitch2.11-2.11.0-12.el7fdp.x86_64 yielded better throughput and did not fall below min queue rate: virsh console vm1 [root@localhost ~]# netperf -4 -t UDP_STREAM -H 172.31.115.1 -l 30 -- -m 10000 MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.31.115.1 () port 0 AF_INET Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 10000 30.00 6067191 0 16179.12 212992 30.00 8258 22.02 root@localhost ~]# netperf -6 -t UDP_STREAM -H 2001:0db8:115::1 -l 30 -- -m 100 MIGRATED UDP STREAM TEST from ::0 (::) port 0 AF_INET6 to 2001:0db8:115::1 () port 0 AF_INET6 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 1000 30.00 5660950 0 1509.58 212992 30.00 71623 19.10 virsh console vm2 [root@localhost ~]# netperf -4 -t UDP_STREAM -H 172.31.115.1 -l 30 -- -m 10000 MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.31.115.1 () port 0 AF_INET Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 10000 30.00 5783864 0 15423.57 212992 30.00 15572 41.53 root@localhost ~]# netperf -6 -t UDP_STREAM -H 2001:0db8:115::1 -l 30 -- -m 100 MIGRATED UDP STREAM TEST from ::0 (::) port 0 AF_INET6 to 2001:0db8:115::1 () port 0 AF_INET6 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec 212992 1000 30.00 5453902 0 1454.37 212992 30.00 142245 37.93
Link to failed beaker job: https://beaker.engineering.redhat.com/jobs/3608026
I have re-tested this on RHEL 8.0.0 but this time used a RHEL 7.6 guest image instead of RHEL 8.0. The lower throughput problem was not present so maybe there is a problem with the RHEL 8.0 guest image I was using. I will try creating another 8.0 image and will re-test.
After making some modifications to the test script related to guest image, the test now passes. Beaker job: https://beaker.engineering.redhat.com/jobs/3619163
Closing this as the original issue appears to be related to the guest image that was used.