Description of problem: create_network_nova_boot_ping scenario[1] failing while checking SLA as ping started taking much time Since Puddle RHOS-16.2-RHEL-8-20220201.n.1. Last good Puddle was RHOS-16.2-RHEL-8-20211129.n.1, so multiple changes were introduced with the puddle in 2 months. Trend Change can be seen b/w build 43[2] and 44[3] of job DFG-perfscale-PerfCI-OSP16.2-neutron-ovn. 42 - vm.wait_for_ping (x4) 0.052 0.059 0.066 0.071 0.075 0.06 100.0% 10 43 - vm.wait_for_ping (x4) 0.046 0.056 0.061 0.072 0.082 0.057 100.0% 10 44 - vm.wait_for_ping (x4) 0.049 0.772 35.345 36.16 36.975 8.137 100.0% 10 45 - vm.wait_for_ping (x4) 0.052 2.248 59.315 60.471 61.627 18.789 100.0% 10 Job History:- https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/perfscale/view/PerfCI/job/DFG-perfscale-PerfCI-OSP16.2-neutron-ovn/ Version-Release number of selected component (if applicable): There were many rpm changes but just pasting few related to start with:- Good Build:- python3-networking-ovn-7.4.2-2.20210601204831.el8ost.13.noarch openstack-neutron-15.3.5-2.20210608154816.el8ost.4.noarch ovn-2021-21.09.0-20.el8fdp.x86_64 openvswitch2.15-2.15.0-38.el8fdp.x86_64 kernel-4.18.0-305.25.1.el8_4.x86_64 Bad build:- python3-networking-ovn-7.4.2-2.20220113214852.a2eba10.el8ost.noarch openstack-neutron-15.3.5-2.20220113150030.94e4cbb.el8ost.noarch ovn-2021-21.12.0-11.el8fdp.x86_64 openvswitch2.15-2.15.0-57.el8fdp.x86_64 kernel-4.18.0-305.34.2.el8_4.x86_64 How reproducible: Always Steps to Reproduce: 1. Reproducible always with Perf CI job DFG-perfscale-PerfCI-OSP16.2-neutron-ovn 2. Performance difference can be seen with manual run of browbeat scenario in 16.1 and 16.2 deployments Actual results: Performance Dropped of scenario as compared to 16.1 equivalent and 16.2 after compose RHOS-16.2-RHEL-8-20220201.n.1. Expected results: Performance shouldn't have dropped that much. Additional info: 16.1 OVN Perf job[4] is all good, It's a regression in 16.2. 54 - vm.wait_for_ping (x4) 0.048 0.054 0.067 0.067 0.068 0.057 100.0% 10 55 - vm.wait_for_ping (x4) 0.047 0.058 0.063 0.063 0.064 0.056 100.0% 10 It's likely same issue as [5] but filing seperate bz just to not mix both issues . Also this is more to track performance aspect and rule out non-ovn related changes. Can be marked duplicate once it's confirmed post further investigation. [1] https://opendev.org/x/browbeat/src/branch/master/rally/rally-plugins/netcreate-boot/netcreate_nova_boot_fip_ping.py [2] https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-perfscale-PerfCI-OSP16.2-neutron-ovn/43/artifact/rally-results.html#/BrowbeatPlugin.create_network_nova_boot_ping [3] https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-perfscale-PerfCI-OSP16.2-neutron-ovn/44/artifact/rally-results.html#/BrowbeatPlugin.create_network_nova_boot_ping [4] https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/perfscale/view/PerfCI/job/DFG-perfscale-PerfCI-OSP16.1-neutron-ovn/ [5] https://bugzilla.redhat.com/show_bug.cgi?id=2069668