Created attachment 1929005 [details] Tests ordered by duration Description of problem: We have several tests that take a of time. After some measurements I have identified ~15 (~45 considering permutations) tests that together take ~35% of overall test time. By optimizing those we could cut the overall CI time at least for the unit tests. In the attachment is full list of all tests order by the duration. Down below are tests that are running longer than 5 seconds. One thing to keep in mind is that the duration may vary between environments but it should be proportional. ---------------------------------- Larger than 5 seconds ---------------------------------- 1020. northd-parallelization runtime -- ovn-northd -- parallelization=yes (ovn-northd.at:7428): Ok (17.99s) 1021. northd-parallelization runtime -- ovn-northd -- parallelization=no (ovn-northd.at:7428): Ok (17.89s) 399. ACLs on Port Groups -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:16466): Ok (17.792s) 400. ACLs on Port Groups -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:16466): Ok (17.649s) 401. ACLs on Port Groups -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:16466): Ok (17.609s) 402. ACLs on Port Groups -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:16466): Ok (17.455s) 565. interconnection -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:24498): Ok (15.347s) 566. interconnection -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:24498): Ok (15.271s) 563. interconnection -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:24498): Ok (15.234s) 564. interconnection -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:24498): Ok (15.089s) 1040. Check default drop -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn-northd.at:8189): Ok (12.628s) 1038. Check default drop -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn-northd.at:8189): Ok (12.608s) 1039. Check default drop -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn-northd.at:8189): Ok (12.57s) 1041. Check default drop -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn-northd.at:8189): Ok (12.532s) 119. 3 HVs, 3 LS, 3 lports/LS, 1 LR -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:4579): Ok (10.736s) 121. 3 HVs, 3 LS, 3 lports/LS, 1 LR -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:4579): Ok (10.692s) 122. 3 HVs, 3 LS, 3 lports/LS, 1 LR -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:4579): Ok (10.598s) 120. 3 HVs, 3 LS, 3 lports/LS, 1 LR -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:4579): Ok (10.596s) 24. 4-term numeric expression normalization (ovn.at:624): Ok (8.756s) 397. Port Groups -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:16246): Ok (7.77s) 396. Port Groups -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:16246): Ok (7.733s) 398. Port Groups -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:16246): Ok (7.685s) 395. Port Groups -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:16246): Ok (7.477s) 502. IGMP snoop/querier/relay -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:20995): Ok (5.905s) 501. IGMP snoop/querier/relay -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:20995): Ok (5.834s) 507. MLD snoop/querier/relay -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:21717): Ok (5.715s) 508. MLD snoop/querier/relay -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:21717): Ok (5.692s) 355. options:activation-strategy for logical port -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:15011): Ok (5.687s) 356. options:activation-strategy for logical port -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:15011): Ok (5.607s) 500. IGMP snoop/querier/relay -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:20995): Ok (5.515s) 357. options:activation-strategy for logical port -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:15011): Ok (5.503s) 509. MLD snoop/querier/relay -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:21717): Ok (5.493s) 499. IGMP snoop/querier/relay -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:20995): Ok (5.466s) 358. options:activation-strategy for logical port -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:15011): Ok (5.457s) 510. MLD snoop/querier/relay -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:21717): Ok (5.426s) 27. 5-term numeric expression normalization (ovn.at:642): Ok (5.326s) 19. 3-term numeric expression simplification (ovn.at:570): Ok (5.261s) 678. ARP replies for SNAT external ips -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no (ovn.at:28040): Ok (5.184s) 677. ARP replies for SNAT external ips -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:28040): Ok (5.12s) 50. 3 HVs, 1 LS, 3 lports/HV -- ovn-northd -- parallelization=no -- ovn_monitor_all=no (ovn.at:2303): Ok (5.045s) 49. 3 HVs, 1 LS, 3 lports/HV -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes (ovn.at:2303): Ok (5.045s) 47. 3 HVs, 1 LS, 3 lports/HV -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:2303): Ok (5.029s) 459. router - check packet length - icmp defrag -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:19051): Ok (5.016s) 1169. ovn-controller - port binding type change handling (ovn-controller.at:534): Ok (5.008s) Total duration: 413.041s
For some reason the times are completely off for system tests. So we would require different method to analyze them.
After some discussion we will tag those slowest one that don't have obvious "fix" with "slowtest".
slowtest mark: https://patchwork.ozlabs.org/project/ovn/patch/20230110083502.2656502-1-amusil@redhat.com/ some optimizations: https://patchwork.ozlabs.org/project/ovn/list/?series=332641
All relevant patches have been merged closing as UPSTREAM.