Bug 2004173

Description of problem:
We are creating a new Openstack 16.2 job that runs first tempest and then rally [1] and we are testing it at staging Jenkins.

All tempest tests passed. One of the last tempest tests rebooted a controller node. After that, rally was executed and most of its tests failed. Many neutron API failures can be seen at [2] (it failed creating networks, security groups, network topoly and ports).

The tempest.log file [3] shows that the controller-0 node was stopped at 11:17:22 and started at 11:18:13:
2021-09-14 11:17:22.123 544380 INFO tempest_helper_plugin.common.utils.linux.node_power [-] Node 6863ca67-6905-4543-9065-d8b8c3108c98 in status:SHUTOFF power:Shutdown
...
2021-09-14 11:18:13.629 544380 INFO tempest_helper_plugin.common.utils.linux.node_power [-] Node 6863ca67-6905-4543-9065-d8b8c3108c98 in status:ACTIVE power:Running



Errors like the following one can be found at neutron server logs [4]. These errors occur during the execution of the rally tests:
2021-09-14 11:35:57.549 19 ERROR neutron.plugins.ml2.managers [req-2edd10e4-97b0-40ba-9bf8-58e4ed570eb2 c69aefde18594911bee885d4c523edb7 03db69baba564569ade87406ae195daf - default default] Mechanism driver 'ovn' failed in create_network_precommit: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1213, 'Deadlock: wsrep aborted transaction')


[root@controller-0 ~]# zgrep "wsrep aborted transaction" /var/log/containers/neutron/server.log.2.gz | wc -l
117

The last one happens at 12:05:06:
2021-09-14 12:05:06.422 18 ERROR neutron.api.v2.resource sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1213, 'Deadlock: wsrep aborted transaction')


The mysql logs on the controller-0 show only warning messages between 11:19 and 12:01 at [5]:
2021-09-14 11:48:58 0 [Warning] InnoDB: A long semaphore wait:
--Thread 139639470708480 has waited at lock0wait.cc line 96 for 241.00 seconds the semaphore:
Mutex at 0x55e7d17cad00, Mutex LOCK_SYS created lock0lock.cc:461, lock var 2







[1] https://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv6-geneve/2/
[2] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv6-geneve/2/infrared/.workspaces/workspace_2021-09-14_07-10-16/rally_results/detailed-20210914T113235.txt
[3] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv6-geneve/2/undercloud-0/home/stack/tempest-dir/tempest.log.gz
[4] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv6-geneve/2/controller-0/var/log/containers/neutron/server.log.1.gz
[5] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv6-geneve/2/controller-0/var/log/containers/mysql/mysqld.log.gz


Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20210903.n.1

How reproducible:
I have only executed the test once

Steps to Reproduce:
1. reboot a controller node and the run rally tests