Description of problem: periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 is failing on neutron tempest tests. Version-Release number of selected component (if applicable): 16.2 How reproducible: Frequently in 16.2 Integration line Additional info: Tempest failure:- ~~~ Captured traceback: ~~~~~~~~~~~~~~~~~~~ b'Traceback (most recent call last):' b' File "/usr/lib/python3.6/site-packages/tempest/api/network/test_networks.py", line 474, in test_bulk_create_delete_network' b' body = self.networks_client.create_bulk_networks(networks=network_list)' b' File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/networks_client.py", line 72, in create_bulk_networks' b' return self.create_resource(uri, kwargs)' b' File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/base.py", line 62, in create_resource' b' resp, body = self.post(req_uri, req_post_data)' b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 300, in post' b" return self.request('POST', url, extra_headers, headers, body, chunked)" b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 704, in request' b' self._error_checker(resp, resp_body)' b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 881, in _error_checker' b' message=message)' b'tempest.lib.exceptions.ServerFault: Got server fault' b'Details: The server has either erred or is incapable of performing the requested operation.<br /><br />' b'' b'' b'' b'' Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ b'2021-09-13 20:20:01,253 197183 INFO [tempest.lib.common.rest_client] Request (BulkNetworkOpsTest:test_bulk_create_delete_network): 500 POST http://192.168.24.1:9696/v2.0/networks 5.819s' b"2021-09-13 20:20:01,254 197183 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}" b' Body: {"networks": [{"name": "tempest-network--484652455"}, {"name": "tempest-network--1692691788"}]}' b" Response - Headers: {'content-length': '186', 'content-type': 'application/json', 'x-openstack-request-id': 'req-22573951-41af-458b-96bb-795f486bc8ea', 'date': 'Mon, 13 Sep 2021 20:20:01 GMT', 'connection': 'close', 'status': '500', 'content-location': 'http://192.168.24.1:9696/v2.0/networks'}" b' Body: b\'{"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\\\\n\\\\n\\\\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}\'' b'' ~~~ Neutron server.log ~~~ 2021-09-13 20:20:01.230 ERROR /var/log/containers/neutron/server.log: 16 ERROR oslo_middleware.catch_errors sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') ~~~
Fix https://review.opendev.org/c/openstack/neutron/+/809191 seems to be working. All tempest tests on the reproducer env prepared by Sandeep Yada passed with that patch. I will wait for Rodolfo's review of that patch now.
According to our records, this should be resolved by openstack-neutron-15.3.5-2.20210608154816.el8ost.4. This build is available now.