Bug 2003921

Summary: periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 is failing on neutron tempest tests with following error in Neutron logs: "ERROR oslo_middleware.catch_errors sqlalchemy.exc.InvalidRequestError:
Product: Red Hat OpenStack Reporter: Sandeep Yadav <sandyada>
Component: openstack-neutronAssignee: Slawek Kaplonski <skaplons>
Status: CLOSED CURRENTRELEASE QA Contact: Alex Katz <akatz>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.2 (Train)CC: ccamposr, chrisw, ekuris, scohen, skaplons
Target Milestone: z2Keywords: TestOnly, Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-15.3.5-2.20210608154815.el8ost.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2008845 (view as bug list) Environment:
Last Closed: 2022-01-24 11:38:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2008845    

Description Sandeep Yadav 2021-09-14 07:24:57 UTC
Description of problem:

periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 is failing on neutron tempest tests.

Version-Release number of selected component (if applicable):
16.2

How reproducible:

Frequently in 16.2 Integration line

Additional info:

Tempest failure:-
~~~
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b'  File "/usr/lib/python3.6/site-packages/tempest/api/network/test_networks.py", line 474, in test_bulk_create_delete_network'
    b'    body = self.networks_client.create_bulk_networks(networks=network_list)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/networks_client.py", line 72, in create_bulk_networks'
    b'    return self.create_resource(uri, kwargs)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/base.py", line 62, in create_resource'
    b'    resp, body = self.post(req_uri, req_post_data)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 300, in post'
    b"    return self.request('POST', url, extra_headers, headers, body, chunked)"
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 704, in request'
    b'    self._error_checker(resp, resp_body)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 881, in _error_checker'
    b'    message=message)'
    b'tempest.lib.exceptions.ServerFault: Got server fault'
    b'Details: The server has either erred or is incapable of performing the requested operation.<br /><br />'
    b''
    b''
    b''
    b''

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    b'2021-09-13 20:20:01,253 197183 INFO     [tempest.lib.common.rest_client] Request (BulkNetworkOpsTest:test_bulk_create_delete_network): 500 POST http://192.168.24.1:9696/v2.0/networks 5.819s'
    b"2021-09-13 20:20:01,254 197183 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}"
    b'        Body: {"networks": [{"name": "tempest-network--484652455"}, {"name": "tempest-network--1692691788"}]}'
    b"    Response - Headers: {'content-length': '186', 'content-type': 'application/json', 'x-openstack-request-id': 'req-22573951-41af-458b-96bb-795f486bc8ea', 'date': 'Mon, 13 Sep 2021 20:20:01 GMT', 'connection': 'close', 'status': '500', 'content-location': 'http://192.168.24.1:9696/v2.0/networks'}"
    b'        Body: b\'{"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\\\\n\\\\n\\\\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}\''
    b''
~~~


Neutron server.log
~~~
2021-09-13 20:20:01.230 ERROR /var/log/containers/neutron/server.log: 16 ERROR oslo_middleware.catch_errors sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction')
~~~

Comment 3 Slawek Kaplonski 2021-09-16 13:14:51 UTC
Fix https://review.opendev.org/c/openstack/neutron/+/809191 seems to be working. All tempest tests on the reproducer env prepared by Sandeep Yada passed with that patch. I will wait for Rodolfo's review of that patch now.

Comment 8 Lon Hohberger 2021-12-22 16:18:00 UTC
According to our records, this should be resolved by openstack-neutron-15.3.5-2.20210608154816.el8ost.4.  This build is available now.