Bug 2003921 - periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 is failing on neutron tempest tests with following error in Neutron logs: "ERROR oslo_middleware.catch_errors sqlalchemy.exc.InvalidRequestError:
Summary: periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 is failing o...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 16.2 (Train)
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: z2
: 16.2 (Train on RHEL 8.4)
Assignee: Slawek Kaplonski
QA Contact: Alex Katz
URL:
Whiteboard:
Depends On:
Blocks: 2008845
TreeView+ depends on / blocked
 
Reported: 2021-09-14 07:24 UTC by Sandeep Yadav
Modified: 2022-01-24 11:38 UTC (History)
5 users (show)

Fixed In Version: openstack-neutron-15.3.5-2.20210608154815.el8ost.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2008845 (view as bug list)
Environment:
Last Closed: 2022-01-24 11:38:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1943714 0 None None None 2021-09-15 13:02:47 UTC
OpenStack gerrit 809191 0 None NEW Rollback db session in case of error during releasing quota reservation 2021-09-15 13:57:32 UTC
Red Hat Issue Tracker OSP-9571 0 None None None 2021-11-15 12:50:30 UTC

Internal Links: 2007724

Description Sandeep Yadav 2021-09-14 07:24:57 UTC
Description of problem:

periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 is failing on neutron tempest tests.

Version-Release number of selected component (if applicable):
16.2

How reproducible:

Frequently in 16.2 Integration line

Additional info:

Tempest failure:-
~~~
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b'  File "/usr/lib/python3.6/site-packages/tempest/api/network/test_networks.py", line 474, in test_bulk_create_delete_network'
    b'    body = self.networks_client.create_bulk_networks(networks=network_list)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/networks_client.py", line 72, in create_bulk_networks'
    b'    return self.create_resource(uri, kwargs)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/base.py", line 62, in create_resource'
    b'    resp, body = self.post(req_uri, req_post_data)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 300, in post'
    b"    return self.request('POST', url, extra_headers, headers, body, chunked)"
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 704, in request'
    b'    self._error_checker(resp, resp_body)'
    b'  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 881, in _error_checker'
    b'    message=message)'
    b'tempest.lib.exceptions.ServerFault: Got server fault'
    b'Details: The server has either erred or is incapable of performing the requested operation.<br /><br />'
    b''
    b''
    b''
    b''

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    b'2021-09-13 20:20:01,253 197183 INFO     [tempest.lib.common.rest_client] Request (BulkNetworkOpsTest:test_bulk_create_delete_network): 500 POST http://192.168.24.1:9696/v2.0/networks 5.819s'
    b"2021-09-13 20:20:01,254 197183 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}"
    b'        Body: {"networks": [{"name": "tempest-network--484652455"}, {"name": "tempest-network--1692691788"}]}'
    b"    Response - Headers: {'content-length': '186', 'content-type': 'application/json', 'x-openstack-request-id': 'req-22573951-41af-458b-96bb-795f486bc8ea', 'date': 'Mon, 13 Sep 2021 20:20:01 GMT', 'connection': 'close', 'status': '500', 'content-location': 'http://192.168.24.1:9696/v2.0/networks'}"
    b'        Body: b\'{"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\\\\n\\\\n\\\\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}\''
    b''
~~~


Neutron server.log
~~~
2021-09-13 20:20:01.230 ERROR /var/log/containers/neutron/server.log: 16 ERROR oslo_middleware.catch_errors sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction')
~~~

Comment 3 Slawek Kaplonski 2021-09-16 13:14:51 UTC
Fix https://review.opendev.org/c/openstack/neutron/+/809191 seems to be working. All tempest tests on the reproducer env prepared by Sandeep Yada passed with that patch. I will wait for Rodolfo's review of that patch now.

Comment 8 Lon Hohberger 2021-12-22 16:18:00 UTC
According to our records, this should be resolved by openstack-neutron-15.3.5-2.20210608154816.el8ost.4.  This build is available now.


Note You need to log in before you can comment on or make changes to this bug.