Bug 1797229
Summary: | Neutron internal server error (HTTP 500) on port create | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Carlos Goncalves <cgoncalves> |
Component: | python-ovsdbapp | Assignee: | Rodolfo Alonso <ralonsoh> |
Status: | CLOSED ERRATA | QA Contact: | Eduardo Olivares <eolivare> |
Severity: | medium | Docs Contact: | |
Priority: | high | ||
Version: | 16.0 (Train) | CC: | amoralej, amuller, bcafarel, chrisw, ekuris, ffernand, gthiemon, itbrown, jlibosva, njohnston, rsafrono, scohen, twilson |
Target Milestone: | beta | Keywords: | Automation, AutomationBlocker, Reopened, Triaged |
Target Release: | 16.1 (Train on RHEL 8.2) | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | python-ovsdbapp-0.17.3-0.20200528074611.9bb0eee.el8ost.noarch.rpm | Doc Type: | No Doc Update |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-08-11 17:00:12 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Carlos Goncalves
2020-02-01 17:40:34 UTC
The cause is that an operation fails for one of the controllers in a role that is run earlier, octava-controller-config: 2020-02-01 12:01:13,689 p=226016 u=root | fatal: [controller-2]: FAILED! => {"changed": false, "cmd": "set -o pipefail\nport_id=\"$(openstack port show octavia-health-manager-controller-2. redhat.local-listen-port -f value -c id 2> /dev/null)\"\nif [[ -z \"${port_id}\" ]]; then\n neutron port-create lb-mgmt-net --binding:host_id=controller-2.redhat. local --no-security-groups --port-security-enabled=False -- device-owner Octavia:health-mgr --name octavia-health-manager-controller-2.redhat.local-listen-port -f value -c id\nfi", "delta": "0:00:07.065055", "end": "2020-02-01 12:01:13.649635", "msg": "non-zero return code", "rc": 1, "start": "2020-02-01 12:01: 06.584580", "stderr": "neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.\nRequest Failed: internal server error while processing your request.\nNeutron server returns request_ids: ['req-0c0b3b90-04be-486d-bb37-e23303ffe571']", "stderr_lines": ["neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.", "Request Failed: internal server error while processing your request.", "Neutron server returns request_ids: ['req-0c0b3b90-04be-486d-bb37-e23303ffe571']"], "stdout": "", "stdout_lines": []} This results in o-hm-ip not being set for that node and why it cannot be referenced later. There are probably several issues here: - Why the internal error in neutron for one of the nodes and not the others? - Why did the ansible continue? Is that by design? - If yes to above, we need to make Octavia deployment more resilient to apparently transient errors. Even so, completely failure to configure resources (i.e. unable to achieve a working overcloud) should fail the deploy. Reopening as we are seeing this again in downstream CI. Neutron is raising MechanismDriverError and returning HTTP 500 to user on port create. 2020-03-28 11:55:46.972 30 INFO neutron.wsgi [req-1a9f51ae-a7de-4311-93cf-86b48b4ad65f 02ef7843a642467091f63be0a9128c72 a2b2e6613a6e46bd8d063bc7db0e6bc6 - default default] 172.17.1.107 "POST /v2.0/ports HTTP/1.1" status: 201 len: 1257 ti me: 1.9680293 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation [req-141a2cbb-86b2-449e-9188-f3a9dae3abca 02ef7843a642467091f63be0a9128c72 a2b2e6613a6e46bd8d063bc7db0e6bc6 - default default] POST failed.: neutron.plugins.ml2.common. exceptions.MechanismDriverError 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 683, in __call__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/pecan/core.py", line 574, in invoke_controller 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 163, in post 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 181, in create 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)} 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 685, in inner 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1430, in create_port 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation return self._after_create_port(context, result, mech_context) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1443, in _after_create_port 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.delete_port(context, result['id'], l3_port_check=False) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation raise value 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1438, in _after_create_port 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self.mechanism_manager.create_port_postcommit(mech_context) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 715, in create_port_postcommit 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation self._call_on_drivers("create_port_postcommit", context) 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation errors=errors 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError 2020-03-28 11:55:46.982 29 ERROR neutron.pecan_wsgi.hooks.translation 2020-03-28 11:55:46.989 29 DEBUG neutron.pecan_wsgi.hooks.notifier [req-141a2cbb-86b2-449e-9188-f3a9dae3abca 02ef7843a642467091f63be0a9128c72 a2b2e6613a6e46bd8d063bc7db0e6bc6 - default default] No notification will be sent due to unsuccessful status code: 500 after /usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/notifier.py:79 2020-03-28 11:55:47.026 29 INFO neutron.wsgi [req-141a2cbb-86b2-449e-9188-f3a9dae3abca 02ef7843a642467091f63be0a9128c72 a2b2e6613a6e46bd8d063bc7db0e6bc6 - default default] 172.17.1.107 "POST /v2.0/ports HTTP/1.1" status: 500 len: 344 time: 2.1275628 Verified that octavia job passed successfully, there were no Neutron server errors on port create https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_2comp_3net-ipv4-geneve-composable/53/ Puddle RHOS-16.1-RHEL-8-20200625.n.0 , python3-ovsdbapp-0.17.3-0.20200528074611.9bb0eee.el8ost.noarch.rpm If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to -. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148 |