Bug 2027931
| Summary: | OSP16.2 network component jobs are failing with "Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback networking_ovn.ml2.mech_driver.OVNMechanismDriver.pre_fork_initialize--<> failed with "name 'idlutils' is not defined" | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Sandeep Yadav <sandyada> |
| Component: | python-networking-ovn | Assignee: | Rodolfo Alonso <ralonsoh> |
| Status: | CLOSED ERRATA | QA Contact: | Eran Kuris <ekuris> |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.2 (Train) | CC: | apevec, chrisw, lhh, majopela, ralonsoh, scohen |
| Target Milestone: | z2 | Keywords: | Triaged |
| Target Release: | 16.2 (Train on RHEL 8.4) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | python-networking-ovn-7.4.2-2.20220113214852.a2eba10.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-23 22:12:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
all component test passing : https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16.2-component-tests-osp/435/ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1001 |
Description of problem: OSP16.2 network component jobs are failing with "Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback networking_ovn.ml2.mech_driver.OVNMechanismDriver.pre_fork_initialize--<> failed with "name 'idlutils' is not defined" Failure started in network component line on 30th Nov, 21. Change in the following packages compared to last successfully tested package:- Old working one:- ~~~ python-networking-vmware-nsx-15.1.1-2.20211122144928.50837fc.el8osttrunk python-networking-ovn-7.4.2-2.20211126164848.4131acb.el8osttrunk ~~~ New rpm in failing jobs:- ~~~ python-networking-vmware-nsx-15.1.1-2.20211129174926.85b2339.el8osttrunk python-networking-ovn-7.4.2-2.20211129154928.6e8296c.el8osttrunk ~~~ Version-Release number of selected component (if applicable): OSP16.2 (downstream network component ci pipeline) How reproducible: Every time Steps to Reproduce: 1. Deploy standalone 2. Run Tempest Actual results: Neutron returning: "503 Service Unavailable: No server is available to handle this request." ~~~ 2021-11-30 20:38:21.251374 | primary | TASK [os_tempest : Ensure private network exists] ****************************** 2021-11-30 20:38:21.251380 | primary | Tuesday 30 November 2021 20:38:21 -0500 (0:00:00.102) 1:42:07.967 ****** 2021-11-30 20:38:23.358950 | primary | FAILED - RETRYING: Ensure private network exists (5 retries left). 2021-11-30 20:38:35.181217 | primary | FAILED - RETRYING: Ensure private network exists (4 retries left). 2021-11-30 20:38:46.870044 | primary | FAILED - RETRYING: Ensure private network exists (3 retries left). 2021-11-30 20:38:58.985085 | primary | FAILED - RETRYING: Ensure private network exists (2 retries left). 2021-11-30 20:39:11.045460 | primary | FAILED - RETRYING: Ensure private network exists (1 retries left). 2021-11-30 20:39:23.173590 | primary | fatal: [undercloud -> undercloud]: FAILED! => {"attempts": 5, "changed": false, "extra_data": {"data": null, "details": "503 Service Unavailable: No server is available to handle this request.", "response": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n"}, "msg": "HttpException: 503: Server Error for url: http://<masked>:9696/v2.0/networks.json?tenant_id=<masked>, 503 Service Unavailable: No server is available to handle this request."} ~~~~ Expected results: Neutron should not return - 503 Service Unavailable: Additional info: Following traceback in neutron server.log ~~ 2021-12-01 00:45:40.493 7 CRITICAL neutron [req-10ea2090-461f-4f07-9815-ae507f7b715a - - - - -] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback networking_ovn.ml2.mech_driver.OVNMechanismDriver.pre_fork_initialize--9223372036839338929 failed with "name 'idlutils' is not defined" 2021-12-01 00:45:40.493 7 ERROR neutron Traceback (most recent call last): 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module> 2021-12-01 00:45:40.493 7 ERROR neutron sys.exit(main()) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main 2021-12-01 00:45:40.493 7 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/server/__init__.py", line 68, in boot_server 2021-12-01 00:45:40.493 7 ERROR neutron server_func() 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/server/wsgi_eventlet.py", line 23, in eventlet_wsgi_server 2021-12-01 00:45:40.493 7 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron/service.py", line 94, in serve_wsgi 2021-12-01 00:45:40.493 7 ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron_lib/callbacks/registry.py", line 60, in publish 2021-12-01 00:45:40.493 7 ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 149, in publish 2021-12-01 00:45:40.493 7 ERROR neutron return self.notify(resource, event, trigger, payload=payload) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron_lib/db/utils.py", line 108, in _wrapped 2021-12-01 00:45:40.493 7 ERROR neutron raise db_exc.RetryRequest(e) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-12-01 00:45:40.493 7 ERROR neutron self.force_reraise() 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-12-01 00:45:40.493 7 ERROR neutron six.reraise(self.type_, self.value, self.tb) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-12-01 00:45:40.493 7 ERROR neutron raise value 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron_lib/db/utils.py", line 103, in _wrapped 2021-12-01 00:45:40.493 7 ERROR neutron return function(*args, **kwargs) 2021-12-01 00:45:40.493 7 ERROR neutron File "/usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 174, in notify 2021-12-01 00:45:40.493 7 ERROR neutron raise exceptions.CallbackFailure(errors=errors) 2021-12-01 00:45:40.493 7 ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback networking_ovn.ml2.mech_driver.OVNMechanismDriver.pre_fork_initialize--9223372036839338929 failed with "name 'idlutils' is not defined" 2021-11-30 14:11:35.640 ERROR /var/log/containers/neutron/server.log: 15 ERROR networking_ovn.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'17ea15f6-b6c2-4cfc-b1f8-14d86e6882e5'". This should never happen in a normal situation, please check the status of your cluster: networking_ovn.common.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'17ea15f6-b6c2-4cfc-b1f8-14d86e6882e5'". This should never happen in a normal situation, please check the status of your cluster ~~~