2114617 – [OSP17][IPv6][TLS-E][OVN] neutron server keeps crashing on: Exception: Could not retrieve schema from ssl

Bug 2114617 - [OSP17][IPv6][TLS-E][OVN] neutron server keeps crashing on: Exception: Could not retrieve schema from ssl

Summary: [OSP17][IPv6][TLS-E][OVN] neutron server keeps crashing on: Exception: Could ...

Keywords:
Status:	CLOSED DUPLICATE of bug 2115035
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	17.0 (Wallaby)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	ga
Target Release:	17.0
Assignee:	Miro Tomaska
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:	2115035
Blocks:
TreeView+	depends on / blocked

Reported:	2022-08-03 00:43 UTC by Marian Krcmarik
Modified:	2022-08-26 21:31 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-08-23 14:07:50 UTC
Target Upstream Version:
Embargoed:
Flags:	mtomaska: needinfo-

Attachments	(Terms of Use)
neutron server log (15.71 MB, text/plain) 2022-08-03 00:43 UTC, Marian Krcmarik	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-17992	0	None	None	None	2022-08-03 00:51:47 UTC

Description Marian Krcmarik 2022-08-03 00:43:55 UTC

Created attachment 1903144 [details]
neutron server log

Description of problem:
The neutron server keeps crashing right after of a fresh deployment of OSP17 with OVN/IPv6/TLS-everwhere enabled. It seems to be the combination of these three characteristics - OVN/IPv6 and TLS-everywhere. I was not able to reproduce with OVN/IPv4/TLS-E or OVN/IPv6/SSL for api endpoints only deployments.

I can't really say where the problem comes from (It could be a problem with configuration too) but neutron-server should handle it better in any case.

This is the traceback thrown by neutron server (I'll attach the log):
2022-08-03 00:00:54.204 2 CRITICAL neutron [req-ab1fc068-217c-40c3-a2d9-77f6f70b66d5 - - - - -] Unhandled error: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
2022-08-03 00:00:54.204 2 ERROR neutron Traceback (most recent call last):
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/bin/neutron-server", line 10, in <module>
2022-08-03 00:00:54.204 2 ERROR neutron     sys.exit(main())
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/neutron/cmd/eventlet/server/__init__.py", line 20, in main
2022-08-03 00:00:54.204 2 ERROR neutron     server.boot_server(wsgi_eventlet.eventlet_wsgi_server)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/neutron/server/__init__.py", line 67, in boot_server
2022-08-03 00:00:54.204 2 ERROR neutron     return server_func()
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/neutron/server/wsgi_eventlet.py", line 23, in eventlet_wsgi_server
2022-08-03 00:00:54.204 2 ERROR neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/neutron/service.py", line 91, in serve_wsgi
2022-08-03 00:00:54.204 2 ERROR neutron     LOG.exception('Unrecoverable error: please check log '
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-08-03 00:00:54.204 2 ERROR neutron     self.force_reraise()
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-08-03 00:00:54.204 2 ERROR neutron     raise self.value
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/neutron/service.py", line 88, in serve_wsgi

...skipped...

2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 722, in __get__
2022-08-03 00:00:54.204 2 ERROR neutron     return self.func(owner)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 158, in schema_helper
2022-08-03 00:00:54.204 2 ERROR neutron     cls._schema_helper = idlutils.get_schema_helper(cls.connection_string,
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper
2022-08-03 00:00:54.204 2 ERROR neutron     return create_schema_helper(fetch_schema_json(connection, schema_name))
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 183, in fetch_schema_json
2022-08-03 00:00:54.204 2 ERROR neutron     stream.Stream.open(c))
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib64/python3.9/site-packages/ovs/stream.py", line 196, in open
2022-08-03 00:00:54.204 2 ERROR neutron     err = cls.check_connection_completion(sock)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib64/python3.9/site-packages/ovs/stream.py", line 777, in check_connection_completion
2022-08-03 00:00:54.204 2 ERROR neutron     return Stream.check_connection_completion(sock)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib64/python3.9/site-packages/ovs/stream.py", line 137, in check_connection_completion
2022-08-03 00:00:54.204 2 ERROR neutron     return ovs.socket_util.check_connection_completion(sock)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib64/python3.9/site-packages/ovs/socket_util.py", line 181, in check_connection_completion
2022-08-03 00:00:54.204 2 ERROR neutron     sock.send("\0".encode(), socket.MSG_DONTWAIT)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/eventlet/green/ssl.py", line 193, in send
2022-08-03 00:00:54.204 2 ERROR neutron     return self._call_trampolining(
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib/python3.9/site-packages/eventlet/green/ssl.py", line 157, in _call_trampolining
2022-08-03 00:00:54.204 2 ERROR neutron     return func(*a, **kw)
2022-08-03 00:00:54.204 2 ERROR neutron   File "/usr/lib64/python3.9/ssl.py", line 1170, in send
2022-08-03 00:00:54.204 2 ERROR neutron     raise ValueError(
2022-08-03 00:00:54.204 2 ERROR neutron ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>

The overcloud deploy command:
openstack overcloud deploy \
--timeout 240 \
--templates /usr/share/openstack-tripleo-heat-templates \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
--stack central \
--libvirt-type kvm \
--ntp-server clock.corp.redhat.com \
--deployed-server \
-e /home/stack/templates/overcloud-vip-deployed.yaml \
-e /home/stack/templates/overcloud-networks-deployed.yaml \
-e /home/stack/templates/overcloud-baremetal-deployed.yaml \
-e /home/stack/templates/overcloud-ceph-deployed.yaml \
--networks-file /home/stack/central/network/network_data_v2.yaml \
-e /home/stack/central/internal.yaml \
-r /home/stack/central/roles/roles_data.yaml \
-e /home/stack/central/network/network-environment_v2.yaml \
-e /home/stack/central/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e /home/stack/central/glance.yaml \
-e /home/stack/central/nodes_data.yaml \
-e /home/stack/central/debug.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-dashboard.yaml \
-e /home/stack/central/glance_update.yaml \
-e /home/stack/central/dcn_ceph_external.yaml \
-e /home/stack/central/config_heat.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/central/barbican.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /home/stack/central/cloud-names.yaml \
-e /home/stack/central/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /home/stack/central/ipaservices-baremetal-ansible.yaml \
--log-file update_overcloud_deployment_93.log

Version-Release number of selected component (if applicable):
openstack-neutron-18.4.1-0.20220705190433.5258354.el9ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy OSP17 with OVN/IPv6 and TLs-everywhere


Actual results:


Expected results:


Additional info:

Comment 7 Miro Tomaska 2022-08-16 17:28:11 UTC

Hi Marian,

Fix for the non-zero exception is available in errata with a new rpm for openvswitch2.17. Can you apply the new rpm package for openvswitch2.17 to your environment and see if that resolves your issue.
Thanks

https://bugzilla.redhat.com/show_bug.cgi?id=2115035#c11

Comment 8 Marian Krcmarik 2022-08-18 20:30:46 UTC

(In reply to Miro Tomaska from comment #7)
> Hi Marian,
> 
> Fix for the non-zero exception is available in errata with a new rpm for
> openvswitch2.17. Can you apply the new rpm package for openvswitch2.17 to
> your environment and see if that resolves your issue.
> Thanks
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=2115035#c11

It seems It was not the problem but probably The "non-zero exception" got hit when such code path got triggered by the true problem. The exception thrown and traceback look like this now:
2022-08-18 20:22:12.540 2 ERROR neutron.service [req-00c70670-1f9e-48ab-8c4a-f5d774fc6ffc - - - - -] Unrecoverable error: please check log for details.: Exception: Could not retrieve schema from ssl:[fd00:fd00:fd00:2000::3c6]:6642,ssl:[fd00:fd00:fd00:2000::369]:6642,ssl:[fd00:fd00:fd00:2000::38b]:6642
2022-08-18 20:22:12.540 2 ERROR neutron.service Traceback (most recent call last):
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/service.py", line 88, in serve_wsgi
2022-08-18 20:22:12.540 2 ERROR neutron.service     service.start()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/service.py", line 64, in start
2022-08-18 20:22:12.540 2 ERROR neutron.service     self.wsgi_app = _run_wsgi(self.app_name)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/service.py", line 321, in _run_wsgi
2022-08-18 20:22:12.540 2 ERROR neutron.service     app = config.load_paste_app(app_name)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/common/config.py", line 134, in load_paste_app
2022-08-18 20:22:12.540 2 ERROR neutron.service     app = loader.load_app(app_name)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/oslo_service/wsgi.py", line 351, in load_app
2022-08-18 20:22:12.540 2 ERROR neutron.service     return deploy.loadapp("config:%s" % self.config_path, name=name)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 253, in loadapp
2022-08-18 20:22:12.540 2 ERROR neutron.service     return loadobj(APP, uri, name=name, **kw)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 278, in loadobj
2022-08-18 20:22:12.540 2 ERROR neutron.service     return context.create()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2022-08-18 20:22:12.540 2 ERROR neutron.service     return self.object_type.invoke(self)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 148, in invoke
2022-08-18 20:22:12.540 2 ERROR neutron.service     return fix_call(context.object,
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/util.py", line 55, in fix_call
2022-08-18 20:22:12.540 2 ERROR neutron.service     val = callable(*args, **kw)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/auth.py", line 62, in pipeline_factory
2022-08-18 20:22:12.540 2 ERROR neutron.service     app = loader.get_app(pipeline[-1])
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 355, in get_app
2022-08-18 20:22:12.540 2 ERROR neutron.service     return self.app_context(
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2022-08-18 20:22:12.540 2 ERROR neutron.service     return self.object_type.invoke(self)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 152, in invoke
2022-08-18 20:22:12.540 2 ERROR neutron.service     return fix_call(context.object, context.global_conf, **context.local_conf)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/paste/deploy/util.py", line 55, in fix_call
2022-08-18 20:22:12.540 2 ERROR neutron.service     val = callable(*args, **kw)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/api/v2/router.py", line 25, in _factory
2022-08-18 20:22:12.540 2 ERROR neutron.service     return pecan_app.v2_factory(global_config, **local_config)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory
2022-08-18 20:22:12.540 2 ERROR neutron.service     startup.initialize_all()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
2022-08-18 20:22:12.540 2 ERROR neutron.service     manager.init()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 301, in init
2022-08-18 20:22:12.540 2 ERROR neutron.service     NeutronManager.get_instance()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 252, in get_instance
2022-08-18 20:22:12.540 2 ERROR neutron.service     cls._create_instance()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2022-08-18 20:22:12.540 2 ERROR neutron.service     return f(*args, **kwargs)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 238, in _create_instance
2022-08-18 20:22:12.540 2 ERROR neutron.service     cls._instance = cls()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 126, in __init__
2022-08-18 20:22:12.540 2 ERROR neutron.service     plugin = self._get_plugin_instance(CORE_PLUGINS_NAMESPACE,
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 162, in _get_plugin_instance
2022-08-18 20:22:12.540 2 ERROR neutron.service     plugin_inst = plugin_class()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/quota/resource_registry.py", line 124, in wrapper
2022-08-18 20:22:12.540 2 ERROR neutron.service     return f(*args, **kwargs)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 273, in __init__
2022-08-18 20:22:12.540 2 ERROR neutron.service     self.mechanism_manager.initialize()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 459, in initialize
2022-08-18 20:22:12.540 2 ERROR neutron.service     driver.obj.initialize()
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 125, in initialize
2022-08-18 20:22:12.540 2 ERROR neutron.service     if impl_idl_ovn.OvsdbSbOvnIdl.schema_has_table('Chassis_Private'):
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 164, in schema_has_table
2022-08-18 20:22:12.540 2 ERROR neutron.service     return table_name in cls.schema_helper.schema_json['tables']
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 722, in __get__
2022-08-18 20:22:12.540 2 ERROR neutron.service     return self.func(owner)
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 158, in schema_helper
2022-08-18 20:22:12.540 2 ERROR neutron.service     cls._schema_helper = idlutils.get_schema_helper(cls.connection_string,
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 215, in get_schema_helper
2022-08-18 20:22:12.540 2 ERROR neutron.service     return create_schema_helper(fetch_schema_json(connection, schema_name))
2022-08-18 20:22:12.540 2 ERROR neutron.service   File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in fetch_schema_json
2022-08-18 20:22:12.540 2 ERROR neutron.service     raise Exception("Could not retrieve schema from %s" % connection)
2022-08-18 20:22:12.540 2 ERROR neutron.service Exception: Could not retrieve schema from ssl:[fd00:fd00:fd00:2000::3c6]:6642,ssl:[fd00:fd00:fd00:2000::369]:6642,ssl:[fd00:fd00:fd00:2000::38b]:6642

Comment 20 Miro Tomaska 2022-08-23 14:07:50 UTC


*** This bug has been marked as a duplicate of bug 2115035 ***

Note You need to log in before you can comment on or make changes to this bug.